Sep 19 04:13:39.078519 ip-10-0-129-215 systemd[1]: Starting Kubernetes Kubelet...
Sep 19 04:13:39.473585 ip-10-0-129-215 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 19 04:13:39.473585 ip-10-0-129-215 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 19 04:13:39.473585 ip-10-0-129-215 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 19 04:13:39.473585 ip-10-0-129-215 kubenswrapper[2579]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 19 04:13:39.473585 ip-10-0-129-215 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 19 04:13:39.474696 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.474602    2579 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 19 04:13:39.480020 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480001    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 19 04:13:39.480061 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480021    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 19 04:13:39.480061 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480025    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 19 04:13:39.480061 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480028    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 19 04:13:39.480061 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480031    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 19 04:13:39.480061 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480034    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 19 04:13:39.480061 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480037    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 19 04:13:39.480061 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480040    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 19 04:13:39.480061 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480043    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 19 04:13:39.480061 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480046    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 19 04:13:39.480061 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480048    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 19 04:13:39.480061 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480051    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 19 04:13:39.480061 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480054    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 19 04:13:39.480061 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480056    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 19 04:13:39.480061 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480059    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 19 04:13:39.480061 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480062    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 19 04:13:39.480061 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480065    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 19 04:13:39.480061 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480068    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 19 04:13:39.480513 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480077    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 19 04:13:39.480513 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480079    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 19 04:13:39.480513 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480082    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 19 04:13:39.480513 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480085    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 19 04:13:39.480513 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480087    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 19 04:13:39.480513 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480090    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 19 04:13:39.480513 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480092    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 19 04:13:39.480513 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480095    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 19 04:13:39.480513 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480098    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 19 04:13:39.480513 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480100    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 19 04:13:39.480513 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480102    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 19 04:13:39.480513 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480105    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 19 04:13:39.480513 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480107    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 19 04:13:39.480513 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480110    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 19 04:13:39.480513 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480112    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 19 04:13:39.480513 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480115    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 19 04:13:39.480513 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480117    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 19 04:13:39.480513 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480120    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 19 04:13:39.480513 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480123    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 19 04:13:39.480513 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480125    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 19 04:13:39.481122 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480127    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 19 04:13:39.481122 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480130    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 19 04:13:39.481122 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480132    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 19 04:13:39.481122 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480134    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 19 04:13:39.481122 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480137    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 19 04:13:39.481122 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480139    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 19 04:13:39.481122 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480141    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 19 04:13:39.481122 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480144    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 19 04:13:39.481122 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480146    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 19 04:13:39.481122 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480149    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 19 04:13:39.481122 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480151    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 19 04:13:39.481122 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480154    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 19 04:13:39.481122 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480156    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 19 04:13:39.481122 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480165    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 19 04:13:39.481122 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480168    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 19 04:13:39.481122 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480170    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 19 04:13:39.481122 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480173    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 19 04:13:39.481122 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480175    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 19 04:13:39.481122 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480178    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 19 04:13:39.481807 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480180    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 19 04:13:39.481807 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480183    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 19 04:13:39.481807 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480185    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 19 04:13:39.481807 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480200    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 19 04:13:39.481807 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480203    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 19 04:13:39.481807 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480205    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 19 04:13:39.481807 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480218    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 19 04:13:39.481807 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480222    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 19 04:13:39.481807 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480225    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 19 04:13:39.481807 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480228    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 19 04:13:39.481807 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480231    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 19 04:13:39.481807 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480233    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 19 04:13:39.481807 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480236    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 19 04:13:39.481807 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480239    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 19 04:13:39.481807 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480242    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 19 04:13:39.481807 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480244    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 19 04:13:39.481807 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480247    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 19 04:13:39.481807 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480250    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 19 04:13:39.481807 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480252    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 19 04:13:39.482469 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480255    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 19 04:13:39.482469 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480257    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 19 04:13:39.482469 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480260    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 19 04:13:39.482469 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480262    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 19 04:13:39.482469 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480264    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 19 04:13:39.482469 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480267    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 19 04:13:39.482469 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480270    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 19 04:13:39.482469 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480273    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 19 04:13:39.482469 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480281    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 19 04:13:39.482469 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480284    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 19 04:13:39.482469 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480288    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 19 04:13:39.482469 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480291    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 19 04:13:39.482469 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480293    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 19 04:13:39.482469 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480295    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 19 04:13:39.482469 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480298    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 19 04:13:39.482469 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480300    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 19 04:13:39.482469 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480303    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 19 04:13:39.482469 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480306    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 19 04:13:39.482469 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480310    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 19 04:13:39.483289 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480313    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 19 04:13:39.483289 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480315    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 19 04:13:39.483289 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480318    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 19 04:13:39.483289 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.480320    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 19 04:13:39.483289 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.482673    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 19 04:13:39.483289 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.482687    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 19 04:13:39.483289 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.482693    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 19 04:13:39.483289 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.482698    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 19 04:13:39.483289 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.482702    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 19 04:13:39.483289 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.482706    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 19 04:13:39.483289 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.482711    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 19 04:13:39.483289 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.482715    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 19 04:13:39.483289 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.482719    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 19 04:13:39.483289 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.482723    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 19 04:13:39.483289 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.482728    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 19 04:13:39.483289 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.482732    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 19 04:13:39.483289 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.482736    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 19 04:13:39.483289 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.482740    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 19 04:13:39.483289 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.482744    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 19 04:13:39.483289 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.482748    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 19 04:13:39.484185 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.482753    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 19 04:13:39.484185 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.482757    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 19 04:13:39.484185 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.482761    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 19 04:13:39.484185 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.482765    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 19 04:13:39.484185 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.482770    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 19 04:13:39.484185 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.482774    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 19 04:13:39.484185 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.482778    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 19 04:13:39.484185 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.482782    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 19 04:13:39.484185 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.482786    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 19 04:13:39.484185 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.482790    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 19 04:13:39.484185 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.482794    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 19 04:13:39.484185 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.482798    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 19 04:13:39.484185 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.482803    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 19 04:13:39.484185 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.482807    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 19 04:13:39.484185 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.482811    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 19 04:13:39.484185 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.482815    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 19 04:13:39.484185 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.482819    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 19 04:13:39.484185 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.482823    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 19 04:13:39.484185 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.482828    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 19 04:13:39.484185 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.482832    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 19 04:13:39.485082 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.482837    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 19 04:13:39.485082 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.482844    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 19 04:13:39.485082 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.482850    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 19 04:13:39.485082 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.482855    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 19 04:13:39.485082 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.482859    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 19 04:13:39.485082 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.482863    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 19 04:13:39.485082 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.482868    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 19 04:13:39.485082 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.482872    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 19 04:13:39.485082 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.482876    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 19 04:13:39.485082 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.482880    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 19 04:13:39.485082 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.482885    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 19 04:13:39.485082 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.482889    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 19 04:13:39.485082 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.482893    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 19 04:13:39.485082 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.482897    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 19 04:13:39.485082 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.482903    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 19 04:13:39.485082 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.482908    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 19 04:13:39.485082 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.482912    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 19 04:13:39.485082 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.482917    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 19 04:13:39.485082 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.482921    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 19 04:13:39.485082 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.482925    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 19 04:13:39.485946 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.482929    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 19 04:13:39.485946 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.482933    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 19 04:13:39.485946 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.482937    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 19 04:13:39.485946 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.482941    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 19 04:13:39.485946 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.482946    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 19 04:13:39.485946 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.482951    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 19 04:13:39.485946 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.482958    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 19 04:13:39.485946 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.482964    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 19 04:13:39.485946 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.482969    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 19 04:13:39.485946 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.482975    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 19 04:13:39.485946 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.482980    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 19 04:13:39.485946 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.482984    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 19 04:13:39.485946 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.482988    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 19 04:13:39.485946 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.482992    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 19 04:13:39.485946 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.482997    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 19 04:13:39.485946 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.483001    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 19 04:13:39.485946 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.483005    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 19 04:13:39.485946 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.483009    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 19 04:13:39.485946 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.483013    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 19 04:13:39.485946 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.483018    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 19 04:13:39.486464 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.483021    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 19 04:13:39.486464 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.483026    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 19 04:13:39.486464 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.483030    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 19 04:13:39.486464 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.483034    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 19 04:13:39.486464 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.483038    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 19 04:13:39.486464 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.483042    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 19 04:13:39.486464 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.483047    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 19 04:13:39.486464 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.483052    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 19 04:13:39.486464 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.483056    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 19 04:13:39.486464 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.483062    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 19 04:13:39.486464 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.483070    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 19 04:13:39.486464 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.483075    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 19 04:13:39.486464 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.483079    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 19 04:13:39.486464 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.483083    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 19 04:13:39.486464 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.483087    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 19 04:13:39.486464 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.483091    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 19 04:13:39.486464 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.483095    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 19 04:13:39.486464 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.483099    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 19 04:13:39.486464 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.483103    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 19 04:13:39.487020 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.483107    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 19 04:13:39.487020 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.483112    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 19 04:13:39.487020 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.483117    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 19 04:13:39.487020 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.483121    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 19 04:13:39.487020 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.483772    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 19 04:13:39.487020 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.483780    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 19 04:13:39.487020 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.483785    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 19 04:13:39.487020 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.483789    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 19 04:13:39.487020 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.483794    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 19 04:13:39.487020 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.483798    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 19 04:13:39.487020 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.483802    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 19 04:13:39.487020 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.483807    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 19 04:13:39.487020 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.483814    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 19 04:13:39.487020 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.483820    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 19 04:13:39.487020 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.483824    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 19 04:13:39.487020 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.483829    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 19 04:13:39.487020 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.483833    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 19 04:13:39.487020 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.483839    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 19 04:13:39.487020 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.483843    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 19 04:13:39.487020 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.483847    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 19 04:13:39.487568 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.483852    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 19 04:13:39.487568 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.483857    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 19 04:13:39.487568 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.483861    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 19 04:13:39.487568 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.483865    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 19 04:13:39.487568 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.483869    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 19 04:13:39.487568 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.483873    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 19 04:13:39.487568 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.483877    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 19 04:13:39.487568 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.483882    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 19 04:13:39.487568 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.483886    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 19 04:13:39.487568 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.483890    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 19 04:13:39.487568 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.483894    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 19 04:13:39.487568 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.483898    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 19 04:13:39.487568 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.483902    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 19 04:13:39.487568 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.483906    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 19 04:13:39.487568 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.483911    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 19 04:13:39.487568 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.483916    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 19 04:13:39.487568 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.483920    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 19 04:13:39.487568 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.483924    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 19 04:13:39.487568 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.483928    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 19 04:13:39.487568 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.483933    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 19 04:13:39.488051 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.483937    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 19 04:13:39.488051 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.483940    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 19 04:13:39.488051 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.483944    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 19 04:13:39.488051 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.483949    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 19 04:13:39.488051 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.483953    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 19 04:13:39.488051 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.483957    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 19 04:13:39.488051 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.483961    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 19 04:13:39.488051 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.483965    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 19 04:13:39.488051 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.483969    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 19 04:13:39.488051 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.483973    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 19 04:13:39.488051 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.483978    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 19 04:13:39.488051 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.483983    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 19 04:13:39.488051 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.483987    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 19 04:13:39.488051 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.483991    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 19 04:13:39.488051 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.483996    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 19 04:13:39.488051 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.483999    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 19 04:13:39.488051 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.484003    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 19 04:13:39.488051 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.484007    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 19 04:13:39.488051 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.484011    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 19 04:13:39.488051 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.484015    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 19 04:13:39.488779 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.484019    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 19 04:13:39.488779 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.484023    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 19 04:13:39.488779 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.484026    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 19 04:13:39.488779 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.484032    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 19 04:13:39.488779 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.484036    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 19 04:13:39.488779 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.484041    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 19 04:13:39.488779 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.484047    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 19 04:13:39.488779 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.484051    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 19 04:13:39.488779 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.484056    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 19 04:13:39.488779 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.484060    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 19 04:13:39.488779 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.484064    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 19 04:13:39.488779 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.484069    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 19 04:13:39.488779 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.484073    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 19 04:13:39.488779 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.484077    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 19 04:13:39.488779 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.484081    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 19 04:13:39.488779 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.484085    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 19 04:13:39.488779 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.484089    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 19 04:13:39.488779 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.484093    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 19 04:13:39.488779 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.484097    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 19 04:13:39.489267 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.484104    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 19 04:13:39.489267 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.484111    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 19 04:13:39.489267 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.484116    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 19 04:13:39.489267 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.484120    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 19 04:13:39.489267 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.484124    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 19 04:13:39.489267 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.484130    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 19 04:13:39.489267 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.484136    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 19 04:13:39.489267 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.484142    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 19 04:13:39.489267 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.484147    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 19 04:13:39.489267 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.484151    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 19 04:13:39.489267 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.484155    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 19 04:13:39.489267 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.484160    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 19 04:13:39.489267 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.484164    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 19 04:13:39.489267 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.484168    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 19 04:13:39.489267 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.484172    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 19 04:13:39.489267 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.484176    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 19 04:13:39.489267 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.484180    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 19 04:13:39.489267 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.484184    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 19 04:13:39.489267 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.484209    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 19 04:13:39.489829 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.484217    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 19 04:13:39.489829 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.484221    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 19 04:13:39.489829 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.484225    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 19 04:13:39.489829 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.484229    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 19 04:13:39.489829 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.484233    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 19 04:13:39.489829 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484340    2579 flags.go:64] FLAG: --address="0.0.0.0"
Sep 19 04:13:39.489829 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484396    2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 19 04:13:39.489829 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484406    2579 flags.go:64] FLAG: --anonymous-auth="true"
Sep 19 04:13:39.489829 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484414    2579 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 19 04:13:39.489829 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484420    2579 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 19 04:13:39.489829 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484425    2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 19 04:13:39.489829 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484432    2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 19 04:13:39.489829 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484439    2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 19 04:13:39.489829 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484444    2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 19 04:13:39.489829 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484448    2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 19 04:13:39.489829 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484454    2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 19 04:13:39.489829 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484459    2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 19 04:13:39.489829 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484464    2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 19 04:13:39.489829 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484469    2579 flags.go:64] FLAG: --cgroup-root=""
Sep 19 04:13:39.489829 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484474    2579 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 19 04:13:39.489829 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484478    2579 flags.go:64] FLAG: --client-ca-file=""
Sep 19 04:13:39.489829 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484484    2579 flags.go:64] FLAG: --cloud-provider="external"
Sep 19 04:13:39.489829 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484489    2579 flags.go:64] FLAG: --cluster-dns="[]"
Sep 19 04:13:39.490493 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484495    2579 flags.go:64] FLAG: --cluster-domain=""
Sep 19 04:13:39.490493 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484499    2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 19 04:13:39.490493 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484504    2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 19 04:13:39.490493 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484509    2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 19 04:13:39.490493 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484514    2579 flags.go:64] FLAG: --container-log-max-files="5"
Sep 19 04:13:39.490493 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484521    2579 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 19 04:13:39.490493 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484525    2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 19 04:13:39.490493 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484530    2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 19 04:13:39.490493 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484535    2579 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 19 04:13:39.490493 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484541    2579 flags.go:64] FLAG: --contention-profiling="false"
Sep 19 04:13:39.490493 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484546    2579 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 19 04:13:39.490493 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484550    2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 19 04:13:39.490493 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484555    2579 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 19 04:13:39.490493 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484560    2579 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 19 04:13:39.490493 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484566    2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 19 04:13:39.490493 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484571    2579 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 19 04:13:39.490493 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484576    2579 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 19 04:13:39.490493 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484580    2579 flags.go:64] FLAG: --enable-load-reader="false"
Sep 19 04:13:39.490493 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484585    2579 flags.go:64] FLAG: --enable-server="true"
Sep 19 04:13:39.490493 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484589    2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 19 04:13:39.490493 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484596    2579 flags.go:64] FLAG: --event-burst="100"
Sep 19 04:13:39.490493 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484601    2579 flags.go:64] FLAG: --event-qps="50"
Sep 19 04:13:39.490493 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484606    2579 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 19 04:13:39.490493 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484610    2579 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 19 04:13:39.491099 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484615    2579 flags.go:64] FLAG: --eviction-hard=""
Sep 19 04:13:39.491099 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484621    2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 19 04:13:39.491099 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484626    2579 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 19 04:13:39.491099 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484630    2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 19 04:13:39.491099 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484635    2579 flags.go:64] FLAG: --eviction-soft=""
Sep 19 04:13:39.491099 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484640    2579 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 19 04:13:39.491099 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484644    2579 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 19 04:13:39.491099 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484650    2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 19 04:13:39.491099 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484655    2579 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 19 04:13:39.491099 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484660    2579 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 19 04:13:39.491099 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484665    2579 flags.go:64] FLAG: --fail-swap-on="true"
Sep 19 04:13:39.491099 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484670    2579 flags.go:64] FLAG: --feature-gates=""
Sep 19 04:13:39.491099 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484676    2579 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 19 04:13:39.491099 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484680    2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 19 04:13:39.491099 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484685    2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 19 04:13:39.491099 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484690    2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 19 04:13:39.491099 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484696    2579 flags.go:64] FLAG: --healthz-port="10248"
Sep 19 04:13:39.491099 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484701    2579 flags.go:64] FLAG: --help="false"
Sep 19 04:13:39.491099 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484706    2579 flags.go:64] FLAG: --hostname-override="ip-10-0-129-215.ec2.internal"
Sep 19 04:13:39.491099 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484711    2579 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 19 04:13:39.491099 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484715    2579 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 19 04:13:39.491099 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484720    2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 19 04:13:39.491099 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484726    2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 19 04:13:39.491099 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484732    2579 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 19 04:13:39.491721 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484736    2579 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 19 04:13:39.491721 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484741    2579 flags.go:64] FLAG: --image-service-endpoint=""
Sep 19 04:13:39.491721 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484746    2579 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 19 04:13:39.491721 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484751    2579 flags.go:64] FLAG: --kube-api-burst="100"
Sep 19 04:13:39.491721 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484756    2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 19 04:13:39.491721 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484761    2579 flags.go:64] FLAG: --kube-api-qps="50"
Sep 19 04:13:39.491721 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484765    2579 flags.go:64] FLAG: --kube-reserved=""
Sep 19 04:13:39.491721 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484770    2579 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 19 04:13:39.491721 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484775    2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 19 04:13:39.491721 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484780    2579 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 19 04:13:39.491721 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484784    2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 19 04:13:39.491721 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484789    2579 flags.go:64] FLAG: --lock-file=""
Sep 19 04:13:39.491721 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484794    2579 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 19 04:13:39.491721 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484798    2579 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 19 04:13:39.491721 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484804    2579 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 19 04:13:39.491721 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484815    2579 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 19 04:13:39.491721 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484821    2579 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 19 04:13:39.491721 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484826    2579 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 19 04:13:39.491721 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484830    2579 flags.go:64] FLAG: --logging-format="text"
Sep 19 04:13:39.491721 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484835    2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 19 04:13:39.491721 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484840    2579 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 19 04:13:39.491721 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484845    2579 flags.go:64] FLAG: --manifest-url=""
Sep 19 04:13:39.491721 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484850    2579 flags.go:64] FLAG: --manifest-url-header=""
Sep 19 04:13:39.491721 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484856    2579 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 19 04:13:39.491721 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484861    2579 flags.go:64] FLAG: --max-open-files="1000000"
Sep 19 04:13:39.492357 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484868    2579 flags.go:64] FLAG: --max-pods="110"
Sep 19 04:13:39.492357 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484873    2579 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 19 04:13:39.492357 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484877    2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 19 04:13:39.492357 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484882    2579 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 19 04:13:39.492357 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484886    2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 19 04:13:39.492357 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484891    2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 19 04:13:39.492357 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484896    2579 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 19 04:13:39.492357 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484900    2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 19 04:13:39.492357 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484912    2579 flags.go:64] FLAG: --node-status-max-images="50"
Sep 19 04:13:39.492357 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484917    2579 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 19 04:13:39.492357 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484922    2579 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 19 04:13:39.492357 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484926    2579 flags.go:64] FLAG: --pod-cidr=""
Sep 19 04:13:39.492357 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484931    2579 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 19 04:13:39.492357 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484939    2579 flags.go:64] FLAG: --pod-manifest-path=""
Sep 19 04:13:39.492357 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484944    2579 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 19 04:13:39.492357 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484949    2579 flags.go:64] FLAG: --pods-per-core="0"
Sep 19 04:13:39.492357 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484953    2579 flags.go:64] FLAG: --port="10250"
Sep 19 04:13:39.492357 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484957    2579 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 19 04:13:39.492357 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484962    2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0933bfe163bb16549"
Sep 19 04:13:39.492357 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484968    2579 flags.go:64] FLAG: --qos-reserved=""
Sep 19 04:13:39.492357 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484972    2579 flags.go:64] FLAG: --read-only-port="10255"
Sep 19 04:13:39.492357 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484977    2579 flags.go:64] FLAG: --register-node="true"
Sep 19 04:13:39.492357 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484984    2579 flags.go:64] FLAG: --register-with-taints=""
Sep 19 04:13:39.492357 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484989    2579 flags.go:64] FLAG: --registry-burst="10"
Sep 19 04:13:39.492944 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484995    2579 flags.go:64] FLAG: --registry-qps="5"
Sep 19 04:13:39.492944 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.484999    2579 flags.go:64] FLAG: --reserved-cpus=""
Sep 19 04:13:39.492944 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.485004    2579 flags.go:64] FLAG: --reserved-memory=""
Sep 19 04:13:39.492944 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.485010    2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 19 04:13:39.492944 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.485014    2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 19 04:13:39.492944 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.485019    2579 flags.go:64] FLAG: --rotate-certificates="false"
Sep 19 04:13:39.492944 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.485024    2579 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 19 04:13:39.492944 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.485028    2579 flags.go:64] FLAG: --runonce="false"
Sep 19 04:13:39.492944 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.485033    2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 19 04:13:39.492944 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.485038    2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 19 04:13:39.492944 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.485042    2579 flags.go:64] FLAG: --seccomp-default="false"
Sep 19 04:13:39.492944 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.485047    2579 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 19 04:13:39.492944 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.485052    2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 19 04:13:39.492944 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.485056    2579 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 19 04:13:39.492944 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.485061    2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 19 04:13:39.492944 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.485065    2579 flags.go:64] FLAG: --storage-driver-password="root"
Sep 19 04:13:39.492944 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.485070    2579 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 19 04:13:39.492944 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.485074    2579 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 19 04:13:39.492944 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.485079    2579 flags.go:64] FLAG: --storage-driver-user="root"
Sep 19 04:13:39.492944 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.485083    2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 19 04:13:39.492944 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.485087    2579 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 19 04:13:39.492944 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.485092    2579 flags.go:64] FLAG: --system-cgroups=""
Sep 19 04:13:39.492944 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.485096    2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 19 04:13:39.492944 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.485104    2579 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 19 04:13:39.492944 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.485108    2579 flags.go:64] FLAG: --tls-cert-file=""
Sep 19 04:13:39.493648 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.485113    2579 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 19 04:13:39.493648 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.485119    2579 flags.go:64] FLAG: --tls-min-version=""
Sep 19 04:13:39.493648 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.485124    2579 flags.go:64] FLAG: --tls-private-key-file=""
Sep 19 04:13:39.493648 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.485128    2579 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 19 04:13:39.493648 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.485133    2579 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 19 04:13:39.493648 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.485139    2579 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 19 04:13:39.493648 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.485143    2579 flags.go:64] FLAG: --v="2"
Sep 19 04:13:39.493648 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.485150    2579 flags.go:64] FLAG: --version="false"
Sep 19 04:13:39.493648 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.485158    2579 flags.go:64] FLAG: --vmodule=""
Sep 19 04:13:39.493648 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.485163    2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 19 04:13:39.493648 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.485169    2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 19 04:13:39.493648 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485380    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 19 04:13:39.493648 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485390    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 19 04:13:39.493648 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485396    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 19 04:13:39.493648 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485401    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 19 04:13:39.493648 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485406    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 19 04:13:39.493648 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485410    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 19 04:13:39.493648 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485414    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 19 04:13:39.493648 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485418    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 19 04:13:39.493648 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485422    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 19 04:13:39.493648 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485427    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 19 04:13:39.493648 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485430    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 19 04:13:39.494284 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485435    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 19 04:13:39.494284 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485439    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 19 04:13:39.494284 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485443    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 19 04:13:39.494284 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485447    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 19 04:13:39.494284 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485451    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 19 04:13:39.494284 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485455    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 19 04:13:39.494284 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485459    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 19 04:13:39.494284 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485463    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 19 04:13:39.494284 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485467    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 19 04:13:39.494284 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485471    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 19 04:13:39.494284 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485477    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 19 04:13:39.494284 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485483    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 19 04:13:39.494284 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485487    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 19 04:13:39.494284 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485516    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 19 04:13:39.494284 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485523    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 19 04:13:39.494284 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485530    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 19 04:13:39.494284 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485535    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 19 04:13:39.494284 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485540    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 19 04:13:39.494284 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485545    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 19 04:13:39.494792 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485550    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 19 04:13:39.494792 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485555    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 19 04:13:39.494792 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485559    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 19 04:13:39.494792 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485565    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 19 04:13:39.494792 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485571    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 19 04:13:39.494792 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485577    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 19 04:13:39.494792 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485581    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 19 04:13:39.494792 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485585    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 19 04:13:39.494792 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485590    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 19 04:13:39.494792 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485594    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 19 04:13:39.494792 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485598    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 19 04:13:39.494792 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485602    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 19 04:13:39.494792 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485607    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 19 04:13:39.494792 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485611    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 19 04:13:39.494792 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485615    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 19 04:13:39.494792 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485619    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 19 04:13:39.494792 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485623    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 19 04:13:39.494792 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485627    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 19 04:13:39.494792 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485631    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 19 04:13:39.494792 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485635    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 19 04:13:39.495311 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485639    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 19 04:13:39.495311 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485644    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 19 04:13:39.495311 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485648    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 19 04:13:39.495311 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485652    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 19 04:13:39.495311 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485656    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 19 04:13:39.495311 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485660    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 19 04:13:39.495311 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485664    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 19 04:13:39.495311 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485668    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 19 04:13:39.495311 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485674    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 19 04:13:39.495311 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485678    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 19 04:13:39.495311 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485682    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 19 04:13:39.495311 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485686    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 19 04:13:39.495311 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485691    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 19 04:13:39.495311 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485696    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 19 04:13:39.495311 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485701    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 19 04:13:39.495311 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485706    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 19 04:13:39.495311 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485710    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 19 04:13:39.495311 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485714    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 19 04:13:39.495311 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485718    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 19 04:13:39.495311 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485722    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 19 04:13:39.495797 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485727    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 19 04:13:39.495797 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485739    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 19 04:13:39.495797 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485744    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 19 04:13:39.495797 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485748    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 19 04:13:39.495797 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485752    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 19 04:13:39.495797 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485756    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 19 04:13:39.495797 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485760    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 19 04:13:39.495797 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485767    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 19 04:13:39.495797 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485772    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 19 04:13:39.495797 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485777    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 19 04:13:39.495797 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485781    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 19 04:13:39.495797 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485786    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 19 04:13:39.495797 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485790    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 19 04:13:39.495797 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485795    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 19 04:13:39.495797 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485798    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 19 04:13:39.495797 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485803    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 19 04:13:39.495797 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485807    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 19 04:13:39.495797 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485811    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 19 04:13:39.495797 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485815    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 19 04:13:39.496282 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485819    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 19 04:13:39.496282 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485826    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 19 04:13:39.496282 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485830    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 19 04:13:39.496282 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485835    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 19 04:13:39.496282 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485839    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 19 04:13:39.496282 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485843    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 19 04:13:39.496282 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485847    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 19 04:13:39.496282 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485852    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 19 04:13:39.496282 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485858    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 19 04:13:39.496282 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.485862    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 19 04:13:39.496282 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.486527    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 19 04:13:39.496282 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.495079    2579 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 19 04:13:39.496282 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.495097    2579 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 19 04:13:39.496282 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.495978    2579 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 19 04:13:39.496640 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.495986    2579 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 19 04:13:39.496640 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496044    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 19 04:13:39.496640 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496050    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 19 04:13:39.496640 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496053    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 19 04:13:39.496640 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496056    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 19 04:13:39.496640 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496059    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 19 04:13:39.496640 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496062    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 19 04:13:39.496640 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496064    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 19 04:13:39.496640 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496067    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 19 04:13:39.496640 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496070    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 19 04:13:39.496640 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496072    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 19 04:13:39.496640 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496075    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 19 04:13:39.496640 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496078    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 19 04:13:39.496640 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496081    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 19 04:13:39.496640 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496083    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 19 04:13:39.496640 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496086    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 19 04:13:39.496640 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496089    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 19 04:13:39.496640 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496091    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 19 04:13:39.496640 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496093    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 19 04:13:39.496640 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496096    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 19 04:13:39.497128 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496098    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 19 04:13:39.497128 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496101    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 19 04:13:39.497128 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496103    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 19 04:13:39.497128 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496106    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 19 04:13:39.497128 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496108    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 19 04:13:39.497128 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496111    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 19 04:13:39.497128 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496113    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 19 04:13:39.497128 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496116    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 19 04:13:39.497128 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496118    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 19 04:13:39.497128 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496120    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 19 04:13:39.497128 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496131    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 19 04:13:39.497128 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496134    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 19 04:13:39.497128 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496137    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 19 04:13:39.497128 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496139    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 19 04:13:39.497128 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496143    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 19 04:13:39.497128 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496147    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 19 04:13:39.497128 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496151    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 19 04:13:39.497128 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496153    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 19 04:13:39.497128 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496156    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 19 04:13:39.497128 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496159    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 19 04:13:39.497641 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496161    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 19 04:13:39.497641 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496164    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 19 04:13:39.497641 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496166    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 19 04:13:39.497641 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496168    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 19 04:13:39.497641 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496171    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 19 04:13:39.497641 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496173    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 19 04:13:39.497641 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496176    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 19 04:13:39.497641 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496179    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 19 04:13:39.497641 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496183    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 19 04:13:39.497641 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496200    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 19 04:13:39.497641 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496203    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 19 04:13:39.497641 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496206    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 19 04:13:39.497641 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496209    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 19 04:13:39.497641 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496212    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 19 04:13:39.497641 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496215    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 19 04:13:39.497641 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496217    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 19 04:13:39.497641 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496220    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 19 04:13:39.497641 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496223    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 19 04:13:39.497641 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496226    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 19 04:13:39.497641 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496229    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 19 04:13:39.498160 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496232    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 19 04:13:39.498160 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496234    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 19 04:13:39.498160 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496237    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 19 04:13:39.498160 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496245    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 19 04:13:39.498160 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496248    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 19 04:13:39.498160 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496251    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 19 04:13:39.498160 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496254    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 19 04:13:39.498160 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496257    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 19 04:13:39.498160 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496259    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 19 04:13:39.498160 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496262    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 19 04:13:39.498160 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496264    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 19 04:13:39.498160 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496267    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 19 04:13:39.498160 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496269    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 19 04:13:39.498160 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496271    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 19 04:13:39.498160 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496274    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 19 04:13:39.498160 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496276    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 19 04:13:39.498160 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496279    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 19 04:13:39.498160 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496281    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 19 04:13:39.498160 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496284    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 19 04:13:39.498160 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496286    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 19 04:13:39.498725 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496289    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 19 04:13:39.498725 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496293    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 19 04:13:39.498725 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496296    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 19 04:13:39.498725 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496299    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 19 04:13:39.498725 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496302    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 19 04:13:39.498725 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496305    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 19 04:13:39.498725 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496308    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 19 04:13:39.498725 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496311    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 19 04:13:39.498725 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496313    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 19 04:13:39.498725 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496317    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 19 04:13:39.498725 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496320    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 19 04:13:39.498725 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496323    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 19 04:13:39.498725 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496326    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 19 04:13:39.498725 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496328    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 19 04:13:39.498725 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496331    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 19 04:13:39.498725 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496334    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 19 04:13:39.498725 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496337    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 19 04:13:39.498725 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496339    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 19 04:13:39.498725 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496342    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 19 04:13:39.499180 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496345    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 19 04:13:39.499180 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.496349    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 19 04:13:39.499180 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496460    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 19 04:13:39.499180 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496465    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 19 04:13:39.499180 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496468    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 19 04:13:39.499180 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496471    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 19 04:13:39.499180 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496473    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 19 04:13:39.499180 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496476    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 19 04:13:39.499180 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496479    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 19 04:13:39.499180 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496482    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 19 04:13:39.499180 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496485    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 19 04:13:39.499180 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496488    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 19 04:13:39.499180 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496491    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 19 04:13:39.499180 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496493    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 19 04:13:39.499180 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496497    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 19 04:13:39.499568 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496499    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 19 04:13:39.499568 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496502    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 19 04:13:39.499568 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496504    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 19 04:13:39.499568 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496507    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 19 04:13:39.499568 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496509    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 19 04:13:39.499568 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496512    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 19 04:13:39.499568 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496514    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 19 04:13:39.499568 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496516    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 19 04:13:39.499568 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496519    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 19 04:13:39.499568 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496522    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 19 04:13:39.499568 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496524    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 19 04:13:39.499568 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496527    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 19 04:13:39.499568 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496530    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 19 04:13:39.499568 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496532    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 19 04:13:39.499568 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496536    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 19 04:13:39.499568 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496540    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 19 04:13:39.499568 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496543    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 19 04:13:39.499568 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496545    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 19 04:13:39.499568 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496548    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 19 04:13:39.500020 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496551    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 19 04:13:39.500020 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496553    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 19 04:13:39.500020 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496556    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 19 04:13:39.500020 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496559    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 19 04:13:39.500020 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496561    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 19 04:13:39.500020 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496563    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 19 04:13:39.500020 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496566    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 19 04:13:39.500020 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496568    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 19 04:13:39.500020 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496571    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 19 04:13:39.500020 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496573    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 19 04:13:39.500020 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496576    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 19 04:13:39.500020 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496578    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 19 04:13:39.500020 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496581    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 19 04:13:39.500020 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496583    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 19 04:13:39.500020 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496586    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 19 04:13:39.500020 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496588    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 19 04:13:39.500020 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496590    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 19 04:13:39.500020 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496593    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 19 04:13:39.500020 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496595    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 19 04:13:39.500020 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496598    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 19 04:13:39.500515 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496600    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 19 04:13:39.500515 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496603    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 19 04:13:39.500515 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496606    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 19 04:13:39.500515 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496608    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 19 04:13:39.500515 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496611    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 19 04:13:39.500515 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496614    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 19 04:13:39.500515 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496617    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 19 04:13:39.500515 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496619    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 19 04:13:39.500515 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496621    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 19 04:13:39.500515 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496624    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 19 04:13:39.500515 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496627    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 19 04:13:39.500515 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496629    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 19 04:13:39.500515 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496632    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 19 04:13:39.500515 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496635    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 19 04:13:39.500515 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496637    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 19 04:13:39.500515 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496640    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 19 04:13:39.500515 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496643    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 19 04:13:39.500515 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496645    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 19 04:13:39.500515 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496648    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 19 04:13:39.500515 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496652    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 19 04:13:39.500991 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496655    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 19 04:13:39.500991 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496658    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 19 04:13:39.500991 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496661    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 19 04:13:39.500991 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496664    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 19 04:13:39.500991 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496667    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 19 04:13:39.500991 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496669    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 19 04:13:39.500991 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496672    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 19 04:13:39.500991 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496674    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 19 04:13:39.500991 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496677    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 19 04:13:39.500991 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496679    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 19 04:13:39.500991 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496683    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 19 04:13:39.500991 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496686    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 19 04:13:39.500991 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496689    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 19 04:13:39.500991 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496691    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 19 04:13:39.500991 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496694    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 19 04:13:39.500991 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496696    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 19 04:13:39.500991 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496699    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 19 04:13:39.500991 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496702    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 19 04:13:39.500991 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496704    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 19 04:13:39.500991 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496706    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 19 04:13:39.501500 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496709    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 19 04:13:39.501500 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496711    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 19 04:13:39.501500 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496714    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 19 04:13:39.501500 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496716    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 19 04:13:39.501500 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496719    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 19 04:13:39.501500 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496721    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 19 04:13:39.501500 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:39.496724    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 19 04:13:39.501500 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.496729    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 19 04:13:39.501500 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.496844    2579 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 19 04:13:39.501500 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.499289    2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 19 04:13:39.501500 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.500438    2579 server.go:1019] "Starting client certificate rotation"
Sep 19 04:13:39.501500 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.500538    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 19 04:13:39.501500 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.500575    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 19 04:13:39.503392 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.503379    2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 19 04:13:39.504873 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.504856    2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 19 04:13:39.519419 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.519399    2579 log.go:25] "Validated CRI v1 runtime API"
Sep 19 04:13:39.521287 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.521270    2579 log.go:25] "Validated CRI v1 image API"
Sep 19 04:13:39.522178 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.522162    2579 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 19 04:13:39.526722 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.526704    2579 fs.go:135] Filesystem UUIDs: map[3bc0921b-0f27-449f-9c4b-9c1cd0cba9ff:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 dcd2c805-1bdc-4783-890a-f743707c8c4e:/dev/nvme0n1p4]
Sep 19 04:13:39.526776 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.526723    2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 19 04:13:39.531507 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.531393    2579 manager.go:216] Machine: {Timestamp:2026-09-19 04:13:39.530847309 +0000 UTC m=+0.356312129 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3122730 MemoryCapacity:32812154880 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a895e7425701e870c7c8870a9a1e0 SystemUUID:ec2a895e-7425-701e-870c-7c8870a9a1e0 BootID:231e7a10-0ab8-4f14-bad1-915a2934c847 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ec:c8:21:75 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ec:c8:21:75 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:3e:0b:29:1a:83:53 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812154880 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 19 04:13:39.531507 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.531502    2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 19 04:13:39.531630 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.531619    2579 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 19 04:13:39.535232 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.535205    2579 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 19 04:13:39.535394 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.535235    2579 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-215.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 19 04:13:39.536154 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.536144    2579 topology_manager.go:138] "Creating topology manager with none policy"
Sep 19 04:13:39.536204 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.536158    2579 container_manager_linux.go:309] "Creating device plugin manager"
Sep 19 04:13:39.536204 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.536170    2579 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 19 04:13:39.536204 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.536184    2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 19 04:13:39.536294 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.536270    2579 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 19 04:13:39.541557 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.541544    2579 state_mem.go:36] "Initialized new in-memory state store"
Sep 19 04:13:39.541687 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.541677    2579 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 19 04:13:39.541721 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.541712    2579 kubelet.go:486] "Attempting to sync node with API server"
Sep 19 04:13:39.541759 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.541723    2579 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 19 04:13:39.541759 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.541745    2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 19 04:13:39.541759 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.541757    2579 kubelet.go:398] "Adding apiserver pod source"
Sep 19 04:13:39.541864 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.541767    2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 19 04:13:39.542120 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.542109    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 19 04:13:39.542159 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.542128    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 19 04:13:39.544577 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.544561    2579 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 19 04:13:39.547567 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.547553    2579 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 19 04:13:39.547631 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.547592    2579 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 19 04:13:39.548904 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.548893    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 19 04:13:39.548937 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.548909    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 19 04:13:39.548937 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.548915    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 19 04:13:39.548937 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.548921    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 19 04:13:39.548937 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.548929    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 19 04:13:39.548937 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.548934    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 19 04:13:39.549070 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.548947    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 19 04:13:39.549070 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.548953    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 19 04:13:39.549070 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.548961    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 19 04:13:39.549070 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.548966    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 19 04:13:39.549070 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.548975    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 19 04:13:39.549070 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.548980    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 19 04:13:39.549900 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.549890    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 19 04:13:39.549936 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.549902    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 19 04:13:39.553786 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.553768    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 19 04:13:39.553928 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.553915    2579 server.go:1268] "Started kubelet"
Sep 19 04:13:39.554067 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.554002    2579 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 19 04:13:39.554127 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.554053    2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 19 04:13:39.554697 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.554679    2579 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 19 04:13:39.554911 ip-10-0-129-215 systemd[1]: Started Kubernetes Kubelet.
Sep 19 04:13:39.560133 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.560048    2579 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 19 04:13:39.561165 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.561129    2579 server.go:310] "Adding debug handlers to kubelet server"
Sep 19 04:13:39.561401 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:39.561372    2579 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 19 04:13:39.563588 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.563566    2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-215.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 19 04:13:39.563666 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:39.563586    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-215.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 19 04:13:39.563666 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:39.563631    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 19 04:13:39.567057 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:39.565866    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-215.ec2.internal.18d69da1a79f5a7f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-215.ec2.internal,UID:ip-10-0-129-215.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-215.ec2.internal,},FirstTimestamp:2026-09-19 04:13:39.553884799 +0000 UTC m=+0.379349619,LastTimestamp:2026-09-19 04:13:39.553884799 +0000 UTC m=+0.379349619,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-215.ec2.internal,}"
Sep 19 04:13:39.568373 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.568353    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 19 04:13:39.569079 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.569059    2579 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 19 04:13:39.569164 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:39.569148    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-215.ec2.internal\" not found"
Sep 19 04:13:39.569853 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.569829    2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 19 04:13:39.569942 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.569832    2579 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 19 04:13:39.569942 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.569870    2579 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 19 04:13:39.570224 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.569968    2579 reconstruct.go:97] "Volume reconstruction finished"
Sep 19 04:13:39.570224 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.569978    2579 reconciler.go:29] "Reconciler: start to sync state"
Sep 19 04:13:39.570224 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.569993    2579 factory.go:55] Registering systemd factory
Sep 19 04:13:39.570224 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.570006    2579 factory.go:223] Registration of the systemd container factory successfully
Sep 19 04:13:39.570224 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.570221    2579 factory.go:153] Registering CRI-O factory
Sep 19 04:13:39.570433 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.570231    2579 factory.go:223] Registration of the crio container factory successfully
Sep 19 04:13:39.570433 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.570271    2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 19 04:13:39.570433 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.570285    2579 factory.go:103] Registering Raw factory
Sep 19 04:13:39.570433 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.570296    2579 manager.go:1196] Started watching for new ooms in manager
Sep 19 04:13:39.570721 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.570704    2579 manager.go:349] Starting recovery of all containers
Sep 19 04:13:39.580566 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.580543    2579 manager.go:354] Recovery completed
Sep 19 04:13:39.582856 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:39.582830    2579 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-215.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 19 04:13:39.582981 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:39.582907    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 19 04:13:39.584918 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.584905    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 19 04:13:39.586795 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.586774    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-215.ec2.internal" event="NodeHasSufficientMemory"
Sep 19 04:13:39.586855 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.586812    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-215.ec2.internal" event="NodeHasNoDiskPressure"
Sep 19 04:13:39.586855 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.586833    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-215.ec2.internal" event="NodeHasSufficientPID"
Sep 19 04:13:39.587559 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.587542    2579 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 19 04:13:39.587626 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.587551    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 19 04:13:39.587626 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.587558    2579 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 19 04:13:39.587686 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.587626    2579 state_mem.go:36] "Initialized new in-memory state store"
Sep 19 04:13:39.588298 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.588285    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-215.ec2.internal" event="NodeHasSufficientMemory"
Sep 19 04:13:39.588344 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.588309    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-215.ec2.internal" event="NodeHasNoDiskPressure"
Sep 19 04:13:39.588344 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.588320    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-215.ec2.internal" event="NodeHasSufficientPID"
Sep 19 04:13:39.589821 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.589807    2579 policy_none.go:49] "None policy: Start"
Sep 19 04:13:39.589934 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.589827    2579 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 19 04:13:39.589934 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.589840    2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 19 04:13:39.591915 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.591901    2579 policy_none.go:47] "Start"
Sep 19 04:13:39.599824 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:39.599755    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-215.ec2.internal.18d69da1a995859d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-215.ec2.internal,UID:ip-10-0-129-215.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-215.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-215.ec2.internal,},FirstTimestamp:2026-09-19 04:13:39.586794909 +0000 UTC m=+0.412259729,LastTimestamp:2026-09-19 04:13:39.586794909 +0000 UTC m=+0.412259729,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-215.ec2.internal,}"
Sep 19 04:13:39.602101 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:39.602030    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-215.ec2.internal.18d69da1a995e215  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-215.ec2.internal,UID:ip-10-0-129-215.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-129-215.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-129-215.ec2.internal,},FirstTimestamp:2026-09-19 04:13:39.586818581 +0000 UTC m=+0.412283400,LastTimestamp:2026-09-19 04:13:39.586818581 +0000 UTC m=+0.412283400,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-215.ec2.internal,}"
Sep 19 04:13:39.610089 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:39.610007    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-215.ec2.internal.18d69da1a9962b2b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-215.ec2.internal,UID:ip-10-0-129-215.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-129-215.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-129-215.ec2.internal,},FirstTimestamp:2026-09-19 04:13:39.586837291 +0000 UTC m=+0.412302110,LastTimestamp:2026-09-19 04:13:39.586837291 +0000 UTC m=+0.412302110,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-215.ec2.internal,}"
Sep 19 04:13:39.612522 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:39.612448    2579 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-129-215.ec2.internal.18d69da1a995859d\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-215.ec2.internal.18d69da1a995859d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-215.ec2.internal,UID:ip-10-0-129-215.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-215.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-215.ec2.internal,},FirstTimestamp:2026-09-19 04:13:39.586794909 +0000 UTC m=+0.412259729,LastTimestamp:2026-09-19 04:13:39.588297293 +0000 UTC m=+0.413762113,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-215.ec2.internal,}"
Sep 19 04:13:39.619784 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:39.619718    2579 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-129-215.ec2.internal.18d69da1a995e215\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-215.ec2.internal.18d69da1a995e215  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-215.ec2.internal,UID:ip-10-0-129-215.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-129-215.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-129-215.ec2.internal,},FirstTimestamp:2026-09-19 04:13:39.586818581 +0000 UTC m=+0.412283400,LastTimestamp:2026-09-19 04:13:39.588314261 +0000 UTC m=+0.413779081,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-215.ec2.internal,}"
Sep 19 04:13:39.624815 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.624801    2579 manager.go:337] "Starting Device Plugin manager"
Sep 19 04:13:39.648657 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:39.624847    2579 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 19 04:13:39.648657 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.624861    2579 server.go:87] "Starting device plugin registration server"
Sep 19 04:13:39.648657 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.625129    2579 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 19 04:13:39.648657 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.625142    2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 19 04:13:39.648657 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.625309    2579 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 19 04:13:39.648657 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.625413    2579 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 19 04:13:39.648657 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.625422    2579 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 19 04:13:39.648657 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:39.625898    2579 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 19 04:13:39.648657 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:39.625938    2579 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-215.ec2.internal\" not found"
Sep 19 04:13:39.648657 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:39.631323    2579 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-129-215.ec2.internal.18d69da1a9962b2b\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-215.ec2.internal.18d69da1a9962b2b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-215.ec2.internal,UID:ip-10-0-129-215.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-129-215.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-129-215.ec2.internal,},FirstTimestamp:2026-09-19 04:13:39.586837291 +0000 UTC m=+0.412302110,LastTimestamp:2026-09-19 04:13:39.588323781 +0000 UTC m=+0.413788600,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-215.ec2.internal,}"
Sep 19 04:13:39.648657 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:39.645342    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-215.ec2.internal.18d69da1abfa1e8b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-215.ec2.internal,UID:ip-10-0-129-215.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-129-215.ec2.internal,},FirstTimestamp:2026-09-19 04:13:39.626942091 +0000 UTC m=+0.452406906,LastTimestamp:2026-09-19 04:13:39.626942091 +0000 UTC m=+0.452406906,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-215.ec2.internal,}"
Sep 19 04:13:39.648657 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.647061    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wt6kk"
Sep 19 04:13:39.662564 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.662543    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wt6kk"
Sep 19 04:13:39.700699 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.700611    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 19 04:13:39.701966 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.701951    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 19 04:13:39.702031 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.702001    2579 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 19 04:13:39.702031 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.702026    2579 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 19 04:13:39.702107 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:39.702067    2579 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 19 04:13:39.712057 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.712033    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 19 04:13:39.725482 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.725470    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 19 04:13:39.726366 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.726350    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-215.ec2.internal" event="NodeHasSufficientMemory"
Sep 19 04:13:39.726429 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.726380    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-215.ec2.internal" event="NodeHasNoDiskPressure"
Sep 19 04:13:39.726429 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.726390    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-215.ec2.internal" event="NodeHasSufficientPID"
Sep 19 04:13:39.726429 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.726411    2579 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-215.ec2.internal"
Sep 19 04:13:39.745185 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.745163    2579 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-215.ec2.internal"
Sep 19 04:13:39.745273 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:39.745203    2579 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-215.ec2.internal\": node \"ip-10-0-129-215.ec2.internal\" not found"
Sep 19 04:13:39.776298 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:39.776264    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-215.ec2.internal\" not found"
Sep 19 04:13:39.802605 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.802581    2579 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-215.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-215.ec2.internal"]
Sep 19 04:13:39.802664 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.802646    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 19 04:13:39.804079 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.804064    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-215.ec2.internal" event="NodeHasSufficientMemory"
Sep 19 04:13:39.804149 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.804095    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-215.ec2.internal" event="NodeHasNoDiskPressure"
Sep 19 04:13:39.804149 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.804105    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-215.ec2.internal" event="NodeHasSufficientPID"
Sep 19 04:13:39.806414 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.806402    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 19 04:13:39.806537 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.806524    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-215.ec2.internal"
Sep 19 04:13:39.806570 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.806559    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 19 04:13:39.807155 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.807137    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-215.ec2.internal" event="NodeHasSufficientMemory"
Sep 19 04:13:39.807251 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.807169    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-215.ec2.internal" event="NodeHasNoDiskPressure"
Sep 19 04:13:39.807251 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.807179    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-215.ec2.internal" event="NodeHasSufficientPID"
Sep 19 04:13:39.807251 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.807145    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-215.ec2.internal" event="NodeHasSufficientMemory"
Sep 19 04:13:39.807251 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.807225    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-215.ec2.internal" event="NodeHasNoDiskPressure"
Sep 19 04:13:39.807251 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.807235    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-215.ec2.internal" event="NodeHasSufficientPID"
Sep 19 04:13:39.809530 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.809512    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-215.ec2.internal"
Sep 19 04:13:39.809579 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.809546    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 19 04:13:39.810306 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.810284    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-215.ec2.internal" event="NodeHasSufficientMemory"
Sep 19 04:13:39.810405 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.810318    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-215.ec2.internal" event="NodeHasNoDiskPressure"
Sep 19 04:13:39.810405 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.810334    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-215.ec2.internal" event="NodeHasSufficientPID"
Sep 19 04:13:39.835822 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:39.835801    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-215.ec2.internal\" not found" node="ip-10-0-129-215.ec2.internal"
Sep 19 04:13:39.840275 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:39.840261    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-215.ec2.internal\" not found" node="ip-10-0-129-215.ec2.internal"
Sep 19 04:13:39.872548 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.872515    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/56d744041b896c3555c7522e205ed4d0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-215.ec2.internal\" (UID: \"56d744041b896c3555c7522e205ed4d0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-215.ec2.internal"
Sep 19 04:13:39.872661 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.872563    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/56d744041b896c3555c7522e205ed4d0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-215.ec2.internal\" (UID: \"56d744041b896c3555c7522e205ed4d0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-215.ec2.internal"
Sep 19 04:13:39.872661 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.872586    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5cccb5a09b0064784f816095b5ca12fb-config\") pod \"kube-apiserver-proxy-ip-10-0-129-215.ec2.internal\" (UID: \"5cccb5a09b0064784f816095b5ca12fb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-215.ec2.internal"
Sep 19 04:13:39.876640 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:39.876617    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-215.ec2.internal\" not found"
Sep 19 04:13:39.973808 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.973664    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/56d744041b896c3555c7522e205ed4d0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-215.ec2.internal\" (UID: \"56d744041b896c3555c7522e205ed4d0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-215.ec2.internal"
Sep 19 04:13:39.973808 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.973750    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/56d744041b896c3555c7522e205ed4d0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-215.ec2.internal\" (UID: \"56d744041b896c3555c7522e205ed4d0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-215.ec2.internal"
Sep 19 04:13:39.973976 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.973790    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/56d744041b896c3555c7522e205ed4d0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-215.ec2.internal\" (UID: \"56d744041b896c3555c7522e205ed4d0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-215.ec2.internal"
Sep 19 04:13:39.973976 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.973850    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5cccb5a09b0064784f816095b5ca12fb-config\") pod \"kube-apiserver-proxy-ip-10-0-129-215.ec2.internal\" (UID: \"5cccb5a09b0064784f816095b5ca12fb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-215.ec2.internal"
Sep 19 04:13:39.973976 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.973819    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/56d744041b896c3555c7522e205ed4d0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-215.ec2.internal\" (UID: \"56d744041b896c3555c7522e205ed4d0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-215.ec2.internal"
Sep 19 04:13:39.973976 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:39.973913    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5cccb5a09b0064784f816095b5ca12fb-config\") pod \"kube-apiserver-proxy-ip-10-0-129-215.ec2.internal\" (UID: \"5cccb5a09b0064784f816095b5ca12fb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-215.ec2.internal"
Sep 19 04:13:39.976979 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:39.976963    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-215.ec2.internal\" not found"
Sep 19 04:13:40.077902 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:40.077865    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-215.ec2.internal\" not found"
Sep 19 04:13:40.138999 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:40.138969    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-215.ec2.internal"
Sep 19 04:13:40.144675 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:40.144383    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-215.ec2.internal"
Sep 19 04:13:40.178594 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:40.178549    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-215.ec2.internal\" not found"
Sep 19 04:13:40.279231 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:40.279131    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-215.ec2.internal\" not found"
Sep 19 04:13:40.379664 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:40.379629    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-215.ec2.internal\" not found"
Sep 19 04:13:40.480256 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:40.480216    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-215.ec2.internal\" not found"
Sep 19 04:13:40.500385 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:40.500355    2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 19 04:13:40.500518 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:40.500506    2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 19 04:13:40.580670 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:40.580500    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-215.ec2.internal\" not found"
Sep 19 04:13:40.621517 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:40.621491    2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 19 04:13:40.668273 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:40.668230    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-18 04:08:41 +0000 UTC" deadline="2028-06-30 09:58:40.43337807 +0000 UTC"
Sep 19 04:13:40.668273 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:40.668266    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15605h44m59.765115327s"
Sep 19 04:13:40.681635 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:40.681614    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-215.ec2.internal\" not found"
Sep 19 04:13:40.695301 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:40.695261    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cccb5a09b0064784f816095b5ca12fb.slice/crio-6c66028bb1599cad85969f80f7a28284cc9a40ce3bb8477d44383f815af0b3be WatchSource:0}: Error finding container 6c66028bb1599cad85969f80f7a28284cc9a40ce3bb8477d44383f815af0b3be: Status 404 returned error can't find the container with id 6c66028bb1599cad85969f80f7a28284cc9a40ce3bb8477d44383f815af0b3be
Sep 19 04:13:40.696067 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:40.696046    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56d744041b896c3555c7522e205ed4d0.slice/crio-9ab03cc8d0d9fbd6afa529ed20cb4272d2b0cd44b920b85bdea01f3a0adf2730 WatchSource:0}: Error finding container 9ab03cc8d0d9fbd6afa529ed20cb4272d2b0cd44b920b85bdea01f3a0adf2730: Status 404 returned error can't find the container with id 9ab03cc8d0d9fbd6afa529ed20cb4272d2b0cd44b920b85bdea01f3a0adf2730
Sep 19 04:13:40.700794 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:40.700779    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 19 04:13:40.705799 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:40.705761    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-215.ec2.internal" event={"ID":"56d744041b896c3555c7522e205ed4d0","Type":"ContainerStarted","Data":"9ab03cc8d0d9fbd6afa529ed20cb4272d2b0cd44b920b85bdea01f3a0adf2730"}
Sep 19 04:13:40.706989 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:40.706968    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-215.ec2.internal" event={"ID":"5cccb5a09b0064784f816095b5ca12fb","Type":"ContainerStarted","Data":"6c66028bb1599cad85969f80f7a28284cc9a40ce3bb8477d44383f815af0b3be"}
Sep 19 04:13:40.734116 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:40.734092    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 19 04:13:40.735159 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:40.735144    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-215.ec2.internal" event="NodeHasSufficientMemory"
Sep 19 04:13:40.735258 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:40.735171    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-215.ec2.internal" event="NodeHasNoDiskPressure"
Sep 19 04:13:40.735258 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:40.735181    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-215.ec2.internal" event="NodeHasSufficientPID"
Sep 19 04:13:40.781913 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:40.781869    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-215.ec2.internal\" not found"
Sep 19 04:13:40.788347 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:40.788324    2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 19 04:13:40.869660 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:40.869591    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-215.ec2.internal"
Sep 19 04:13:40.887602 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:40.887580    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 19 04:13:40.888274 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:40.888262    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-215.ec2.internal"
Sep 19 04:13:40.930324 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:40.930305    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 19 04:13:42.718621 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:42.718595    2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 19 04:13:43.185685 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.185658    2579 apiserver.go:52] "Watching apiserver"
Sep 19 04:13:43.192964 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.192938    2579 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 19 04:13:43.194730 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.194708    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gnm2x","openshift-cluster-node-tuning-operator/tuned-dxttg","openshift-dns/node-resolver-2ptq2","openshift-image-registry/node-ca-5bzkh","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-215.ec2.internal","openshift-multus/multus-nwgjm","openshift-multus/network-metrics-daemon-wmk74","openshift-network-diagnostics/network-check-target-z7dgk","kube-system/global-pull-secret-syncer-5t728","kube-system/konnectivity-agent-szpwc","kube-system/kube-apiserver-proxy-ip-10-0-129-215.ec2.internal","openshift-multus/multus-additional-cni-plugins-wzbjr","openshift-network-operator/iptables-alerter-kcb82","openshift-ovn-kubernetes/ovnkube-node-rtwn6"]
Sep 19 04:13:43.197212 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.197187    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wmk74"
Sep 19 04:13:43.197312 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:43.197275    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wmk74" podUID="3738f1c7-67af-4201-8e8d-1d0406e0edb9"
Sep 19 04:13:43.199353 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.199335    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7dgk"
Sep 19 04:13:43.199430 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:43.199406    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7dgk" podUID="f2fe8238-2a1d-4e7a-b4d7-4cc7d518f366"
Sep 19 04:13:43.201483 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.201462    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2ptq2"
Sep 19 04:13:43.203627 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.203610    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5bzkh"
Sep 19 04:13:43.203962 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.203941    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 19 04:13:43.204084 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.204021    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-rf9c9\""
Sep 19 04:13:43.204329 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.204312    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 19 04:13:43.205966 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.205944    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gnm2x"
Sep 19 04:13:43.208248 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.208224    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-nwgjm"
Sep 19 04:13:43.208941 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.208771    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9xfg7\""
Sep 19 04:13:43.208941 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.208910    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 19 04:13:43.209061 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.208977    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 19 04:13:43.209116 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.209095    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 19 04:13:43.209216 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.209199    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 19 04:13:43.209277 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.209238    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 19 04:13:43.209332 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.209293    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-xhwtw\""
Sep 19 04:13:43.209369 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.209357    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 19 04:13:43.210611 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.210595    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5t728"
Sep 19 04:13:43.210682 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:43.210659    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5t728" podUID="64cb49fc-ede9-4369-aa8b-76a95d1edab6"
Sep 19 04:13:43.210767 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.210734    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 19 04:13:43.210767 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.210739    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-hsjsd\""
Sep 19 04:13:43.210923 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.210908    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 19 04:13:43.212309 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.212293    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 19 04:13:43.213439 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.213414    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 19 04:13:43.213555 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.213542    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kcb82"
Sep 19 04:13:43.213672 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.213634    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 19 04:13:43.215909 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.215893    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dxttg"
Sep 19 04:13:43.216434 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.216415    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 19 04:13:43.216590 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.216512    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-xh47w\""
Sep 19 04:13:43.216685 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.216641    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 19 04:13:43.217173 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.217158    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 19 04:13:43.218592 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.218572    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-szpwc"
Sep 19 04:13:43.218717 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.218645    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 19 04:13:43.219041 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.219009    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 19 04:13:43.219134 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.219061    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-tvn8m\""
Sep 19 04:13:43.219750 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.219720    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 19 04:13:43.222719 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.221613    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rtwn6"
Sep 19 04:13:43.222719 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.221613    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 19 04:13:43.224603 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.223189    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 19 04:13:43.224603 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.224294    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 19 04:13:43.224797 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.224759    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-5hxrf\""
Sep 19 04:13:43.226308 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.226283    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5194aa7-dba0-46dc-b3e9-91a4ea69e79e-tmp-dir\") pod \"node-resolver-2ptq2\" (UID: \"a5194aa7-dba0-46dc-b3e9-91a4ea69e79e\") " pod="openshift-dns/node-resolver-2ptq2"
Sep 19 04:13:43.226400 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.226326    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/64c734ac-0f91-4e8b-83e9-a2e9307b3bfe-device-dir\") pod \"aws-ebs-csi-driver-node-gnm2x\" (UID: \"64c734ac-0f91-4e8b-83e9-a2e9307b3bfe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gnm2x"
Sep 19 04:13:43.226400 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.226353    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bc229f7a-0722-442f-940c-7ec542b81888-host-run-k8s-cni-cncf-io\") pod \"multus-nwgjm\" (UID: \"bc229f7a-0722-442f-940c-7ec542b81888\") " pod="openshift-multus/multus-nwgjm"
Sep 19 04:13:43.226400 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.226377    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4ece1d13-8010-47bb-8e97-40634dd06c5d-host-slash\") pod \"iptables-alerter-kcb82\" (UID: \"4ece1d13-8010-47bb-8e97-40634dd06c5d\") " pod="openshift-network-operator/iptables-alerter-kcb82"
Sep 19 04:13:43.226564 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.226411    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a8d1a2a7-a76e-4741-be3a-d87124254f5d-serviceca\") pod \"node-ca-5bzkh\" (UID: \"a8d1a2a7-a76e-4741-be3a-d87124254f5d\") " pod="openshift-image-registry/node-ca-5bzkh"
Sep 19 04:13:43.226564 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.226417    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wzbjr"
Sep 19 04:13:43.226564 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.226435    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/64c734ac-0f91-4e8b-83e9-a2e9307b3bfe-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gnm2x\" (UID: \"64c734ac-0f91-4e8b-83e9-a2e9307b3bfe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gnm2x"
Sep 19 04:13:43.226564 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.226457    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/64c734ac-0f91-4e8b-83e9-a2e9307b3bfe-registration-dir\") pod \"aws-ebs-csi-driver-node-gnm2x\" (UID: \"64c734ac-0f91-4e8b-83e9-a2e9307b3bfe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gnm2x"
Sep 19 04:13:43.226564 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.226480    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqmvn\" (UniqueName: \"kubernetes.io/projected/a5194aa7-dba0-46dc-b3e9-91a4ea69e79e-kube-api-access-jqmvn\") pod \"node-resolver-2ptq2\" (UID: \"a5194aa7-dba0-46dc-b3e9-91a4ea69e79e\") " pod="openshift-dns/node-resolver-2ptq2"
Sep 19 04:13:43.226564 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.226505    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bc229f7a-0722-442f-940c-7ec542b81888-cnibin\") pod \"multus-nwgjm\" (UID: \"bc229f7a-0722-442f-940c-7ec542b81888\") " pod="openshift-multus/multus-nwgjm"
Sep 19 04:13:43.226564 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.226527    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/64cb49fc-ede9-4369-aa8b-76a95d1edab6-kubelet-config\") pod \"global-pull-secret-syncer-5t728\" (UID: \"64cb49fc-ede9-4369-aa8b-76a95d1edab6\") " pod="kube-system/global-pull-secret-syncer-5t728"
Sep 19 04:13:43.226564 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.226548    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/64cb49fc-ede9-4369-aa8b-76a95d1edab6-dbus\") pod \"global-pull-secret-syncer-5t728\" (UID: \"64cb49fc-ede9-4369-aa8b-76a95d1edab6\") " pod="kube-system/global-pull-secret-syncer-5t728"
Sep 19 04:13:43.226975 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.226571    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7mbf\" (UniqueName: \"kubernetes.io/projected/64c734ac-0f91-4e8b-83e9-a2e9307b3bfe-kube-api-access-m7mbf\") pod \"aws-ebs-csi-driver-node-gnm2x\" (UID: \"64c734ac-0f91-4e8b-83e9-a2e9307b3bfe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gnm2x"
Sep 19 04:13:43.226975 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.226594    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a5194aa7-dba0-46dc-b3e9-91a4ea69e79e-hosts-file\") pod \"node-resolver-2ptq2\" (UID: \"a5194aa7-dba0-46dc-b3e9-91a4ea69e79e\") " pod="openshift-dns/node-resolver-2ptq2"
Sep 19 04:13:43.226975 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.226674    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bc229f7a-0722-442f-940c-7ec542b81888-multus-conf-dir\") pod \"multus-nwgjm\" (UID: \"bc229f7a-0722-442f-940c-7ec542b81888\") " pod="openshift-multus/multus-nwgjm"
Sep 19 04:13:43.226975 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.226723    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/64cb49fc-ede9-4369-aa8b-76a95d1edab6-original-pull-secret\") pod \"global-pull-secret-syncer-5t728\" (UID: \"64cb49fc-ede9-4369-aa8b-76a95d1edab6\") " pod="kube-system/global-pull-secret-syncer-5t728"
Sep 19 04:13:43.226975 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.226744    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4ece1d13-8010-47bb-8e97-40634dd06c5d-iptables-alerter-script\") pod \"iptables-alerter-kcb82\" (UID: \"4ece1d13-8010-47bb-8e97-40634dd06c5d\") " pod="openshift-network-operator/iptables-alerter-kcb82"
Sep 19 04:13:43.226975 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.226761    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtktn\" (UniqueName: \"kubernetes.io/projected/a8d1a2a7-a76e-4741-be3a-d87124254f5d-kube-api-access-dtktn\") pod \"node-ca-5bzkh\" (UID: \"a8d1a2a7-a76e-4741-be3a-d87124254f5d\") " pod="openshift-image-registry/node-ca-5bzkh"
Sep 19 04:13:43.226975 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.226777    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhhsm\" (UniqueName: \"kubernetes.io/projected/3738f1c7-67af-4201-8e8d-1d0406e0edb9-kube-api-access-dhhsm\") pod \"network-metrics-daemon-wmk74\" (UID: \"3738f1c7-67af-4201-8e8d-1d0406e0edb9\") " pod="openshift-multus/network-metrics-daemon-wmk74"
Sep 19 04:13:43.226975 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.226801    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bc229f7a-0722-442f-940c-7ec542b81888-etc-kubernetes\") pod \"multus-nwgjm\" (UID: \"bc229f7a-0722-442f-940c-7ec542b81888\") " pod="openshift-multus/multus-nwgjm"
Sep 19 04:13:43.226975 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.226839    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx5h4\" (UniqueName: \"kubernetes.io/projected/bc229f7a-0722-442f-940c-7ec542b81888-kube-api-access-jx5h4\") pod \"multus-nwgjm\" (UID: \"bc229f7a-0722-442f-940c-7ec542b81888\") " pod="openshift-multus/multus-nwgjm"
Sep 19 04:13:43.226975 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.226888    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bc229f7a-0722-442f-940c-7ec542b81888-host-var-lib-cni-multus\") pod \"multus-nwgjm\" (UID: \"bc229f7a-0722-442f-940c-7ec542b81888\") " pod="openshift-multus/multus-nwgjm"
Sep 19 04:13:43.226975 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.226910    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bc229f7a-0722-442f-940c-7ec542b81888-hostroot\") pod \"multus-nwgjm\" (UID: \"bc229f7a-0722-442f-940c-7ec542b81888\") " pod="openshift-multus/multus-nwgjm"
Sep 19 04:13:43.226975 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.226933    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bc229f7a-0722-442f-940c-7ec542b81888-multus-daemon-config\") pod \"multus-nwgjm\" (UID: \"bc229f7a-0722-442f-940c-7ec542b81888\") " pod="openshift-multus/multus-nwgjm"
Sep 19 04:13:43.226975 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.226959    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bc229f7a-0722-442f-940c-7ec542b81888-host-run-multus-certs\") pod \"multus-nwgjm\" (UID: \"bc229f7a-0722-442f-940c-7ec542b81888\") " pod="openshift-multus/multus-nwgjm"
Sep 19 04:13:43.227568 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.226983    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bc229f7a-0722-442f-940c-7ec542b81888-cni-binary-copy\") pod \"multus-nwgjm\" (UID: \"bc229f7a-0722-442f-940c-7ec542b81888\") " pod="openshift-multus/multus-nwgjm"
Sep 19 04:13:43.227568 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.227004    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bc229f7a-0722-442f-940c-7ec542b81888-multus-cni-dir\") pod \"multus-nwgjm\" (UID: \"bc229f7a-0722-442f-940c-7ec542b81888\") " pod="openshift-multus/multus-nwgjm"
Sep 19 04:13:43.227568 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.227018    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bc229f7a-0722-442f-940c-7ec542b81888-host-var-lib-cni-bin\") pod \"multus-nwgjm\" (UID: \"bc229f7a-0722-442f-940c-7ec542b81888\") " pod="openshift-multus/multus-nwgjm"
Sep 19 04:13:43.227568 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.227054    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-625bx\" (UniqueName: \"kubernetes.io/projected/4ece1d13-8010-47bb-8e97-40634dd06c5d-kube-api-access-625bx\") pod \"iptables-alerter-kcb82\" (UID: \"4ece1d13-8010-47bb-8e97-40634dd06c5d\") " pod="openshift-network-operator/iptables-alerter-kcb82"
Sep 19 04:13:43.227568 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.227078    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/64c734ac-0f91-4e8b-83e9-a2e9307b3bfe-etc-selinux\") pod \"aws-ebs-csi-driver-node-gnm2x\" (UID: \"64c734ac-0f91-4e8b-83e9-a2e9307b3bfe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gnm2x"
Sep 19 04:13:43.227568 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.227098    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bc229f7a-0722-442f-940c-7ec542b81888-system-cni-dir\") pod \"multus-nwgjm\" (UID: \"bc229f7a-0722-442f-940c-7ec542b81888\") " pod="openshift-multus/multus-nwgjm"
Sep 19 04:13:43.227568 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.227118    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bc229f7a-0722-442f-940c-7ec542b81888-host-run-netns\") pod \"multus-nwgjm\" (UID: \"bc229f7a-0722-442f-940c-7ec542b81888\") " pod="openshift-multus/multus-nwgjm"
Sep 19 04:13:43.227568 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.227132    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bc229f7a-0722-442f-940c-7ec542b81888-host-var-lib-kubelet\") pod \"multus-nwgjm\" (UID: \"bc229f7a-0722-442f-940c-7ec542b81888\") " pod="openshift-multus/multus-nwgjm"
Sep 19 04:13:43.227568 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.227144    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a8d1a2a7-a76e-4741-be3a-d87124254f5d-host\") pod \"node-ca-5bzkh\" (UID: \"a8d1a2a7-a76e-4741-be3a-d87124254f5d\") " pod="openshift-image-registry/node-ca-5bzkh"
Sep 19 04:13:43.227568 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.227157    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/64c734ac-0f91-4e8b-83e9-a2e9307b3bfe-socket-dir\") pod \"aws-ebs-csi-driver-node-gnm2x\" (UID: \"64c734ac-0f91-4e8b-83e9-a2e9307b3bfe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gnm2x"
Sep 19 04:13:43.227568 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.227184    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/64c734ac-0f91-4e8b-83e9-a2e9307b3bfe-sys-fs\") pod \"aws-ebs-csi-driver-node-gnm2x\" (UID: \"64c734ac-0f91-4e8b-83e9-a2e9307b3bfe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gnm2x"
Sep 19 04:13:43.227568 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.227210    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bc229f7a-0722-442f-940c-7ec542b81888-multus-socket-dir-parent\") pod \"multus-nwgjm\" (UID: \"bc229f7a-0722-442f-940c-7ec542b81888\") " pod="openshift-multus/multus-nwgjm"
Sep 19 04:13:43.227568 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.227230    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3738f1c7-67af-4201-8e8d-1d0406e0edb9-metrics-certs\") pod \"network-metrics-daemon-wmk74\" (UID: \"3738f1c7-67af-4201-8e8d-1d0406e0edb9\") " pod="openshift-multus/network-metrics-daemon-wmk74"
Sep 19 04:13:43.227568 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.227249    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j856z\" (UniqueName: \"kubernetes.io/projected/f2fe8238-2a1d-4e7a-b4d7-4cc7d518f366-kube-api-access-j856z\") pod \"network-check-target-z7dgk\" (UID: \"f2fe8238-2a1d-4e7a-b4d7-4cc7d518f366\") " pod="openshift-network-diagnostics/network-check-target-z7dgk"
Sep 19 04:13:43.228264 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.228002    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 19 04:13:43.228264 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.228016    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 19 04:13:43.228264 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.228006    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 19 04:13:43.228264 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.228000    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 19 04:13:43.228264 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.228098    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7xx4p\""
Sep 19 04:13:43.228501 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.228319    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 19 04:13:43.228501 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.228321    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 19 04:13:43.229314 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.229296    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 19 04:13:43.229524 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.229507    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 19 04:13:43.229811 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.229795    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-c5xfx\""
Sep 19 04:13:43.269345 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.269289    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-5fkh7"
Sep 19 04:13:43.280356 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.280323    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-5fkh7"
Sep 19 04:13:43.313949 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.313918    2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 19 04:13:43.328403 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.328374    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82js7\" (UniqueName: \"kubernetes.io/projected/1beab173-cbd6-4858-a590-c08e2d521ed9-kube-api-access-82js7\") pod \"ovnkube-node-rtwn6\" (UID: \"1beab173-cbd6-4858-a590-c08e2d521ed9\") " pod="openshift-ovn-kubernetes/ovnkube-node-rtwn6"
Sep 19 04:13:43.328581 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.328412    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bcd5da2a-a24f-40fa-80aa-03ef54cc22a6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wzbjr\" (UID: \"bcd5da2a-a24f-40fa-80aa-03ef54cc22a6\") " pod="openshift-multus/multus-additional-cni-plugins-wzbjr"
Sep 19 04:13:43.328581 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.328444    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/64c734ac-0f91-4e8b-83e9-a2e9307b3bfe-registration-dir\") pod \"aws-ebs-csi-driver-node-gnm2x\" (UID: \"64c734ac-0f91-4e8b-83e9-a2e9307b3bfe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gnm2x"
Sep 19 04:13:43.328581 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.328495    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/84819fb3-13e8-482a-b88e-c30a2ef0bc3d-lib-modules\") pod \"tuned-dxttg\" (UID: \"84819fb3-13e8-482a-b88e-c30a2ef0bc3d\") " pod="openshift-cluster-node-tuning-operator/tuned-dxttg"
Sep 19 04:13:43.328581 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.328530    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/64c734ac-0f91-4e8b-83e9-a2e9307b3bfe-registration-dir\") pod \"aws-ebs-csi-driver-node-gnm2x\" (UID: \"64c734ac-0f91-4e8b-83e9-a2e9307b3bfe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gnm2x"
Sep 19 04:13:43.328581 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.328535    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1beab173-cbd6-4858-a590-c08e2d521ed9-ovnkube-script-lib\") pod \"ovnkube-node-rtwn6\" (UID: \"1beab173-cbd6-4858-a590-c08e2d521ed9\") " pod="openshift-ovn-kubernetes/ovnkube-node-rtwn6"
Sep 19 04:13:43.328581 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.328569    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqmvn\" (UniqueName: \"kubernetes.io/projected/a5194aa7-dba0-46dc-b3e9-91a4ea69e79e-kube-api-access-jqmvn\") pod \"node-resolver-2ptq2\" (UID: \"a5194aa7-dba0-46dc-b3e9-91a4ea69e79e\") " pod="openshift-dns/node-resolver-2ptq2"
Sep 19 04:13:43.328907 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.328608    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bc229f7a-0722-442f-940c-7ec542b81888-cnibin\") pod \"multus-nwgjm\" (UID: \"bc229f7a-0722-442f-940c-7ec542b81888\") " pod="openshift-multus/multus-nwgjm"
Sep 19 04:13:43.328907 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.328653    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/64cb49fc-ede9-4369-aa8b-76a95d1edab6-kubelet-config\") pod \"global-pull-secret-syncer-5t728\" (UID: \"64cb49fc-ede9-4369-aa8b-76a95d1edab6\") " pod="kube-system/global-pull-secret-syncer-5t728"
Sep 19 04:13:43.328907 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.328681    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/64cb49fc-ede9-4369-aa8b-76a95d1edab6-dbus\") pod \"global-pull-secret-syncer-5t728\" (UID: \"64cb49fc-ede9-4369-aa8b-76a95d1edab6\") " pod="kube-system/global-pull-secret-syncer-5t728"
Sep 19 04:13:43.328907 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.328732    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f4b6c76b-7816-4e8b-9904-2f0cb5faca57-konnectivity-ca\") pod \"konnectivity-agent-szpwc\" (UID: \"f4b6c76b-7816-4e8b-9904-2f0cb5faca57\") " pod="kube-system/konnectivity-agent-szpwc"
Sep 19 04:13:43.328907 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.328760    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1beab173-cbd6-4858-a590-c08e2d521ed9-systemd-units\") pod \"ovnkube-node-rtwn6\" (UID: \"1beab173-cbd6-4858-a590-c08e2d521ed9\") " pod="openshift-ovn-kubernetes/ovnkube-node-rtwn6"
Sep 19 04:13:43.328907 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.328762    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bc229f7a-0722-442f-940c-7ec542b81888-cnibin\") pod \"multus-nwgjm\" (UID: \"bc229f7a-0722-442f-940c-7ec542b81888\") " pod="openshift-multus/multus-nwgjm"
Sep 19 04:13:43.328907 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.328799    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1beab173-cbd6-4858-a590-c08e2d521ed9-etc-openvswitch\") pod \"ovnkube-node-rtwn6\" (UID: \"1beab173-cbd6-4858-a590-c08e2d521ed9\") " pod="openshift-ovn-kubernetes/ovnkube-node-rtwn6"
Sep 19 04:13:43.328907 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.328820    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1beab173-cbd6-4858-a590-c08e2d521ed9-log-socket\") pod \"ovnkube-node-rtwn6\" (UID: \"1beab173-cbd6-4858-a590-c08e2d521ed9\") " pod="openshift-ovn-kubernetes/ovnkube-node-rtwn6"
Sep 19 04:13:43.328907 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.328831    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/64cb49fc-ede9-4369-aa8b-76a95d1edab6-kubelet-config\") pod \"global-pull-secret-syncer-5t728\" (UID: \"64cb49fc-ede9-4369-aa8b-76a95d1edab6\") " pod="kube-system/global-pull-secret-syncer-5t728"
Sep 19 04:13:43.328907 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.328848    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a5194aa7-dba0-46dc-b3e9-91a4ea69e79e-hosts-file\") pod \"node-resolver-2ptq2\" (UID: \"a5194aa7-dba0-46dc-b3e9-91a4ea69e79e\") " pod="openshift-dns/node-resolver-2ptq2"
Sep 19 04:13:43.328907 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.328871    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bc229f7a-0722-442f-940c-7ec542b81888-multus-conf-dir\") pod \"multus-nwgjm\" (UID: \"bc229f7a-0722-442f-940c-7ec542b81888\") " pod="openshift-multus/multus-nwgjm"
Sep 19 04:13:43.328907 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.328909    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtktn\" (UniqueName: \"kubernetes.io/projected/a8d1a2a7-a76e-4741-be3a-d87124254f5d-kube-api-access-dtktn\") pod \"node-ca-5bzkh\" (UID: \"a8d1a2a7-a76e-4741-be3a-d87124254f5d\") " pod="openshift-image-registry/node-ca-5bzkh"
Sep 19 04:13:43.329485 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.328935    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/84819fb3-13e8-482a-b88e-c30a2ef0bc3d-etc-sysctl-d\") pod \"tuned-dxttg\" (UID: \"84819fb3-13e8-482a-b88e-c30a2ef0bc3d\") " pod="openshift-cluster-node-tuning-operator/tuned-dxttg"
Sep 19 04:13:43.329485 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.328939    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a5194aa7-dba0-46dc-b3e9-91a4ea69e79e-hosts-file\") pod \"node-resolver-2ptq2\" (UID: \"a5194aa7-dba0-46dc-b3e9-91a4ea69e79e\") " pod="openshift-dns/node-resolver-2ptq2"
Sep 19 04:13:43.329485 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.328959    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/84819fb3-13e8-482a-b88e-c30a2ef0bc3d-host\") pod \"tuned-dxttg\" (UID: \"84819fb3-13e8-482a-b88e-c30a2ef0bc3d\") " pod="openshift-cluster-node-tuning-operator/tuned-dxttg"
Sep 19 04:13:43.329485 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.329001    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bc229f7a-0722-442f-940c-7ec542b81888-multus-conf-dir\") pod \"multus-nwgjm\" (UID: \"bc229f7a-0722-442f-940c-7ec542b81888\") " pod="openshift-multus/multus-nwgjm"
Sep 19 04:13:43.329485 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.329001    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjp4w\" (UniqueName: \"kubernetes.io/projected/84819fb3-13e8-482a-b88e-c30a2ef0bc3d-kube-api-access-wjp4w\") pod \"tuned-dxttg\" (UID: \"84819fb3-13e8-482a-b88e-c30a2ef0bc3d\") " pod="openshift-cluster-node-tuning-operator/tuned-dxttg"
Sep 19 04:13:43.329485 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.329033    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhhsm\" (UniqueName: \"kubernetes.io/projected/3738f1c7-67af-4201-8e8d-1d0406e0edb9-kube-api-access-dhhsm\") pod \"network-metrics-daemon-wmk74\" (UID: \"3738f1c7-67af-4201-8e8d-1d0406e0edb9\") " pod="openshift-multus/network-metrics-daemon-wmk74"
Sep 19 04:13:43.329485 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.329064    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/84819fb3-13e8-482a-b88e-c30a2ef0bc3d-run\") pod \"tuned-dxttg\" (UID: \"84819fb3-13e8-482a-b88e-c30a2ef0bc3d\") " pod="openshift-cluster-node-tuning-operator/tuned-dxttg"
Sep 19 04:13:43.329485 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.329066    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/64cb49fc-ede9-4369-aa8b-76a95d1edab6-dbus\") pod \"global-pull-secret-syncer-5t728\" (UID: \"64cb49fc-ede9-4369-aa8b-76a95d1edab6\") " pod="kube-system/global-pull-secret-syncer-5t728"
Sep 19 04:13:43.329485 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.329098    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1beab173-cbd6-4858-a590-c08e2d521ed9-env-overrides\") pod \"ovnkube-node-rtwn6\" (UID: \"1beab173-cbd6-4858-a590-c08e2d521ed9\") " pod="openshift-ovn-kubernetes/ovnkube-node-rtwn6"
Sep 19 04:13:43.329485 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.329204    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bc229f7a-0722-442f-940c-7ec542b81888-host-var-lib-cni-multus\") pod \"multus-nwgjm\" (UID: \"bc229f7a-0722-442f-940c-7ec542b81888\") " pod="openshift-multus/multus-nwgjm"
Sep 19 04:13:43.329485 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.329270    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bc229f7a-0722-442f-940c-7ec542b81888-host-var-lib-cni-multus\") pod \"multus-nwgjm\" (UID: \"bc229f7a-0722-442f-940c-7ec542b81888\") " pod="openshift-multus/multus-nwgjm"
Sep 19 04:13:43.329485 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.329302    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bc229f7a-0722-442f-940c-7ec542b81888-hostroot\") pod \"multus-nwgjm\" (UID: \"bc229f7a-0722-442f-940c-7ec542b81888\") " pod="openshift-multus/multus-nwgjm"
Sep 19 04:13:43.329485 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.329319    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bc229f7a-0722-442f-940c-7ec542b81888-host-run-multus-certs\") pod \"multus-nwgjm\" (UID: \"bc229f7a-0722-442f-940c-7ec542b81888\") " pod="openshift-multus/multus-nwgjm"
Sep 19 04:13:43.329485 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.329334    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84819fb3-13e8-482a-b88e-c30a2ef0bc3d-etc-kubernetes\") pod \"tuned-dxttg\" (UID: \"84819fb3-13e8-482a-b88e-c30a2ef0bc3d\") " pod="openshift-cluster-node-tuning-operator/tuned-dxttg"
Sep 19 04:13:43.329485 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.329348    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/84819fb3-13e8-482a-b88e-c30a2ef0bc3d-etc-systemd\") pod \"tuned-dxttg\" (UID: \"84819fb3-13e8-482a-b88e-c30a2ef0bc3d\") " pod="openshift-cluster-node-tuning-operator/tuned-dxttg"
Sep 19 04:13:43.329485 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.329364    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/84819fb3-13e8-482a-b88e-c30a2ef0bc3d-sys\") pod \"tuned-dxttg\" (UID: \"84819fb3-13e8-482a-b88e-c30a2ef0bc3d\") " pod="openshift-cluster-node-tuning-operator/tuned-dxttg"
Sep 19 04:13:43.329485 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.329399    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1beab173-cbd6-4858-a590-c08e2d521ed9-host-cni-bin\") pod \"ovnkube-node-rtwn6\" (UID: \"1beab173-cbd6-4858-a590-c08e2d521ed9\") " pod="openshift-ovn-kubernetes/ovnkube-node-rtwn6"
Sep 19 04:13:43.330090 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.329409    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bc229f7a-0722-442f-940c-7ec542b81888-host-run-multus-certs\") pod \"multus-nwgjm\" (UID: \"bc229f7a-0722-442f-940c-7ec542b81888\") " pod="openshift-multus/multus-nwgjm"
Sep 19 04:13:43.330090 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.329452    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bc229f7a-0722-442f-940c-7ec542b81888-hostroot\") pod \"multus-nwgjm\" (UID: \"bc229f7a-0722-442f-940c-7ec542b81888\") " pod="openshift-multus/multus-nwgjm"
Sep 19 04:13:43.330090 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.329451    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bcd5da2a-a24f-40fa-80aa-03ef54cc22a6-system-cni-dir\") pod \"multus-additional-cni-plugins-wzbjr\" (UID: \"bcd5da2a-a24f-40fa-80aa-03ef54cc22a6\") " pod="openshift-multus/multus-additional-cni-plugins-wzbjr"
Sep 19 04:13:43.330090 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.329485    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bc229f7a-0722-442f-940c-7ec542b81888-cni-binary-copy\") pod \"multus-nwgjm\" (UID: \"bc229f7a-0722-442f-940c-7ec542b81888\") " pod="openshift-multus/multus-nwgjm"
Sep 19 04:13:43.330090 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.329508    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-625bx\" (UniqueName: \"kubernetes.io/projected/4ece1d13-8010-47bb-8e97-40634dd06c5d-kube-api-access-625bx\") pod \"iptables-alerter-kcb82\" (UID: \"4ece1d13-8010-47bb-8e97-40634dd06c5d\") " pod="openshift-network-operator/iptables-alerter-kcb82"
Sep 19 04:13:43.330090 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.329528    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/64c734ac-0f91-4e8b-83e9-a2e9307b3bfe-etc-selinux\") pod \"aws-ebs-csi-driver-node-gnm2x\" (UID: \"64c734ac-0f91-4e8b-83e9-a2e9307b3bfe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gnm2x"
Sep 19 04:13:43.330090 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.329557    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bcd5da2a-a24f-40fa-80aa-03ef54cc22a6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wzbjr\" (UID: \"bcd5da2a-a24f-40fa-80aa-03ef54cc22a6\") " pod="openshift-multus/multus-additional-cni-plugins-wzbjr"
Sep 19 04:13:43.330090 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.329575    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bc229f7a-0722-442f-940c-7ec542b81888-host-run-netns\") pod \"multus-nwgjm\" (UID: \"bc229f7a-0722-442f-940c-7ec542b81888\") " pod="openshift-multus/multus-nwgjm"
Sep 19 04:13:43.330090 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.329587    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a8d1a2a7-a76e-4741-be3a-d87124254f5d-host\") pod \"node-ca-5bzkh\" (UID: \"a8d1a2a7-a76e-4741-be3a-d87124254f5d\") " pod="openshift-image-registry/node-ca-5bzkh"
Sep 19 04:13:43.330090 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.329609    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/64c734ac-0f91-4e8b-83e9-a2e9307b3bfe-sys-fs\") pod \"aws-ebs-csi-driver-node-gnm2x\" (UID: \"64c734ac-0f91-4e8b-83e9-a2e9307b3bfe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gnm2x"
Sep 19 04:13:43.330090 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.329661    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/84819fb3-13e8-482a-b88e-c30a2ef0bc3d-etc-sysctl-conf\") pod \"tuned-dxttg\" (UID: \"84819fb3-13e8-482a-b88e-c30a2ef0bc3d\") " pod="openshift-cluster-node-tuning-operator/tuned-dxttg"
Sep 19 04:13:43.330090 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.329753    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a8d1a2a7-a76e-4741-be3a-d87124254f5d-host\") pod \"node-ca-5bzkh\" (UID: \"a8d1a2a7-a76e-4741-be3a-d87124254f5d\") " pod="openshift-image-registry/node-ca-5bzkh"
Sep 19 04:13:43.330090 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.329913    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/64c734ac-0f91-4e8b-83e9-a2e9307b3bfe-sys-fs\") pod \"aws-ebs-csi-driver-node-gnm2x\" (UID: \"64c734ac-0f91-4e8b-83e9-a2e9307b3bfe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gnm2x"
Sep 19 04:13:43.330090 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.329966    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bcd5da2a-a24f-40fa-80aa-03ef54cc22a6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wzbjr\" (UID: \"bcd5da2a-a24f-40fa-80aa-03ef54cc22a6\") " pod="openshift-multus/multus-additional-cni-plugins-wzbjr"
Sep 19 04:13:43.330090 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.329988    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bc229f7a-0722-442f-940c-7ec542b81888-multus-socket-dir-parent\") pod \"multus-nwgjm\" (UID: \"bc229f7a-0722-442f-940c-7ec542b81888\") " pod="openshift-multus/multus-nwgjm"
Sep 19 04:13:43.330090 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.329998    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/64c734ac-0f91-4e8b-83e9-a2e9307b3bfe-etc-selinux\") pod \"aws-ebs-csi-driver-node-gnm2x\" (UID: \"64c734ac-0f91-4e8b-83e9-a2e9307b3bfe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gnm2x"
Sep 19 04:13:43.330090 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.330016    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3738f1c7-67af-4201-8e8d-1d0406e0edb9-metrics-certs\") pod \"network-metrics-daemon-wmk74\" (UID: \"3738f1c7-67af-4201-8e8d-1d0406e0edb9\") " pod="openshift-multus/network-metrics-daemon-wmk74"
Sep 19 04:13:43.330688 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.330031    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bc229f7a-0722-442f-940c-7ec542b81888-host-run-netns\") pod \"multus-nwgjm\" (UID: \"bc229f7a-0722-442f-940c-7ec542b81888\") " pod="openshift-multus/multus-nwgjm"
Sep 19 04:13:43.330688 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.330031    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j856z\" (UniqueName: \"kubernetes.io/projected/f2fe8238-2a1d-4e7a-b4d7-4cc7d518f366-kube-api-access-j856z\") pod \"network-check-target-z7dgk\" (UID: \"f2fe8238-2a1d-4e7a-b4d7-4cc7d518f366\") " pod="openshift-network-diagnostics/network-check-target-z7dgk"
Sep 19 04:13:43.330688 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.330040    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bc229f7a-0722-442f-940c-7ec542b81888-cni-binary-copy\") pod \"multus-nwgjm\" (UID: \"bc229f7a-0722-442f-940c-7ec542b81888\") " pod="openshift-multus/multus-nwgjm"
Sep 19 04:13:43.330688 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.330057    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5194aa7-dba0-46dc-b3e9-91a4ea69e79e-tmp-dir\") pod \"node-resolver-2ptq2\" (UID: \"a5194aa7-dba0-46dc-b3e9-91a4ea69e79e\") " pod="openshift-dns/node-resolver-2ptq2"
Sep 19 04:13:43.330688 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.330073    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/84819fb3-13e8-482a-b88e-c30a2ef0bc3d-tmp\") pod \"tuned-dxttg\" (UID: \"84819fb3-13e8-482a-b88e-c30a2ef0bc3d\") " pod="openshift-cluster-node-tuning-operator/tuned-dxttg"
Sep 19 04:13:43.330688 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.330088    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bc229f7a-0722-442f-940c-7ec542b81888-host-run-k8s-cni-cncf-io\") pod \"multus-nwgjm\" (UID: \"bc229f7a-0722-442f-940c-7ec542b81888\") " pod="openshift-multus/multus-nwgjm"
Sep 19 04:13:43.330688 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.330103    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4ece1d13-8010-47bb-8e97-40634dd06c5d-host-slash\") pod \"iptables-alerter-kcb82\" (UID: \"4ece1d13-8010-47bb-8e97-40634dd06c5d\") " pod="openshift-network-operator/iptables-alerter-kcb82"
Sep 19 04:13:43.330688 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.330118    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a8d1a2a7-a76e-4741-be3a-d87124254f5d-serviceca\") pod \"node-ca-5bzkh\" (UID: \"a8d1a2a7-a76e-4741-be3a-d87124254f5d\") " pod="openshift-image-registry/node-ca-5bzkh"
Sep 19 04:13:43.330688 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.330134    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/64c734ac-0f91-4e8b-83e9-a2e9307b3bfe-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gnm2x\" (UID: \"64c734ac-0f91-4e8b-83e9-a2e9307b3bfe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gnm2x"
Sep 19 04:13:43.330688 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.330150    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/84819fb3-13e8-482a-b88e-c30a2ef0bc3d-var-lib-kubelet\") pod \"tuned-dxttg\" (UID: \"84819fb3-13e8-482a-b88e-c30a2ef0bc3d\") " pod="openshift-cluster-node-tuning-operator/tuned-dxttg"
Sep 19 04:13:43.330688 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.330164    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1beab173-cbd6-4858-a590-c08e2d521ed9-host-slash\") pod \"ovnkube-node-rtwn6\" (UID: \"1beab173-cbd6-4858-a590-c08e2d521ed9\") " pod="openshift-ovn-kubernetes/ovnkube-node-rtwn6"
Sep 19 04:13:43.330688 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.330180    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1beab173-cbd6-4858-a590-c08e2d521ed9-var-lib-openvswitch\") pod \"ovnkube-node-rtwn6\" (UID: \"1beab173-cbd6-4858-a590-c08e2d521ed9\") " pod="openshift-ovn-kubernetes/ovnkube-node-rtwn6"
Sep 19 04:13:43.330688 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.330197    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7mbf\" (UniqueName: \"kubernetes.io/projected/64c734ac-0f91-4e8b-83e9-a2e9307b3bfe-kube-api-access-m7mbf\") pod \"aws-ebs-csi-driver-node-gnm2x\" (UID: \"64c734ac-0f91-4e8b-83e9-a2e9307b3bfe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gnm2x"
Sep 19 04:13:43.330688 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:43.330214    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 04:13:43.330688 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.330225    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f4b6c76b-7816-4e8b-9904-2f0cb5faca57-agent-certs\") pod \"konnectivity-agent-szpwc\" (UID: \"f4b6c76b-7816-4e8b-9904-2f0cb5faca57\") " pod="kube-system/konnectivity-agent-szpwc"
Sep 19 04:13:43.330688 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.330242    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1beab173-cbd6-4858-a590-c08e2d521ed9-host-run-ovn-kubernetes\") pod \"ovnkube-node-rtwn6\" (UID: \"1beab173-cbd6-4858-a590-c08e2d521ed9\") " pod="openshift-ovn-kubernetes/ovnkube-node-rtwn6"
Sep 19 04:13:43.330688 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:43.330273    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3738f1c7-67af-4201-8e8d-1d0406e0edb9-metrics-certs podName:3738f1c7-67af-4201-8e8d-1d0406e0edb9 nodeName:}" failed. No retries permitted until 2026-09-19 04:13:43.830246731 +0000 UTC m=+3.013207858 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3738f1c7-67af-4201-8e8d-1d0406e0edb9-metrics-certs") pod "network-metrics-daemon-wmk74" (UID: "3738f1c7-67af-4201-8e8d-1d0406e0edb9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 04:13:43.331394 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.330290    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4ece1d13-8010-47bb-8e97-40634dd06c5d-host-slash\") pod \"iptables-alerter-kcb82\" (UID: \"4ece1d13-8010-47bb-8e97-40634dd06c5d\") " pod="openshift-network-operator/iptables-alerter-kcb82"
Sep 19 04:13:43.331394 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.330290    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5194aa7-dba0-46dc-b3e9-91a4ea69e79e-tmp-dir\") pod \"node-resolver-2ptq2\" (UID: \"a5194aa7-dba0-46dc-b3e9-91a4ea69e79e\") " pod="openshift-dns/node-resolver-2ptq2"
Sep 19 04:13:43.331394 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.330103    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bc229f7a-0722-442f-940c-7ec542b81888-multus-socket-dir-parent\") pod \"multus-nwgjm\" (UID: \"bc229f7a-0722-442f-940c-7ec542b81888\") " pod="openshift-multus/multus-nwgjm"
Sep 19 04:13:43.331394 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.330273    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bc229f7a-0722-442f-940c-7ec542b81888-host-run-k8s-cni-cncf-io\") pod \"multus-nwgjm\" (UID: \"bc229f7a-0722-442f-940c-7ec542b81888\") " pod="openshift-multus/multus-nwgjm"
Sep 19 04:13:43.331394 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.330403    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bcd5da2a-a24f-40fa-80aa-03ef54cc22a6-cnibin\") pod \"multus-additional-cni-plugins-wzbjr\" (UID: \"bcd5da2a-a24f-40fa-80aa-03ef54cc22a6\") " pod="openshift-multus/multus-additional-cni-plugins-wzbjr"
Sep 19 04:13:43.331394 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.330421    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/64cb49fc-ede9-4369-aa8b-76a95d1edab6-original-pull-secret\") pod \"global-pull-secret-syncer-5t728\" (UID: \"64cb49fc-ede9-4369-aa8b-76a95d1edab6\") " pod="kube-system/global-pull-secret-syncer-5t728"
Sep 19 04:13:43.331394 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.330437    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4ece1d13-8010-47bb-8e97-40634dd06c5d-iptables-alerter-script\") pod \"iptables-alerter-kcb82\" (UID: \"4ece1d13-8010-47bb-8e97-40634dd06c5d\") " pod="openshift-network-operator/iptables-alerter-kcb82"
Sep 19 04:13:43.331394 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.330452    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/84819fb3-13e8-482a-b88e-c30a2ef0bc3d-etc-modprobe-d\") pod \"tuned-dxttg\" (UID: \"84819fb3-13e8-482a-b88e-c30a2ef0bc3d\") " pod="openshift-cluster-node-tuning-operator/tuned-dxttg"
Sep 19 04:13:43.331394 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.330467    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1beab173-cbd6-4858-a590-c08e2d521ed9-ovn-node-metrics-cert\") pod \"ovnkube-node-rtwn6\" (UID: \"1beab173-cbd6-4858-a590-c08e2d521ed9\") " pod="openshift-ovn-kubernetes/ovnkube-node-rtwn6"
Sep 19 04:13:43.331394 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.330483    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bcd5da2a-a24f-40fa-80aa-03ef54cc22a6-cni-binary-copy\") pod \"multus-additional-cni-plugins-wzbjr\" (UID: \"bcd5da2a-a24f-40fa-80aa-03ef54cc22a6\") " pod="openshift-multus/multus-additional-cni-plugins-wzbjr"
Sep 19 04:13:43.331394 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.330498    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bc229f7a-0722-442f-940c-7ec542b81888-etc-kubernetes\") pod \"multus-nwgjm\" (UID: \"bc229f7a-0722-442f-940c-7ec542b81888\") " pod="openshift-multus/multus-nwgjm"
Sep 19 04:13:43.331394 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.330513    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx5h4\" (UniqueName: \"kubernetes.io/projected/bc229f7a-0722-442f-940c-7ec542b81888-kube-api-access-jx5h4\") pod \"multus-nwgjm\" (UID: \"bc229f7a-0722-442f-940c-7ec542b81888\") " pod="openshift-multus/multus-nwgjm"
Sep 19 04:13:43.331394 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:43.330523    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 19 04:13:43.331394 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.330541    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a8d1a2a7-a76e-4741-be3a-d87124254f5d-serviceca\") pod \"node-ca-5bzkh\" (UID: \"a8d1a2a7-a76e-4741-be3a-d87124254f5d\") " pod="openshift-image-registry/node-ca-5bzkh"
Sep 19 04:13:43.331394 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:43.330568    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64cb49fc-ede9-4369-aa8b-76a95d1edab6-original-pull-secret podName:64cb49fc-ede9-4369-aa8b-76a95d1edab6 nodeName:}" failed. No retries permitted until 2026-09-19 04:13:43.830555882 +0000 UTC m=+3.013517008 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/64cb49fc-ede9-4369-aa8b-76a95d1edab6-original-pull-secret") pod "global-pull-secret-syncer-5t728" (UID: "64cb49fc-ede9-4369-aa8b-76a95d1edab6") : object "kube-system"/"original-pull-secret" not registered
Sep 19 04:13:43.331394 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.330542    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/84819fb3-13e8-482a-b88e-c30a2ef0bc3d-etc-sysconfig\") pod \"tuned-dxttg\" (UID: \"84819fb3-13e8-482a-b88e-c30a2ef0bc3d\") " pod="openshift-cluster-node-tuning-operator/tuned-dxttg"
Sep 19 04:13:43.331394 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.330589    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1beab173-cbd6-4858-a590-c08e2d521ed9-host-run-netns\") pod \"ovnkube-node-rtwn6\" (UID: \"1beab173-cbd6-4858-a590-c08e2d521ed9\") " pod="openshift-ovn-kubernetes/ovnkube-node-rtwn6"
Sep 19 04:13:43.332326 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.330613    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bc229f7a-0722-442f-940c-7ec542b81888-multus-daemon-config\") pod \"multus-nwgjm\" (UID: \"bc229f7a-0722-442f-940c-7ec542b81888\") " pod="openshift-multus/multus-nwgjm"
Sep 19 04:13:43.332326 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.330627    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1beab173-cbd6-4858-a590-c08e2d521ed9-node-log\") pod \"ovnkube-node-rtwn6\" (UID: \"1beab173-cbd6-4858-a590-c08e2d521ed9\") " pod="openshift-ovn-kubernetes/ovnkube-node-rtwn6"
Sep 19 04:13:43.332326 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.330642    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1beab173-cbd6-4858-a590-c08e2d521ed9-ovnkube-config\") pod \"ovnkube-node-rtwn6\" (UID: \"1beab173-cbd6-4858-a590-c08e2d521ed9\") " pod="openshift-ovn-kubernetes/ovnkube-node-rtwn6"
Sep 19 04:13:43.332326 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.330673    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t6w9\" (UniqueName: \"kubernetes.io/projected/bcd5da2a-a24f-40fa-80aa-03ef54cc22a6-kube-api-access-9t6w9\") pod \"multus-additional-cni-plugins-wzbjr\" (UID: \"bcd5da2a-a24f-40fa-80aa-03ef54cc22a6\") " pod="openshift-multus/multus-additional-cni-plugins-wzbjr"
Sep 19 04:13:43.332326 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.330690    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bc229f7a-0722-442f-940c-7ec542b81888-multus-cni-dir\") pod \"multus-nwgjm\" (UID: \"bc229f7a-0722-442f-940c-7ec542b81888\") " pod="openshift-multus/multus-nwgjm"
Sep 19 04:13:43.332326 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.330722    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bc229f7a-0722-442f-940c-7ec542b81888-host-var-lib-cni-bin\") pod \"multus-nwgjm\" (UID: \"bc229f7a-0722-442f-940c-7ec542b81888\") " pod="openshift-multus/multus-nwgjm"
Sep 19 04:13:43.332326 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.330742    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1beab173-cbd6-4858-a590-c08e2d521ed9-host-kubelet\") pod \"ovnkube-node-rtwn6\" (UID: \"1beab173-cbd6-4858-a590-c08e2d521ed9\") " pod="openshift-ovn-kubernetes/ovnkube-node-rtwn6"
Sep 19 04:13:43.332326 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.330745    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bc229f7a-0722-442f-940c-7ec542b81888-etc-kubernetes\") pod \"multus-nwgjm\" (UID: \"bc229f7a-0722-442f-940c-7ec542b81888\") " pod="openshift-multus/multus-nwgjm"
Sep 19 04:13:43.332326 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.330810    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bc229f7a-0722-442f-940c-7ec542b81888-host-var-lib-cni-bin\") pod \"multus-nwgjm\" (UID: \"bc229f7a-0722-442f-940c-7ec542b81888\") " pod="openshift-multus/multus-nwgjm"
Sep 19 04:13:43.332326 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.330827    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bc229f7a-0722-442f-940c-7ec542b81888-multus-cni-dir\") pod \"multus-nwgjm\" (UID: \"bc229f7a-0722-442f-940c-7ec542b81888\") " pod="openshift-multus/multus-nwgjm"
Sep 19 04:13:43.332326 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.330845    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/64c734ac-0f91-4e8b-83e9-a2e9307b3bfe-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gnm2x\" (UID: \"64c734ac-0f91-4e8b-83e9-a2e9307b3bfe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gnm2x"
Sep 19 04:13:43.332326 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.330766    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bc229f7a-0722-442f-940c-7ec542b81888-system-cni-dir\") pod \"multus-nwgjm\" (UID: \"bc229f7a-0722-442f-940c-7ec542b81888\") " pod="openshift-multus/multus-nwgjm"
Sep 19 04:13:43.332326 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.330922    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bc229f7a-0722-442f-940c-7ec542b81888-host-var-lib-kubelet\") pod \"multus-nwgjm\" (UID: \"bc229f7a-0722-442f-940c-7ec542b81888\") " pod="openshift-multus/multus-nwgjm"
Sep 19 04:13:43.332326 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.330978    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/64c734ac-0f91-4e8b-83e9-a2e9307b3bfe-socket-dir\") pod \"aws-ebs-csi-driver-node-gnm2x\" (UID: \"64c734ac-0f91-4e8b-83e9-a2e9307b3bfe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gnm2x"
Sep 19 04:13:43.332326 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.330983    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bc229f7a-0722-442f-940c-7ec542b81888-host-var-lib-kubelet\") pod \"multus-nwgjm\" (UID: \"bc229f7a-0722-442f-940c-7ec542b81888\") " pod="openshift-multus/multus-nwgjm"
Sep 19 04:13:43.332326 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.331009    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bc229f7a-0722-442f-940c-7ec542b81888-system-cni-dir\") pod \"multus-nwgjm\" (UID: \"bc229f7a-0722-442f-940c-7ec542b81888\") " pod="openshift-multus/multus-nwgjm"
Sep 19 04:13:43.332326 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.331010    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/84819fb3-13e8-482a-b88e-c30a2ef0bc3d-etc-tuned\") pod \"tuned-dxttg\" (UID: \"84819fb3-13e8-482a-b88e-c30a2ef0bc3d\") " pod="openshift-cluster-node-tuning-operator/tuned-dxttg"
Sep 19 04:13:43.333122 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.331011    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4ece1d13-8010-47bb-8e97-40634dd06c5d-iptables-alerter-script\") pod \"iptables-alerter-kcb82\" (UID: \"4ece1d13-8010-47bb-8e97-40634dd06c5d\") " pod="openshift-network-operator/iptables-alerter-kcb82"
Sep 19 04:13:43.333122 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.331062    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1beab173-cbd6-4858-a590-c08e2d521ed9-run-systemd\") pod \"ovnkube-node-rtwn6\" (UID: \"1beab173-cbd6-4858-a590-c08e2d521ed9\") " pod="openshift-ovn-kubernetes/ovnkube-node-rtwn6"
Sep 19 04:13:43.333122 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.331080    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1beab173-cbd6-4858-a590-c08e2d521ed9-run-openvswitch\") pod \"ovnkube-node-rtwn6\" (UID: \"1beab173-cbd6-4858-a590-c08e2d521ed9\") " pod="openshift-ovn-kubernetes/ovnkube-node-rtwn6"
Sep 19 04:13:43.333122 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.331085    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/64c734ac-0f91-4e8b-83e9-a2e9307b3bfe-socket-dir\") pod \"aws-ebs-csi-driver-node-gnm2x\" (UID: \"64c734ac-0f91-4e8b-83e9-a2e9307b3bfe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gnm2x"
Sep 19 04:13:43.333122 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.331111    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1beab173-cbd6-4858-a590-c08e2d521ed9-host-cni-netd\") pod \"ovnkube-node-rtwn6\" (UID: \"1beab173-cbd6-4858-a590-c08e2d521ed9\") " pod="openshift-ovn-kubernetes/ovnkube-node-rtwn6"
Sep 19 04:13:43.333122 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.331143    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/64c734ac-0f91-4e8b-83e9-a2e9307b3bfe-device-dir\") pod \"aws-ebs-csi-driver-node-gnm2x\" (UID: \"64c734ac-0f91-4e8b-83e9-a2e9307b3bfe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gnm2x"
Sep 19 04:13:43.333122 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.331179    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1beab173-cbd6-4858-a590-c08e2d521ed9-run-ovn\") pod \"ovnkube-node-rtwn6\" (UID: \"1beab173-cbd6-4858-a590-c08e2d521ed9\") " pod="openshift-ovn-kubernetes/ovnkube-node-rtwn6"
Sep 19 04:13:43.333122 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.331207    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/64c734ac-0f91-4e8b-83e9-a2e9307b3bfe-device-dir\") pod \"aws-ebs-csi-driver-node-gnm2x\" (UID: \"64c734ac-0f91-4e8b-83e9-a2e9307b3bfe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gnm2x"
Sep 19 04:13:43.333122 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.331236    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1beab173-cbd6-4858-a590-c08e2d521ed9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rtwn6\" (UID: \"1beab173-cbd6-4858-a590-c08e2d521ed9\") " pod="openshift-ovn-kubernetes/ovnkube-node-rtwn6"
Sep 19 04:13:43.333122 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.331514    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bc229f7a-0722-442f-940c-7ec542b81888-multus-daemon-config\") pod \"multus-nwgjm\" (UID: \"bc229f7a-0722-442f-940c-7ec542b81888\") " pod="openshift-multus/multus-nwgjm"
Sep 19 04:13:43.347117 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.347087    2579 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 19 04:13:43.352171 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.352145    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx5h4\" (UniqueName: \"kubernetes.io/projected/bc229f7a-0722-442f-940c-7ec542b81888-kube-api-access-jx5h4\") pod \"multus-nwgjm\" (UID: \"bc229f7a-0722-442f-940c-7ec542b81888\") " pod="openshift-multus/multus-nwgjm"
Sep 19 04:13:43.353162 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.353139    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqmvn\" (UniqueName: \"kubernetes.io/projected/a5194aa7-dba0-46dc-b3e9-91a4ea69e79e-kube-api-access-jqmvn\") pod \"node-resolver-2ptq2\" (UID: \"a5194aa7-dba0-46dc-b3e9-91a4ea69e79e\") " pod="openshift-dns/node-resolver-2ptq2"
Sep 19 04:13:43.355939 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.355916    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtktn\" (UniqueName: \"kubernetes.io/projected/a8d1a2a7-a76e-4741-be3a-d87124254f5d-kube-api-access-dtktn\") pod \"node-ca-5bzkh\" (UID: \"a8d1a2a7-a76e-4741-be3a-d87124254f5d\") " pod="openshift-image-registry/node-ca-5bzkh"
Sep 19 04:13:43.361604 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.361576    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhhsm\" (UniqueName: \"kubernetes.io/projected/3738f1c7-67af-4201-8e8d-1d0406e0edb9-kube-api-access-dhhsm\") pod \"network-metrics-daemon-wmk74\" (UID: \"3738f1c7-67af-4201-8e8d-1d0406e0edb9\") " pod="openshift-multus/network-metrics-daemon-wmk74"
Sep 19 04:13:43.363543 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.363519    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-625bx\" (UniqueName: \"kubernetes.io/projected/4ece1d13-8010-47bb-8e97-40634dd06c5d-kube-api-access-625bx\") pod \"iptables-alerter-kcb82\" (UID: \"4ece1d13-8010-47bb-8e97-40634dd06c5d\") " pod="openshift-network-operator/iptables-alerter-kcb82"
Sep 19 04:13:43.363866 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:43.363842    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 19 04:13:43.363949 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:43.363872    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 19 04:13:43.363949 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:43.363882    2579 projected.go:196] Error preparing data for projected volume kube-api-access-j856z for pod openshift-network-diagnostics/network-check-target-z7dgk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 04:13:43.363949 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:43.363934    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f2fe8238-2a1d-4e7a-b4d7-4cc7d518f366-kube-api-access-j856z podName:f2fe8238-2a1d-4e7a-b4d7-4cc7d518f366 nodeName:}" failed. No retries permitted until 2026-09-19 04:13:43.863920634 +0000 UTC m=+3.046881763 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-j856z" (UniqueName: "kubernetes.io/projected/f2fe8238-2a1d-4e7a-b4d7-4cc7d518f366-kube-api-access-j856z") pod "network-check-target-z7dgk" (UID: "f2fe8238-2a1d-4e7a-b4d7-4cc7d518f366") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 04:13:43.371064 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.371043    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7mbf\" (UniqueName: \"kubernetes.io/projected/64c734ac-0f91-4e8b-83e9-a2e9307b3bfe-kube-api-access-m7mbf\") pod \"aws-ebs-csi-driver-node-gnm2x\" (UID: \"64c734ac-0f91-4e8b-83e9-a2e9307b3bfe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gnm2x"
Sep 19 04:13:43.431949 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.431910    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f4b6c76b-7816-4e8b-9904-2f0cb5faca57-konnectivity-ca\") pod \"konnectivity-agent-szpwc\" (UID: \"f4b6c76b-7816-4e8b-9904-2f0cb5faca57\") " pod="kube-system/konnectivity-agent-szpwc"
Sep 19 04:13:43.431949 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.431950    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1beab173-cbd6-4858-a590-c08e2d521ed9-systemd-units\") pod \"ovnkube-node-rtwn6\" (UID: \"1beab173-cbd6-4858-a590-c08e2d521ed9\") " pod="openshift-ovn-kubernetes/ovnkube-node-rtwn6"
Sep 19 04:13:43.432188 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.431971    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1beab173-cbd6-4858-a590-c08e2d521ed9-etc-openvswitch\") pod \"ovnkube-node-rtwn6\" (UID: \"1beab173-cbd6-4858-a590-c08e2d521ed9\") " pod="openshift-ovn-kubernetes/ovnkube-node-rtwn6"
Sep 19 04:13:43.432188 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.431994    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1beab173-cbd6-4858-a590-c08e2d521ed9-log-socket\") pod \"ovnkube-node-rtwn6\" (UID: \"1beab173-cbd6-4858-a590-c08e2d521ed9\") " pod="openshift-ovn-kubernetes/ovnkube-node-rtwn6"
Sep 19 04:13:43.432188 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.432015    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/84819fb3-13e8-482a-b88e-c30a2ef0bc3d-etc-sysctl-d\") pod \"tuned-dxttg\" (UID: \"84819fb3-13e8-482a-b88e-c30a2ef0bc3d\") " pod="openshift-cluster-node-tuning-operator/tuned-dxttg"
Sep 19 04:13:43.432188 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.432033    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/84819fb3-13e8-482a-b88e-c30a2ef0bc3d-host\") pod \"tuned-dxttg\" (UID: \"84819fb3-13e8-482a-b88e-c30a2ef0bc3d\") " pod="openshift-cluster-node-tuning-operator/tuned-dxttg"
Sep 19 04:13:43.432188 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.432055    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjp4w\" (UniqueName: \"kubernetes.io/projected/84819fb3-13e8-482a-b88e-c30a2ef0bc3d-kube-api-access-wjp4w\") pod \"tuned-dxttg\" (UID: \"84819fb3-13e8-482a-b88e-c30a2ef0bc3d\") " pod="openshift-cluster-node-tuning-operator/tuned-dxttg"
Sep 19 04:13:43.432188 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.432059    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1beab173-cbd6-4858-a590-c08e2d521ed9-log-socket\") pod \"ovnkube-node-rtwn6\" (UID: \"1beab173-cbd6-4858-a590-c08e2d521ed9\") " pod="openshift-ovn-kubernetes/ovnkube-node-rtwn6"
Sep 19 04:13:43.432188 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.432077    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/84819fb3-13e8-482a-b88e-c30a2ef0bc3d-run\") pod \"tuned-dxttg\" (UID: \"84819fb3-13e8-482a-b88e-c30a2ef0bc3d\") " pod="openshift-cluster-node-tuning-operator/tuned-dxttg"
Sep 19 04:13:43.432188 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.432032    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1beab173-cbd6-4858-a590-c08e2d521ed9-systemd-units\") pod \"ovnkube-node-rtwn6\" (UID: \"1beab173-cbd6-4858-a590-c08e2d521ed9\") " pod="openshift-ovn-kubernetes/ovnkube-node-rtwn6"
Sep 19 04:13:43.432188 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.432108    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/84819fb3-13e8-482a-b88e-c30a2ef0bc3d-host\") pod \"tuned-dxttg\" (UID: \"84819fb3-13e8-482a-b88e-c30a2ef0bc3d\") " pod="openshift-cluster-node-tuning-operator/tuned-dxttg"
Sep 19 04:13:43.432188 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.432116    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1beab173-cbd6-4858-a590-c08e2d521ed9-env-overrides\") pod \"ovnkube-node-rtwn6\" (UID: \"1beab173-cbd6-4858-a590-c08e2d521ed9\") " pod="openshift-ovn-kubernetes/ovnkube-node-rtwn6"
Sep 19 04:13:43.432188 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.432128    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/84819fb3-13e8-482a-b88e-c30a2ef0bc3d-run\") pod \"tuned-dxttg\" (UID: \"84819fb3-13e8-482a-b88e-c30a2ef0bc3d\") " pod="openshift-cluster-node-tuning-operator/tuned-dxttg"
Sep 19 04:13:43.432188 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.432161    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84819fb3-13e8-482a-b88e-c30a2ef0bc3d-etc-kubernetes\") pod \"tuned-dxttg\" (UID: \"84819fb3-13e8-482a-b88e-c30a2ef0bc3d\") " pod="openshift-cluster-node-tuning-operator/tuned-dxttg"
Sep 19 04:13:43.432188 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.432102    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1beab173-cbd6-4858-a590-c08e2d521ed9-etc-openvswitch\") pod \"ovnkube-node-rtwn6\" (UID: \"1beab173-cbd6-4858-a590-c08e2d521ed9\") " pod="openshift-ovn-kubernetes/ovnkube-node-rtwn6"
Sep 19 04:13:43.432188 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.432184    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/84819fb3-13e8-482a-b88e-c30a2ef0bc3d-etc-systemd\") pod \"tuned-dxttg\" (UID: \"84819fb3-13e8-482a-b88e-c30a2ef0bc3d\") " pod="openshift-cluster-node-tuning-operator/tuned-dxttg"
Sep 19 04:13:43.432881 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.432207    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/84819fb3-13e8-482a-b88e-c30a2ef0bc3d-sys\") pod \"tuned-dxttg\" (UID: \"84819fb3-13e8-482a-b88e-c30a2ef0bc3d\") " pod="openshift-cluster-node-tuning-operator/tuned-dxttg"
Sep 19 04:13:43.432881 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.432219    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84819fb3-13e8-482a-b88e-c30a2ef0bc3d-etc-kubernetes\") pod \"tuned-dxttg\" (UID: \"84819fb3-13e8-482a-b88e-c30a2ef0bc3d\") " pod="openshift-cluster-node-tuning-operator/tuned-dxttg"
Sep 19 04:13:43.432881 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.432230    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1beab173-cbd6-4858-a590-c08e2d521ed9-host-cni-bin\") pod \"ovnkube-node-rtwn6\" (UID: \"1beab173-cbd6-4858-a590-c08e2d521ed9\") " pod="openshift-ovn-kubernetes/ovnkube-node-rtwn6"
Sep 19 04:13:43.432881 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.432257    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/84819fb3-13e8-482a-b88e-c30a2ef0bc3d-etc-systemd\") pod \"tuned-dxttg\" (UID: \"84819fb3-13e8-482a-b88e-c30a2ef0bc3d\") " pod="openshift-cluster-node-tuning-operator/tuned-dxttg"
Sep 19 04:13:43.432881 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.432270    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1beab173-cbd6-4858-a590-c08e2d521ed9-host-cni-bin\") pod \"ovnkube-node-rtwn6\" (UID: \"1beab173-cbd6-4858-a590-c08e2d521ed9\") " pod="openshift-ovn-kubernetes/ovnkube-node-rtwn6"
Sep 19 04:13:43.432881 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.432262    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bcd5da2a-a24f-40fa-80aa-03ef54cc22a6-system-cni-dir\") pod \"multus-additional-cni-plugins-wzbjr\" (UID: \"bcd5da2a-a24f-40fa-80aa-03ef54cc22a6\") " pod="openshift-multus/multus-additional-cni-plugins-wzbjr"
Sep 19 04:13:43.432881 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.432270    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/84819fb3-13e8-482a-b88e-c30a2ef0bc3d-sys\") pod \"tuned-dxttg\" (UID: \"84819fb3-13e8-482a-b88e-c30a2ef0bc3d\") " pod="openshift-cluster-node-tuning-operator/tuned-dxttg"
Sep 19 04:13:43.432881 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.432296    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bcd5da2a-a24f-40fa-80aa-03ef54cc22a6-system-cni-dir\") pod \"multus-additional-cni-plugins-wzbjr\" (UID: \"bcd5da2a-a24f-40fa-80aa-03ef54cc22a6\") " pod="openshift-multus/multus-additional-cni-plugins-wzbjr"
Sep 19 04:13:43.432881 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.432219    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/84819fb3-13e8-482a-b88e-c30a2ef0bc3d-etc-sysctl-d\") pod \"tuned-dxttg\" (UID: \"84819fb3-13e8-482a-b88e-c30a2ef0bc3d\") " pod="openshift-cluster-node-tuning-operator/tuned-dxttg"
Sep 19 04:13:43.432881 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.432312    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bcd5da2a-a24f-40fa-80aa-03ef54cc22a6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wzbjr\" (UID: \"bcd5da2a-a24f-40fa-80aa-03ef54cc22a6\") " pod="openshift-multus/multus-additional-cni-plugins-wzbjr"
Sep 19 04:13:43.432881 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.432343    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/84819fb3-13e8-482a-b88e-c30a2ef0bc3d-etc-sysctl-conf\") pod \"tuned-dxttg\" (UID: \"84819fb3-13e8-482a-b88e-c30a2ef0bc3d\") " pod="openshift-cluster-node-tuning-operator/tuned-dxttg"
Sep 19 04:13:43.432881 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.432370    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bcd5da2a-a24f-40fa-80aa-03ef54cc22a6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wzbjr\" (UID: \"bcd5da2a-a24f-40fa-80aa-03ef54cc22a6\") " pod="openshift-multus/multus-additional-cni-plugins-wzbjr"
Sep 19 04:13:43.432881 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.432427    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/84819fb3-13e8-482a-b88e-c30a2ef0bc3d-tmp\") pod \"tuned-dxttg\" (UID: \"84819fb3-13e8-482a-b88e-c30a2ef0bc3d\") " pod="openshift-cluster-node-tuning-operator/tuned-dxttg"
Sep 19 04:13:43.432881 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.432476    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/84819fb3-13e8-482a-b88e-c30a2ef0bc3d-var-lib-kubelet\") pod \"tuned-dxttg\" (UID: \"84819fb3-13e8-482a-b88e-c30a2ef0bc3d\") " pod="openshift-cluster-node-tuning-operator/tuned-dxttg"
Sep 19 04:13:43.432881 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.432488    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/84819fb3-13e8-482a-b88e-c30a2ef0bc3d-etc-sysctl-conf\") pod \"tuned-dxttg\" (UID: \"84819fb3-13e8-482a-b88e-c30a2ef0bc3d\") " pod="openshift-cluster-node-tuning-operator/tuned-dxttg"
Sep 19 04:13:43.432881 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.432497    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1beab173-cbd6-4858-a590-c08e2d521ed9-host-slash\") pod \"ovnkube-node-rtwn6\" (UID: \"1beab173-cbd6-4858-a590-c08e2d521ed9\") " pod="openshift-ovn-kubernetes/ovnkube-node-rtwn6"
Sep 19 04:13:43.432881 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.432518    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1beab173-cbd6-4858-a590-c08e2d521ed9-var-lib-openvswitch\") pod \"ovnkube-node-rtwn6\" (UID: \"1beab173-cbd6-4858-a590-c08e2d521ed9\") " pod="openshift-ovn-kubernetes/ovnkube-node-rtwn6"
Sep 19 04:13:43.433729 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.432536    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bcd5da2a-a24f-40fa-80aa-03ef54cc22a6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wzbjr\" (UID: \"bcd5da2a-a24f-40fa-80aa-03ef54cc22a6\") " pod="openshift-multus/multus-additional-cni-plugins-wzbjr"
Sep 19 04:13:43.433729 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.432541    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/84819fb3-13e8-482a-b88e-c30a2ef0bc3d-var-lib-kubelet\") pod \"tuned-dxttg\" (UID: \"84819fb3-13e8-482a-b88e-c30a2ef0bc3d\") " pod="openshift-cluster-node-tuning-operator/tuned-dxttg"
Sep 19 04:13:43.433729 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.432542    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f4b6c76b-7816-4e8b-9904-2f0cb5faca57-agent-certs\") pod \"konnectivity-agent-szpwc\" (UID: \"f4b6c76b-7816-4e8b-9904-2f0cb5faca57\") " pod="kube-system/konnectivity-agent-szpwc"
Sep 19 04:13:43.433729 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.432559    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f4b6c76b-7816-4e8b-9904-2f0cb5faca57-konnectivity-ca\") pod \"konnectivity-agent-szpwc\" (UID: \"f4b6c76b-7816-4e8b-9904-2f0cb5faca57\") " pod="kube-system/konnectivity-agent-szpwc"
Sep 19 04:13:43.433729 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.432565    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1beab173-cbd6-4858-a590-c08e2d521ed9-env-overrides\") pod \"ovnkube-node-rtwn6\" (UID: \"1beab173-cbd6-4858-a590-c08e2d521ed9\") " pod="openshift-ovn-kubernetes/ovnkube-node-rtwn6"
Sep 19 04:13:43.433729 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.432578    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1beab173-cbd6-4858-a590-c08e2d521ed9-host-run-ovn-kubernetes\") pod \"ovnkube-node-rtwn6\" (UID: \"1beab173-cbd6-4858-a590-c08e2d521ed9\") " pod="openshift-ovn-kubernetes/ovnkube-node-rtwn6"
Sep 19 04:13:43.433729 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.432579    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1beab173-cbd6-4858-a590-c08e2d521ed9-host-slash\") pod \"ovnkube-node-rtwn6\" (UID: \"1beab173-cbd6-4858-a590-c08e2d521ed9\") " pod="openshift-ovn-kubernetes/ovnkube-node-rtwn6"
Sep 19 04:13:43.433729 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.432608    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1beab173-cbd6-4858-a590-c08e2d521ed9-var-lib-openvswitch\") pod \"ovnkube-node-rtwn6\" (UID: \"1beab173-cbd6-4858-a590-c08e2d521ed9\") " pod="openshift-ovn-kubernetes/ovnkube-node-rtwn6"
Sep 19 04:13:43.433729 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.432614    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bcd5da2a-a24f-40fa-80aa-03ef54cc22a6-cnibin\") pod \"multus-additional-cni-plugins-wzbjr\" (UID: \"bcd5da2a-a24f-40fa-80aa-03ef54cc22a6\") " pod="openshift-multus/multus-additional-cni-plugins-wzbjr"
Sep 19 04:13:43.433729 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.432650    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bcd5da2a-a24f-40fa-80aa-03ef54cc22a6-cnibin\") pod \"multus-additional-cni-plugins-wzbjr\" (UID: \"bcd5da2a-a24f-40fa-80aa-03ef54cc22a6\") " pod="openshift-multus/multus-additional-cni-plugins-wzbjr"
Sep 19 04:13:43.433729 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.432656    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/84819fb3-13e8-482a-b88e-c30a2ef0bc3d-etc-modprobe-d\") pod \"tuned-dxttg\" (UID: \"84819fb3-13e8-482a-b88e-c30a2ef0bc3d\") " pod="openshift-cluster-node-tuning-operator/tuned-dxttg"
Sep 19 04:13:43.433729 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.432611    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1beab173-cbd6-4858-a590-c08e2d521ed9-host-run-ovn-kubernetes\") pod \"ovnkube-node-rtwn6\" (UID: \"1beab173-cbd6-4858-a590-c08e2d521ed9\") " pod="openshift-ovn-kubernetes/ovnkube-node-rtwn6"
Sep 19 04:13:43.433729 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.432720    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1beab173-cbd6-4858-a590-c08e2d521ed9-ovn-node-metrics-cert\") pod \"ovnkube-node-rtwn6\" (UID: \"1beab173-cbd6-4858-a590-c08e2d521ed9\") " pod="openshift-ovn-kubernetes/ovnkube-node-rtwn6"
Sep 19 04:13:43.433729 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.432747    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bcd5da2a-a24f-40fa-80aa-03ef54cc22a6-cni-binary-copy\") pod \"multus-additional-cni-plugins-wzbjr\" (UID: \"bcd5da2a-a24f-40fa-80aa-03ef54cc22a6\") " pod="openshift-multus/multus-additional-cni-plugins-wzbjr"
Sep 19 04:13:43.433729 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.432774    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/84819fb3-13e8-482a-b88e-c30a2ef0bc3d-etc-sysconfig\") pod \"tuned-dxttg\" (UID: \"84819fb3-13e8-482a-b88e-c30a2ef0bc3d\") " pod="openshift-cluster-node-tuning-operator/tuned-dxttg"
Sep 19 04:13:43.433729 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.432776    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/84819fb3-13e8-482a-b88e-c30a2ef0bc3d-etc-modprobe-d\") pod \"tuned-dxttg\" (UID: \"84819fb3-13e8-482a-b88e-c30a2ef0bc3d\") " pod="openshift-cluster-node-tuning-operator/tuned-dxttg"
Sep 19 04:13:43.433729 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.432799    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1beab173-cbd6-4858-a590-c08e2d521ed9-host-run-netns\") pod \"ovnkube-node-rtwn6\" (UID: \"1beab173-cbd6-4858-a590-c08e2d521ed9\") " pod="openshift-ovn-kubernetes/ovnkube-node-rtwn6"
Sep 19 04:13:43.434317 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.432829    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1beab173-cbd6-4858-a590-c08e2d521ed9-node-log\") pod \"ovnkube-node-rtwn6\" (UID: \"1beab173-cbd6-4858-a590-c08e2d521ed9\") " pod="openshift-ovn-kubernetes/ovnkube-node-rtwn6"
Sep 19 04:13:43.434317 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.432838    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1beab173-cbd6-4858-a590-c08e2d521ed9-host-run-netns\") pod \"ovnkube-node-rtwn6\" (UID: \"1beab173-cbd6-4858-a590-c08e2d521ed9\") " pod="openshift-ovn-kubernetes/ovnkube-node-rtwn6"
Sep 19 04:13:43.434317 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.432854    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1beab173-cbd6-4858-a590-c08e2d521ed9-ovnkube-config\") pod \"ovnkube-node-rtwn6\" (UID: \"1beab173-cbd6-4858-a590-c08e2d521ed9\") " pod="openshift-ovn-kubernetes/ovnkube-node-rtwn6"
Sep 19 04:13:43.434317 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.432824    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/84819fb3-13e8-482a-b88e-c30a2ef0bc3d-etc-sysconfig\") pod \"tuned-dxttg\" (UID: \"84819fb3-13e8-482a-b88e-c30a2ef0bc3d\") " pod="openshift-cluster-node-tuning-operator/tuned-dxttg"
Sep 19 04:13:43.434317 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.432877    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1beab173-cbd6-4858-a590-c08e2d521ed9-node-log\") pod \"ovnkube-node-rtwn6\" (UID: \"1beab173-cbd6-4858-a590-c08e2d521ed9\") " pod="openshift-ovn-kubernetes/ovnkube-node-rtwn6"
Sep 19 04:13:43.434317 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.432876    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t6w9\" (UniqueName: \"kubernetes.io/projected/bcd5da2a-a24f-40fa-80aa-03ef54cc22a6-kube-api-access-9t6w9\") pod \"multus-additional-cni-plugins-wzbjr\" (UID: \"bcd5da2a-a24f-40fa-80aa-03ef54cc22a6\") " pod="openshift-multus/multus-additional-cni-plugins-wzbjr"
Sep 19 04:13:43.434317 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.432917    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1beab173-cbd6-4858-a590-c08e2d521ed9-host-kubelet\") pod \"ovnkube-node-rtwn6\" (UID: \"1beab173-cbd6-4858-a590-c08e2d521ed9\") " pod="openshift-ovn-kubernetes/ovnkube-node-rtwn6"
Sep 19 04:13:43.434317 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.432947    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/84819fb3-13e8-482a-b88e-c30a2ef0bc3d-etc-tuned\") pod \"tuned-dxttg\" (UID: \"84819fb3-13e8-482a-b88e-c30a2ef0bc3d\") " pod="openshift-cluster-node-tuning-operator/tuned-dxttg"
Sep 19 04:13:43.434317 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.432967    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bcd5da2a-a24f-40fa-80aa-03ef54cc22a6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wzbjr\" (UID: \"bcd5da2a-a24f-40fa-80aa-03ef54cc22a6\") " pod="openshift-multus/multus-additional-cni-plugins-wzbjr"
Sep 19 04:13:43.434317 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.432972    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1beab173-cbd6-4858-a590-c08e2d521ed9-run-systemd\") pod \"ovnkube-node-rtwn6\" (UID: \"1beab173-cbd6-4858-a590-c08e2d521ed9\") " pod="openshift-ovn-kubernetes/ovnkube-node-rtwn6"
Sep 19 04:13:43.434317 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.433000    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1beab173-cbd6-4858-a590-c08e2d521ed9-run-systemd\") pod \"ovnkube-node-rtwn6\" (UID: \"1beab173-cbd6-4858-a590-c08e2d521ed9\") " pod="openshift-ovn-kubernetes/ovnkube-node-rtwn6"
Sep 19 04:13:43.434317 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.433024    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1beab173-cbd6-4858-a590-c08e2d521ed9-run-openvswitch\") pod \"ovnkube-node-rtwn6\" (UID: \"1beab173-cbd6-4858-a590-c08e2d521ed9\") " pod="openshift-ovn-kubernetes/ovnkube-node-rtwn6"
Sep 19 04:13:43.434317 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.433036    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1beab173-cbd6-4858-a590-c08e2d521ed9-host-kubelet\") pod \"ovnkube-node-rtwn6\" (UID: \"1beab173-cbd6-4858-a590-c08e2d521ed9\") " pod="openshift-ovn-kubernetes/ovnkube-node-rtwn6"
Sep 19 04:13:43.434317 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.433050    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1beab173-cbd6-4858-a590-c08e2d521ed9-host-cni-netd\") pod \"ovnkube-node-rtwn6\" (UID: \"1beab173-cbd6-4858-a590-c08e2d521ed9\") " pod="openshift-ovn-kubernetes/ovnkube-node-rtwn6"
Sep 19 04:13:43.434317 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.433079    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1beab173-cbd6-4858-a590-c08e2d521ed9-run-ovn\") pod \"ovnkube-node-rtwn6\" (UID: \"1beab173-cbd6-4858-a590-c08e2d521ed9\") " pod="openshift-ovn-kubernetes/ovnkube-node-rtwn6"
Sep 19 04:13:43.434317 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.433104    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1beab173-cbd6-4858-a590-c08e2d521ed9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rtwn6\" (UID: \"1beab173-cbd6-4858-a590-c08e2d521ed9\") " pod="openshift-ovn-kubernetes/ovnkube-node-rtwn6"
Sep 19 04:13:43.434317 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.433130    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82js7\" (UniqueName: \"kubernetes.io/projected/1beab173-cbd6-4858-a590-c08e2d521ed9-kube-api-access-82js7\") pod \"ovnkube-node-rtwn6\" (UID: \"1beab173-cbd6-4858-a590-c08e2d521ed9\") " pod="openshift-ovn-kubernetes/ovnkube-node-rtwn6"
Sep 19 04:13:43.434856 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.433134    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1beab173-cbd6-4858-a590-c08e2d521ed9-run-openvswitch\") pod \"ovnkube-node-rtwn6\" (UID: \"1beab173-cbd6-4858-a590-c08e2d521ed9\") " pod="openshift-ovn-kubernetes/ovnkube-node-rtwn6"
Sep 19 04:13:43.434856 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.433158    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bcd5da2a-a24f-40fa-80aa-03ef54cc22a6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wzbjr\" (UID: \"bcd5da2a-a24f-40fa-80aa-03ef54cc22a6\") " pod="openshift-multus/multus-additional-cni-plugins-wzbjr"
Sep 19 04:13:43.434856 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.433167    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1beab173-cbd6-4858-a590-c08e2d521ed9-run-ovn\") pod \"ovnkube-node-rtwn6\" (UID: \"1beab173-cbd6-4858-a590-c08e2d521ed9\") " pod="openshift-ovn-kubernetes/ovnkube-node-rtwn6"
Sep 19 04:13:43.434856 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.433173    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1beab173-cbd6-4858-a590-c08e2d521ed9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rtwn6\" (UID: \"1beab173-cbd6-4858-a590-c08e2d521ed9\") " pod="openshift-ovn-kubernetes/ovnkube-node-rtwn6"
Sep 19 04:13:43.434856 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.433184    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/84819fb3-13e8-482a-b88e-c30a2ef0bc3d-lib-modules\") pod \"tuned-dxttg\" (UID: \"84819fb3-13e8-482a-b88e-c30a2ef0bc3d\") " pod="openshift-cluster-node-tuning-operator/tuned-dxttg"
Sep 19 04:13:43.434856 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.433209    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1beab173-cbd6-4858-a590-c08e2d521ed9-ovnkube-script-lib\") pod \"ovnkube-node-rtwn6\" (UID: \"1beab173-cbd6-4858-a590-c08e2d521ed9\") " pod="openshift-ovn-kubernetes/ovnkube-node-rtwn6"
Sep 19 04:13:43.434856 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.433217    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1beab173-cbd6-4858-a590-c08e2d521ed9-host-cni-netd\") pod \"ovnkube-node-rtwn6\" (UID: \"1beab173-cbd6-4858-a590-c08e2d521ed9\") " pod="openshift-ovn-kubernetes/ovnkube-node-rtwn6"
Sep 19 04:13:43.434856 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.433238    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bcd5da2a-a24f-40fa-80aa-03ef54cc22a6-cni-binary-copy\") pod \"multus-additional-cni-plugins-wzbjr\" (UID: \"bcd5da2a-a24f-40fa-80aa-03ef54cc22a6\") " pod="openshift-multus/multus-additional-cni-plugins-wzbjr"
Sep 19 04:13:43.434856 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.433276    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1beab173-cbd6-4858-a590-c08e2d521ed9-ovnkube-config\") pod \"ovnkube-node-rtwn6\" (UID: \"1beab173-cbd6-4858-a590-c08e2d521ed9\") " pod="openshift-ovn-kubernetes/ovnkube-node-rtwn6"
Sep 19 04:13:43.434856 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.433335    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/84819fb3-13e8-482a-b88e-c30a2ef0bc3d-lib-modules\") pod \"tuned-dxttg\" (UID: \"84819fb3-13e8-482a-b88e-c30a2ef0bc3d\") " pod="openshift-cluster-node-tuning-operator/tuned-dxttg"
Sep 19 04:13:43.434856 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.434294    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bcd5da2a-a24f-40fa-80aa-03ef54cc22a6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wzbjr\" (UID: \"bcd5da2a-a24f-40fa-80aa-03ef54cc22a6\") " pod="openshift-multus/multus-additional-cni-plugins-wzbjr"
Sep 19 04:13:43.435492 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.435470    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1beab173-cbd6-4858-a590-c08e2d521ed9-ovnkube-script-lib\") pod \"ovnkube-node-rtwn6\" (UID: \"1beab173-cbd6-4858-a590-c08e2d521ed9\") " pod="openshift-ovn-kubernetes/ovnkube-node-rtwn6"
Sep 19 04:13:43.435605 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.435585    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f4b6c76b-7816-4e8b-9904-2f0cb5faca57-agent-certs\") pod \"konnectivity-agent-szpwc\" (UID: \"f4b6c76b-7816-4e8b-9904-2f0cb5faca57\") " pod="kube-system/konnectivity-agent-szpwc"
Sep 19 04:13:43.435757 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.435685    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/84819fb3-13e8-482a-b88e-c30a2ef0bc3d-tmp\") pod \"tuned-dxttg\" (UID: \"84819fb3-13e8-482a-b88e-c30a2ef0bc3d\") " pod="openshift-cluster-node-tuning-operator/tuned-dxttg"
Sep 19 04:13:43.435866 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.435850    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/84819fb3-13e8-482a-b88e-c30a2ef0bc3d-etc-tuned\") pod \"tuned-dxttg\" (UID: \"84819fb3-13e8-482a-b88e-c30a2ef0bc3d\") " pod="openshift-cluster-node-tuning-operator/tuned-dxttg"
Sep 19 04:13:43.435917 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.435903    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1beab173-cbd6-4858-a590-c08e2d521ed9-ovn-node-metrics-cert\") pod \"ovnkube-node-rtwn6\" (UID: \"1beab173-cbd6-4858-a590-c08e2d521ed9\") " pod="openshift-ovn-kubernetes/ovnkube-node-rtwn6"
Sep 19 04:13:43.451886 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.451858    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t6w9\" (UniqueName: \"kubernetes.io/projected/bcd5da2a-a24f-40fa-80aa-03ef54cc22a6-kube-api-access-9t6w9\") pod \"multus-additional-cni-plugins-wzbjr\" (UID: \"bcd5da2a-a24f-40fa-80aa-03ef54cc22a6\") " pod="openshift-multus/multus-additional-cni-plugins-wzbjr"
Sep 19 04:13:43.452772 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.452750    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjp4w\" (UniqueName: \"kubernetes.io/projected/84819fb3-13e8-482a-b88e-c30a2ef0bc3d-kube-api-access-wjp4w\") pod \"tuned-dxttg\" (UID: \"84819fb3-13e8-482a-b88e-c30a2ef0bc3d\") " pod="openshift-cluster-node-tuning-operator/tuned-dxttg"
Sep 19 04:13:43.452865 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.452802    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-82js7\" (UniqueName: \"kubernetes.io/projected/1beab173-cbd6-4858-a590-c08e2d521ed9-kube-api-access-82js7\") pod \"ovnkube-node-rtwn6\" (UID: \"1beab173-cbd6-4858-a590-c08e2d521ed9\") " pod="openshift-ovn-kubernetes/ovnkube-node-rtwn6"
Sep 19 04:13:43.511880 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.511852    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2ptq2"
Sep 19 04:13:43.520225 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.520203    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5bzkh"
Sep 19 04:13:43.531087 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.531065    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gnm2x"
Sep 19 04:13:43.536806 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.536784    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-nwgjm"
Sep 19 04:13:43.542917 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.542901    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kcb82"
Sep 19 04:13:43.551289 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.551268    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dxttg"
Sep 19 04:13:43.558942 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.558913    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-szpwc"
Sep 19 04:13:43.566535 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.566517    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rtwn6"
Sep 19 04:13:43.571770 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.571750    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wzbjr"
Sep 19 04:13:43.836166 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.836081    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3738f1c7-67af-4201-8e8d-1d0406e0edb9-metrics-certs\") pod \"network-metrics-daemon-wmk74\" (UID: \"3738f1c7-67af-4201-8e8d-1d0406e0edb9\") " pod="openshift-multus/network-metrics-daemon-wmk74"
Sep 19 04:13:43.836166 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.836150    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/64cb49fc-ede9-4369-aa8b-76a95d1edab6-original-pull-secret\") pod \"global-pull-secret-syncer-5t728\" (UID: \"64cb49fc-ede9-4369-aa8b-76a95d1edab6\") " pod="kube-system/global-pull-secret-syncer-5t728"
Sep 19 04:13:43.836384 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:43.836226    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 04:13:43.836384 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:43.836273    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 19 04:13:43.836384 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:43.836298    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3738f1c7-67af-4201-8e8d-1d0406e0edb9-metrics-certs podName:3738f1c7-67af-4201-8e8d-1d0406e0edb9 nodeName:}" failed. No retries permitted until 2026-09-19 04:13:44.836282673 +0000 UTC m=+4.019243804 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3738f1c7-67af-4201-8e8d-1d0406e0edb9-metrics-certs") pod "network-metrics-daemon-wmk74" (UID: "3738f1c7-67af-4201-8e8d-1d0406e0edb9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 04:13:43.836384 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:43.836329    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64cb49fc-ede9-4369-aa8b-76a95d1edab6-original-pull-secret podName:64cb49fc-ede9-4369-aa8b-76a95d1edab6 nodeName:}" failed. No retries permitted until 2026-09-19 04:13:44.836313006 +0000 UTC m=+4.019274146 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/64cb49fc-ede9-4369-aa8b-76a95d1edab6-original-pull-secret") pod "global-pull-secret-syncer-5t728" (UID: "64cb49fc-ede9-4369-aa8b-76a95d1edab6") : object "kube-system"/"original-pull-secret" not registered
Sep 19 04:13:43.937255 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:43.937219    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j856z\" (UniqueName: \"kubernetes.io/projected/f2fe8238-2a1d-4e7a-b4d7-4cc7d518f366-kube-api-access-j856z\") pod \"network-check-target-z7dgk\" (UID: \"f2fe8238-2a1d-4e7a-b4d7-4cc7d518f366\") " pod="openshift-network-diagnostics/network-check-target-z7dgk"
Sep 19 04:13:43.937427 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:43.937357    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 19 04:13:43.937427 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:43.937382    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 19 04:13:43.937427 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:43.937394    2579 projected.go:196] Error preparing data for projected volume kube-api-access-j856z for pod openshift-network-diagnostics/network-check-target-z7dgk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 04:13:43.937543 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:43.937456    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f2fe8238-2a1d-4e7a-b4d7-4cc7d518f366-kube-api-access-j856z podName:f2fe8238-2a1d-4e7a-b4d7-4cc7d518f366 nodeName:}" failed. No retries permitted until 2026-09-19 04:13:44.937442943 +0000 UTC m=+4.120404069 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-j856z" (UniqueName: "kubernetes.io/projected/f2fe8238-2a1d-4e7a-b4d7-4cc7d518f366-kube-api-access-j856z") pod "network-check-target-z7dgk" (UID: "f2fe8238-2a1d-4e7a-b4d7-4cc7d518f366") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 04:13:44.162770 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:44.162741    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4b6c76b_7816_4e8b_9904_2f0cb5faca57.slice/crio-696ee5a5032197139b831250d770e824f885ac7a6123eff86b156f4efb559d89 WatchSource:0}: Error finding container 696ee5a5032197139b831250d770e824f885ac7a6123eff86b156f4efb559d89: Status 404 returned error can't find the container with id 696ee5a5032197139b831250d770e824f885ac7a6123eff86b156f4efb559d89
Sep 19 04:13:44.163882 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:44.163858    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8d1a2a7_a76e_4741_be3a_d87124254f5d.slice/crio-cc9efab82a51b3b0f992859f6a4b5e005181359cee30b0bdb3b8682a0420d32d WatchSource:0}: Error finding container cc9efab82a51b3b0f992859f6a4b5e005181359cee30b0bdb3b8682a0420d32d: Status 404 returned error can't find the container with id cc9efab82a51b3b0f992859f6a4b5e005181359cee30b0bdb3b8682a0420d32d
Sep 19 04:13:44.164920 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:44.164902    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5194aa7_dba0_46dc_b3e9_91a4ea69e79e.slice/crio-92d71b79b458bf717d028297ab9be7681f43152cc118d25686e5883c75792277 WatchSource:0}: Error finding container 92d71b79b458bf717d028297ab9be7681f43152cc118d25686e5883c75792277: Status 404 returned error can't find the container with id 92d71b79b458bf717d028297ab9be7681f43152cc118d25686e5883c75792277
Sep 19 04:13:44.186522 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:44.186495    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcd5da2a_a24f_40fa_80aa_03ef54cc22a6.slice/crio-e2a79cde58b0c7f583289321d64ca646db1b1ec09229c6ae671950579bcd38e5 WatchSource:0}: Error finding container e2a79cde58b0c7f583289321d64ca646db1b1ec09229c6ae671950579bcd38e5: Status 404 returned error can't find the container with id e2a79cde58b0c7f583289321d64ca646db1b1ec09229c6ae671950579bcd38e5
Sep 19 04:13:44.187534 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:44.187508    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1beab173_cbd6_4858_a590_c08e2d521ed9.slice/crio-3921ee9864feac3ab279738f1e808ff30a31c34d4496e6b8106a35070ac733a9 WatchSource:0}: Error finding container 3921ee9864feac3ab279738f1e808ff30a31c34d4496e6b8106a35070ac733a9: Status 404 returned error can't find the container with id 3921ee9864feac3ab279738f1e808ff30a31c34d4496e6b8106a35070ac733a9
Sep 19 04:13:44.188466 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:44.188430    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64c734ac_0f91_4e8b_83e9_a2e9307b3bfe.slice/crio-d70b45c20dfb878f30f79fd6ba1d3b473465faab4abe1ea96015cc928e2818f0 WatchSource:0}: Error finding container d70b45c20dfb878f30f79fd6ba1d3b473465faab4abe1ea96015cc928e2818f0: Status 404 returned error can't find the container with id d70b45c20dfb878f30f79fd6ba1d3b473465faab4abe1ea96015cc928e2818f0
Sep 19 04:13:44.189418 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:44.189396    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84819fb3_13e8_482a_b88e_c30a2ef0bc3d.slice/crio-10846e77b4724ce54c1e7c5d25bc253760e912f7198e3a39ad9ff9391efc0625 WatchSource:0}: Error finding container 10846e77b4724ce54c1e7c5d25bc253760e912f7198e3a39ad9ff9391efc0625: Status 404 returned error can't find the container with id 10846e77b4724ce54c1e7c5d25bc253760e912f7198e3a39ad9ff9391efc0625
Sep 19 04:13:44.191203 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:44.191180    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc229f7a_0722_442f_940c_7ec542b81888.slice/crio-f6cb295d081c5eb750fc0e2388916e580389152005385722e6749aff5d4f2425 WatchSource:0}: Error finding container f6cb295d081c5eb750fc0e2388916e580389152005385722e6749aff5d4f2425: Status 404 returned error can't find the container with id f6cb295d081c5eb750fc0e2388916e580389152005385722e6749aff5d4f2425
Sep 19 04:13:44.193212 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:13:44.193193    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ece1d13_8010_47bb_8e97_40634dd06c5d.slice/crio-13a633e899d6b40c13895dd8c1a3406bd00a4de331d8c4ac27fd45c4e780eef2 WatchSource:0}: Error finding container 13a633e899d6b40c13895dd8c1a3406bd00a4de331d8c4ac27fd45c4e780eef2: Status 404 returned error can't find the container with id 13a633e899d6b40c13895dd8c1a3406bd00a4de331d8c4ac27fd45c4e780eef2
Sep 19 04:13:44.284413 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:44.284381    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-18 04:08:43 +0000 UTC" deadline="2028-05-26 01:07:06.716851557 +0000 UTC"
Sep 19 04:13:44.284413 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:44.284408    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14756h53m22.432445945s"
Sep 19 04:13:44.355091 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:44.355051    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-215.ec2.internal" event={"ID":"5cccb5a09b0064784f816095b5ca12fb","Type":"ContainerStarted","Data":"9677e4dcfc0aa2a31ac4e318fdedabb62481fa87c06bf10659ed308159dbce14"}
Sep 19 04:13:44.356213 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:44.356185    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kcb82" event={"ID":"4ece1d13-8010-47bb-8e97-40634dd06c5d","Type":"ContainerStarted","Data":"13a633e899d6b40c13895dd8c1a3406bd00a4de331d8c4ac27fd45c4e780eef2"}
Sep 19 04:13:44.357271 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:44.357251    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-nwgjm" event={"ID":"bc229f7a-0722-442f-940c-7ec542b81888","Type":"ContainerStarted","Data":"f6cb295d081c5eb750fc0e2388916e580389152005385722e6749aff5d4f2425"}
Sep 19 04:13:44.358446 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:44.358427    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rtwn6" event={"ID":"1beab173-cbd6-4858-a590-c08e2d521ed9","Type":"ContainerStarted","Data":"3921ee9864feac3ab279738f1e808ff30a31c34d4496e6b8106a35070ac733a9"}
Sep 19 04:13:44.359464 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:44.359443    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wzbjr" event={"ID":"bcd5da2a-a24f-40fa-80aa-03ef54cc22a6","Type":"ContainerStarted","Data":"e2a79cde58b0c7f583289321d64ca646db1b1ec09229c6ae671950579bcd38e5"}
Sep 19 04:13:44.360553 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:44.360535    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-szpwc" event={"ID":"f4b6c76b-7816-4e8b-9904-2f0cb5faca57","Type":"ContainerStarted","Data":"696ee5a5032197139b831250d770e824f885ac7a6123eff86b156f4efb559d89"}
Sep 19 04:13:44.364356 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:44.364317    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dxttg" event={"ID":"84819fb3-13e8-482a-b88e-c30a2ef0bc3d","Type":"ContainerStarted","Data":"10846e77b4724ce54c1e7c5d25bc253760e912f7198e3a39ad9ff9391efc0625"}
Sep 19 04:13:44.365558 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:44.365529    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gnm2x" event={"ID":"64c734ac-0f91-4e8b-83e9-a2e9307b3bfe","Type":"ContainerStarted","Data":"d70b45c20dfb878f30f79fd6ba1d3b473465faab4abe1ea96015cc928e2818f0"}
Sep 19 04:13:44.366747 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:44.366722    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2ptq2" event={"ID":"a5194aa7-dba0-46dc-b3e9-91a4ea69e79e","Type":"ContainerStarted","Data":"92d71b79b458bf717d028297ab9be7681f43152cc118d25686e5883c75792277"}
Sep 19 04:13:44.367995 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:44.367973    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5bzkh" event={"ID":"a8d1a2a7-a76e-4741-be3a-d87124254f5d","Type":"ContainerStarted","Data":"cc9efab82a51b3b0f992859f6a4b5e005181359cee30b0bdb3b8682a0420d32d"}
Sep 19 04:13:44.393096 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:44.393045    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-215.ec2.internal" podStartSLOduration=2.39303051 podStartE2EDuration="2.39303051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 04:13:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:13:44.392864964 +0000 UTC m=+3.575826112" watchObservedRunningTime="2026-09-19 04:13:44.39303051 +0000 UTC m=+3.575991658"
Sep 19 04:13:44.842670 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:44.842586    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3738f1c7-67af-4201-8e8d-1d0406e0edb9-metrics-certs\") pod \"network-metrics-daemon-wmk74\" (UID: \"3738f1c7-67af-4201-8e8d-1d0406e0edb9\") " pod="openshift-multus/network-metrics-daemon-wmk74"
Sep 19 04:13:44.842670 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:44.842659    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/64cb49fc-ede9-4369-aa8b-76a95d1edab6-original-pull-secret\") pod \"global-pull-secret-syncer-5t728\" (UID: \"64cb49fc-ede9-4369-aa8b-76a95d1edab6\") " pod="kube-system/global-pull-secret-syncer-5t728"
Sep 19 04:13:44.842916 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:44.842821    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 04:13:44.842916 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:44.842888    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3738f1c7-67af-4201-8e8d-1d0406e0edb9-metrics-certs podName:3738f1c7-67af-4201-8e8d-1d0406e0edb9 nodeName:}" failed. No retries permitted until 2026-09-19 04:13:46.842869101 +0000 UTC m=+6.025830230 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3738f1c7-67af-4201-8e8d-1d0406e0edb9-metrics-certs") pod "network-metrics-daemon-wmk74" (UID: "3738f1c7-67af-4201-8e8d-1d0406e0edb9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 04:13:44.842916 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:44.842821    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 19 04:13:44.843065 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:44.842930    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64cb49fc-ede9-4369-aa8b-76a95d1edab6-original-pull-secret podName:64cb49fc-ede9-4369-aa8b-76a95d1edab6 nodeName:}" failed. No retries permitted until 2026-09-19 04:13:46.84292 +0000 UTC m=+6.025881126 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/64cb49fc-ede9-4369-aa8b-76a95d1edab6-original-pull-secret") pod "global-pull-secret-syncer-5t728" (UID: "64cb49fc-ede9-4369-aa8b-76a95d1edab6") : object "kube-system"/"original-pull-secret" not registered
Sep 19 04:13:44.943226 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:44.943189    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j856z\" (UniqueName: \"kubernetes.io/projected/f2fe8238-2a1d-4e7a-b4d7-4cc7d518f366-kube-api-access-j856z\") pod \"network-check-target-z7dgk\" (UID: \"f2fe8238-2a1d-4e7a-b4d7-4cc7d518f366\") " pod="openshift-network-diagnostics/network-check-target-z7dgk"
Sep 19 04:13:44.943402 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:44.943384    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 19 04:13:44.943472 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:44.943413    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 19 04:13:44.943472 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:44.943427    2579 projected.go:196] Error preparing data for projected volume kube-api-access-j856z for pod openshift-network-diagnostics/network-check-target-z7dgk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 04:13:44.943573 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:44.943486    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f2fe8238-2a1d-4e7a-b4d7-4cc7d518f366-kube-api-access-j856z podName:f2fe8238-2a1d-4e7a-b4d7-4cc7d518f366 nodeName:}" failed. No retries permitted until 2026-09-19 04:13:46.943467976 +0000 UTC m=+6.126429108 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-j856z" (UniqueName: "kubernetes.io/projected/f2fe8238-2a1d-4e7a-b4d7-4cc7d518f366-kube-api-access-j856z") pod "network-check-target-z7dgk" (UID: "f2fe8238-2a1d-4e7a-b4d7-4cc7d518f366") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 04:13:45.285586 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:45.285277    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-18 04:08:43 +0000 UTC" deadline="2028-05-28 04:34:25.050938628 +0000 UTC"
Sep 19 04:13:45.285586 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:45.285543    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14808h20m39.76540183s"
Sep 19 04:13:45.345762 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:45.345713    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wmk74"
Sep 19 04:13:45.345919 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:45.345861    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wmk74" podUID="3738f1c7-67af-4201-8e8d-1d0406e0edb9"
Sep 19 04:13:45.346427 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:45.346247    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7dgk"
Sep 19 04:13:45.346427 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:45.346327    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7dgk" podUID="f2fe8238-2a1d-4e7a-b4d7-4cc7d518f366"
Sep 19 04:13:45.348595 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:45.348570    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5t728"
Sep 19 04:13:45.348723 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:45.348676    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5t728" podUID="64cb49fc-ede9-4369-aa8b-76a95d1edab6"
Sep 19 04:13:45.391912 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:45.390773    2579 generic.go:356] "Generic (PLEG): container finished" podID="56d744041b896c3555c7522e205ed4d0" containerID="5263eee0639509df5935d66c439ed10c3c196d2175f464fa0431c7f8b673a72f" exitCode=0
Sep 19 04:13:45.391912 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:45.391713    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-215.ec2.internal" event={"ID":"56d744041b896c3555c7522e205ed4d0","Type":"ContainerDied","Data":"5263eee0639509df5935d66c439ed10c3c196d2175f464fa0431c7f8b673a72f"}
Sep 19 04:13:46.431921 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:46.431881    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-215.ec2.internal" event={"ID":"56d744041b896c3555c7522e205ed4d0","Type":"ContainerStarted","Data":"a7dbd48d631e89a567f18969eed7ce9f846d52aaaf4557781066582929fbb17f"}
Sep 19 04:13:46.861646 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:46.860823    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3738f1c7-67af-4201-8e8d-1d0406e0edb9-metrics-certs\") pod \"network-metrics-daemon-wmk74\" (UID: \"3738f1c7-67af-4201-8e8d-1d0406e0edb9\") " pod="openshift-multus/network-metrics-daemon-wmk74"
Sep 19 04:13:46.861646 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:46.860889    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/64cb49fc-ede9-4369-aa8b-76a95d1edab6-original-pull-secret\") pod \"global-pull-secret-syncer-5t728\" (UID: \"64cb49fc-ede9-4369-aa8b-76a95d1edab6\") " pod="kube-system/global-pull-secret-syncer-5t728"
Sep 19 04:13:46.861646 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:46.861013    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 19 04:13:46.861646 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:46.861075    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64cb49fc-ede9-4369-aa8b-76a95d1edab6-original-pull-secret podName:64cb49fc-ede9-4369-aa8b-76a95d1edab6 nodeName:}" failed. No retries permitted until 2026-09-19 04:13:50.861055485 +0000 UTC m=+10.044016637 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/64cb49fc-ede9-4369-aa8b-76a95d1edab6-original-pull-secret") pod "global-pull-secret-syncer-5t728" (UID: "64cb49fc-ede9-4369-aa8b-76a95d1edab6") : object "kube-system"/"original-pull-secret" not registered
Sep 19 04:13:46.861646 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:46.861456    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 04:13:46.861646 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:46.861502    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3738f1c7-67af-4201-8e8d-1d0406e0edb9-metrics-certs podName:3738f1c7-67af-4201-8e8d-1d0406e0edb9 nodeName:}" failed. No retries permitted until 2026-09-19 04:13:50.861487499 +0000 UTC m=+10.044448628 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3738f1c7-67af-4201-8e8d-1d0406e0edb9-metrics-certs") pod "network-metrics-daemon-wmk74" (UID: "3738f1c7-67af-4201-8e8d-1d0406e0edb9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 04:13:46.962051 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:46.962010    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j856z\" (UniqueName: \"kubernetes.io/projected/f2fe8238-2a1d-4e7a-b4d7-4cc7d518f366-kube-api-access-j856z\") pod \"network-check-target-z7dgk\" (UID: \"f2fe8238-2a1d-4e7a-b4d7-4cc7d518f366\") " pod="openshift-network-diagnostics/network-check-target-z7dgk"
Sep 19 04:13:46.962211 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:46.962159    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 19 04:13:46.962211 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:46.962177    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 19 04:13:46.962211 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:46.962191    2579 projected.go:196] Error preparing data for projected volume kube-api-access-j856z for pod openshift-network-diagnostics/network-check-target-z7dgk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 04:13:46.962390 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:46.962250    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f2fe8238-2a1d-4e7a-b4d7-4cc7d518f366-kube-api-access-j856z podName:f2fe8238-2a1d-4e7a-b4d7-4cc7d518f366 nodeName:}" failed. No retries permitted until 2026-09-19 04:13:50.962231385 +0000 UTC m=+10.145192530 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-j856z" (UniqueName: "kubernetes.io/projected/f2fe8238-2a1d-4e7a-b4d7-4cc7d518f366-kube-api-access-j856z") pod "network-check-target-z7dgk" (UID: "f2fe8238-2a1d-4e7a-b4d7-4cc7d518f366") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 04:13:47.346237 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:47.346090    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wmk74"
Sep 19 04:13:47.346237 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:47.346146    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5t728"
Sep 19 04:13:47.346411 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:47.346239    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wmk74" podUID="3738f1c7-67af-4201-8e8d-1d0406e0edb9"
Sep 19 04:13:47.346502 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:47.346096    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7dgk"
Sep 19 04:13:47.346580 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:47.346547    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5t728" podUID="64cb49fc-ede9-4369-aa8b-76a95d1edab6"
Sep 19 04:13:47.346656 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:47.346638    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7dgk" podUID="f2fe8238-2a1d-4e7a-b4d7-4cc7d518f366"
Sep 19 04:13:49.345859 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:49.345821    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wmk74"
Sep 19 04:13:49.346426 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:49.345822    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7dgk"
Sep 19 04:13:49.346426 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:49.345934    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wmk74" podUID="3738f1c7-67af-4201-8e8d-1d0406e0edb9"
Sep 19 04:13:49.346426 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:49.345973    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5t728"
Sep 19 04:13:49.346426 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:49.346052    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7dgk" podUID="f2fe8238-2a1d-4e7a-b4d7-4cc7d518f366"
Sep 19 04:13:49.346426 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:49.346127    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5t728" podUID="64cb49fc-ede9-4369-aa8b-76a95d1edab6"
Sep 19 04:13:50.903308 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:50.903250    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3738f1c7-67af-4201-8e8d-1d0406e0edb9-metrics-certs\") pod \"network-metrics-daemon-wmk74\" (UID: \"3738f1c7-67af-4201-8e8d-1d0406e0edb9\") " pod="openshift-multus/network-metrics-daemon-wmk74"
Sep 19 04:13:50.903791 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:50.903318    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/64cb49fc-ede9-4369-aa8b-76a95d1edab6-original-pull-secret\") pod \"global-pull-secret-syncer-5t728\" (UID: \"64cb49fc-ede9-4369-aa8b-76a95d1edab6\") " pod="kube-system/global-pull-secret-syncer-5t728"
Sep 19 04:13:50.903791 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:50.903379    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 04:13:50.903791 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:50.903421    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 19 04:13:50.903791 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:50.903453    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3738f1c7-67af-4201-8e8d-1d0406e0edb9-metrics-certs podName:3738f1c7-67af-4201-8e8d-1d0406e0edb9 nodeName:}" failed. No retries permitted until 2026-09-19 04:13:58.903436808 +0000 UTC m=+18.086397956 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3738f1c7-67af-4201-8e8d-1d0406e0edb9-metrics-certs") pod "network-metrics-daemon-wmk74" (UID: "3738f1c7-67af-4201-8e8d-1d0406e0edb9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 04:13:50.903791 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:50.903482    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64cb49fc-ede9-4369-aa8b-76a95d1edab6-original-pull-secret podName:64cb49fc-ede9-4369-aa8b-76a95d1edab6 nodeName:}" failed. No retries permitted until 2026-09-19 04:13:58.903463113 +0000 UTC m=+18.086424250 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/64cb49fc-ede9-4369-aa8b-76a95d1edab6-original-pull-secret") pod "global-pull-secret-syncer-5t728" (UID: "64cb49fc-ede9-4369-aa8b-76a95d1edab6") : object "kube-system"/"original-pull-secret" not registered
Sep 19 04:13:51.003836 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:51.003802    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j856z\" (UniqueName: \"kubernetes.io/projected/f2fe8238-2a1d-4e7a-b4d7-4cc7d518f366-kube-api-access-j856z\") pod \"network-check-target-z7dgk\" (UID: \"f2fe8238-2a1d-4e7a-b4d7-4cc7d518f366\") " pod="openshift-network-diagnostics/network-check-target-z7dgk"
Sep 19 04:13:51.004008 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:51.003974    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 19 04:13:51.004008 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:51.003993    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 19 04:13:51.004008 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:51.004005    2579 projected.go:196] Error preparing data for projected volume kube-api-access-j856z for pod openshift-network-diagnostics/network-check-target-z7dgk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 04:13:51.004130 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:51.004059    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f2fe8238-2a1d-4e7a-b4d7-4cc7d518f366-kube-api-access-j856z podName:f2fe8238-2a1d-4e7a-b4d7-4cc7d518f366 nodeName:}" failed. No retries permitted until 2026-09-19 04:13:59.004042331 +0000 UTC m=+18.187003472 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-j856z" (UniqueName: "kubernetes.io/projected/f2fe8238-2a1d-4e7a-b4d7-4cc7d518f366-kube-api-access-j856z") pod "network-check-target-z7dgk" (UID: "f2fe8238-2a1d-4e7a-b4d7-4cc7d518f366") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 04:13:51.346870 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:51.346786    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wmk74"
Sep 19 04:13:51.347066 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:51.346926    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wmk74" podUID="3738f1c7-67af-4201-8e8d-1d0406e0edb9"
Sep 19 04:13:51.347310 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:51.347292    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7dgk"
Sep 19 04:13:51.347405 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:51.347383    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7dgk" podUID="f2fe8238-2a1d-4e7a-b4d7-4cc7d518f366"
Sep 19 04:13:51.348842 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:51.348813    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5t728"
Sep 19 04:13:51.348967 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:51.348904    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5t728" podUID="64cb49fc-ede9-4369-aa8b-76a95d1edab6"
Sep 19 04:13:53.345524 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:53.345461    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7dgk"
Sep 19 04:13:53.345982 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:53.345526    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5t728"
Sep 19 04:13:53.345982 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:53.345630    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wmk74"
Sep 19 04:13:53.345982 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:53.345652    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7dgk" podUID="f2fe8238-2a1d-4e7a-b4d7-4cc7d518f366"
Sep 19 04:13:53.345982 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:53.345776    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wmk74" podUID="3738f1c7-67af-4201-8e8d-1d0406e0edb9"
Sep 19 04:13:53.345982 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:53.345834    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5t728" podUID="64cb49fc-ede9-4369-aa8b-76a95d1edab6"
Sep 19 04:13:55.348625 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:55.348589    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5t728"
Sep 19 04:13:55.349087 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:55.348634    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wmk74"
Sep 19 04:13:55.349087 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:55.348719    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7dgk"
Sep 19 04:13:55.349087 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:55.348737    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5t728" podUID="64cb49fc-ede9-4369-aa8b-76a95d1edab6"
Sep 19 04:13:55.349087 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:55.348787    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7dgk" podUID="f2fe8238-2a1d-4e7a-b4d7-4cc7d518f366"
Sep 19 04:13:55.349087 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:55.348904    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wmk74" podUID="3738f1c7-67af-4201-8e8d-1d0406e0edb9"
Sep 19 04:13:57.346179 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:57.345514    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5t728"
Sep 19 04:13:57.346179 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:57.345643    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5t728" podUID="64cb49fc-ede9-4369-aa8b-76a95d1edab6"
Sep 19 04:13:57.346179 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:57.345529    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7dgk"
Sep 19 04:13:57.346179 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:57.345746    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7dgk" podUID="f2fe8238-2a1d-4e7a-b4d7-4cc7d518f366"
Sep 19 04:13:57.346179 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:57.345522    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wmk74"
Sep 19 04:13:57.346179 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:57.345822    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wmk74" podUID="3738f1c7-67af-4201-8e8d-1d0406e0edb9"
Sep 19 04:13:58.964085 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:58.964046    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3738f1c7-67af-4201-8e8d-1d0406e0edb9-metrics-certs\") pod \"network-metrics-daemon-wmk74\" (UID: \"3738f1c7-67af-4201-8e8d-1d0406e0edb9\") " pod="openshift-multus/network-metrics-daemon-wmk74"
Sep 19 04:13:58.964614 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:58.964102    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/64cb49fc-ede9-4369-aa8b-76a95d1edab6-original-pull-secret\") pod \"global-pull-secret-syncer-5t728\" (UID: \"64cb49fc-ede9-4369-aa8b-76a95d1edab6\") " pod="kube-system/global-pull-secret-syncer-5t728"
Sep 19 04:13:58.964614 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:58.964202    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 19 04:13:58.964614 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:58.964209    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 04:13:58.964614 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:58.964277    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64cb49fc-ede9-4369-aa8b-76a95d1edab6-original-pull-secret podName:64cb49fc-ede9-4369-aa8b-76a95d1edab6 nodeName:}" failed. No retries permitted until 2026-09-19 04:14:14.964257611 +0000 UTC m=+34.147218777 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/64cb49fc-ede9-4369-aa8b-76a95d1edab6-original-pull-secret") pod "global-pull-secret-syncer-5t728" (UID: "64cb49fc-ede9-4369-aa8b-76a95d1edab6") : object "kube-system"/"original-pull-secret" not registered
Sep 19 04:13:58.964614 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:58.964301    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3738f1c7-67af-4201-8e8d-1d0406e0edb9-metrics-certs podName:3738f1c7-67af-4201-8e8d-1d0406e0edb9 nodeName:}" failed. No retries permitted until 2026-09-19 04:14:14.964287982 +0000 UTC m=+34.147249108 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3738f1c7-67af-4201-8e8d-1d0406e0edb9-metrics-certs") pod "network-metrics-daemon-wmk74" (UID: "3738f1c7-67af-4201-8e8d-1d0406e0edb9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 04:13:59.064638 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:59.064575    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j856z\" (UniqueName: \"kubernetes.io/projected/f2fe8238-2a1d-4e7a-b4d7-4cc7d518f366-kube-api-access-j856z\") pod \"network-check-target-z7dgk\" (UID: \"f2fe8238-2a1d-4e7a-b4d7-4cc7d518f366\") " pod="openshift-network-diagnostics/network-check-target-z7dgk"
Sep 19 04:13:59.064830 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:59.064774    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 19 04:13:59.064830 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:59.064796    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 19 04:13:59.064830 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:59.064805    2579 projected.go:196] Error preparing data for projected volume kube-api-access-j856z for pod openshift-network-diagnostics/network-check-target-z7dgk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 04:13:59.064995 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:59.064864    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f2fe8238-2a1d-4e7a-b4d7-4cc7d518f366-kube-api-access-j856z podName:f2fe8238-2a1d-4e7a-b4d7-4cc7d518f366 nodeName:}" failed. No retries permitted until 2026-09-19 04:14:15.064847264 +0000 UTC m=+34.247808389 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-j856z" (UniqueName: "kubernetes.io/projected/f2fe8238-2a1d-4e7a-b4d7-4cc7d518f366-kube-api-access-j856z") pod "network-check-target-z7dgk" (UID: "f2fe8238-2a1d-4e7a-b4d7-4cc7d518f366") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 04:13:59.349077 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:59.348985    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5t728"
Sep 19 04:13:59.349077 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:59.349009    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7dgk"
Sep 19 04:13:59.349077 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:13:59.349020    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wmk74"
Sep 19 04:13:59.349326 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:59.349117    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5t728" podUID="64cb49fc-ede9-4369-aa8b-76a95d1edab6"
Sep 19 04:13:59.349326 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:59.349190    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7dgk" podUID="f2fe8238-2a1d-4e7a-b4d7-4cc7d518f366"
Sep 19 04:13:59.349326 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:13:59.349282    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wmk74" podUID="3738f1c7-67af-4201-8e8d-1d0406e0edb9"
Sep 19 04:14:01.348732 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:01.348681    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7dgk"
Sep 19 04:14:01.349115 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:01.348789    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5t728"
Sep 19 04:14:01.349115 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:01.348802    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7dgk" podUID="f2fe8238-2a1d-4e7a-b4d7-4cc7d518f366"
Sep 19 04:14:01.349115 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:01.348816    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wmk74"
Sep 19 04:14:01.349115 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:01.348888    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5t728" podUID="64cb49fc-ede9-4369-aa8b-76a95d1edab6"
Sep 19 04:14:01.349115 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:01.348970    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wmk74" podUID="3738f1c7-67af-4201-8e8d-1d0406e0edb9"
Sep 19 04:14:03.348728 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:03.348536    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5t728"
Sep 19 04:14:03.349518 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:03.348602    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7dgk"
Sep 19 04:14:03.349518 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:03.348813    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5t728" podUID="64cb49fc-ede9-4369-aa8b-76a95d1edab6"
Sep 19 04:14:03.349518 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:03.348605    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wmk74"
Sep 19 04:14:03.349518 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:03.348876    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7dgk" podUID="f2fe8238-2a1d-4e7a-b4d7-4cc7d518f366"
Sep 19 04:14:03.349518 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:03.348934    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wmk74" podUID="3738f1c7-67af-4201-8e8d-1d0406e0edb9"
Sep 19 04:14:03.467071 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:03.466981    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2ptq2" event={"ID":"a5194aa7-dba0-46dc-b3e9-91a4ea69e79e","Type":"ContainerStarted","Data":"6bb862d8512c17f78f099dde3f64d2b996d0e930052d100218eb49e1ed8fc029"}
Sep 19 04:14:03.468675 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:03.468649    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5bzkh" event={"ID":"a8d1a2a7-a76e-4741-be3a-d87124254f5d","Type":"ContainerStarted","Data":"39a88d0ce1abc1f35604e59d11e3165073bfb7f0fe8bc43a8b4f04cd5cf972c5"}
Sep 19 04:14:03.470538 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:03.470491    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-nwgjm" event={"ID":"bc229f7a-0722-442f-940c-7ec542b81888","Type":"ContainerStarted","Data":"ad42eadd1aecb0642c2bc684ad6e9e5c7395121947bb9d02b6a8951c09df5879"}
Sep 19 04:14:03.474013 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:03.473987    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rtwn6" event={"ID":"1beab173-cbd6-4858-a590-c08e2d521ed9","Type":"ContainerStarted","Data":"43eb516d38c9165b2bfa4c8130f857f964f84d4ec33367b2d1ea213954542c1d"}
Sep 19 04:14:03.474110 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:03.474066    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rtwn6" event={"ID":"1beab173-cbd6-4858-a590-c08e2d521ed9","Type":"ContainerStarted","Data":"f9a7cdb96a3d20c7ea663c0288c831f939183daa09b4bbb1007dfdc7cc22da23"}
Sep 19 04:14:03.474167 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:03.474158    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rtwn6" event={"ID":"1beab173-cbd6-4858-a590-c08e2d521ed9","Type":"ContainerStarted","Data":"98cb829a41c3a8019fec0fcf02d7ae13d337aef6af9a9273109edff234afe499"}
Sep 19 04:14:03.474223 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:03.474205    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rtwn6" event={"ID":"1beab173-cbd6-4858-a590-c08e2d521ed9","Type":"ContainerStarted","Data":"6a08084d8b674edddacb551836392de8fc2012d48dde3e99d27d68e647c9028e"}
Sep 19 04:14:03.474276 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:03.474250    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rtwn6" event={"ID":"1beab173-cbd6-4858-a590-c08e2d521ed9","Type":"ContainerStarted","Data":"686f92177d5a59a31ad612c496ba3eecabdd286b0cf6681bd487d9f6d5e80e29"}
Sep 19 04:14:03.474330 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:03.474297    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rtwn6" event={"ID":"1beab173-cbd6-4858-a590-c08e2d521ed9","Type":"ContainerStarted","Data":"caefe794ff854d6d0070528d6cc7e5101881c7af01cfc5aa22fadce471a9cad0"}
Sep 19 04:14:03.476063 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:03.476039    2579 generic.go:356] "Generic (PLEG): container finished" podID="bcd5da2a-a24f-40fa-80aa-03ef54cc22a6" containerID="344321313a415b89e3193727de0cb49d677841f400f1cfbd5f86babdd5a61b2f" exitCode=0
Sep 19 04:14:03.476184 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:03.476124    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wzbjr" event={"ID":"bcd5da2a-a24f-40fa-80aa-03ef54cc22a6","Type":"ContainerDied","Data":"344321313a415b89e3193727de0cb49d677841f400f1cfbd5f86babdd5a61b2f"}
Sep 19 04:14:03.478479 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:03.478209    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-szpwc" event={"ID":"f4b6c76b-7816-4e8b-9904-2f0cb5faca57","Type":"ContainerStarted","Data":"0b39fc78b7505c503d7270e0efb4b87931986251d3b181d22083233adaa0a3b8"}
Sep 19 04:14:03.479877 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:03.479856    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dxttg" event={"ID":"84819fb3-13e8-482a-b88e-c30a2ef0bc3d","Type":"ContainerStarted","Data":"2761d3712411dd280d90728307379ab88f192f54c74d05534db9d8c50601f5dd"}
Sep 19 04:14:03.481901 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:03.481876    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gnm2x" event={"ID":"64c734ac-0f91-4e8b-83e9-a2e9307b3bfe","Type":"ContainerStarted","Data":"b17cde0b9417d6b716825012868b454bc17bcd6022240e97fceb5e3089a7a427"}
Sep 19 04:14:03.503513 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:03.503456    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-2ptq2" podStartSLOduration=4.105245135 podStartE2EDuration="22.503438152s" totalImagesPullingTime="18.398193017s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 04:13:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:14:03.503289687 +0000 UTC m=+22.686250833" watchObservedRunningTime="2026-09-19 04:14:03.503438152 +0000 UTC m=+22.686399301"
Sep 19 04:14:03.503680 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:03.503603    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-215.ec2.internal" podStartSLOduration=21.503593325 podStartE2EDuration="21.503593325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 04:13:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:13:46.451673729 +0000 UTC m=+5.634634877" watchObservedRunningTime="2026-09-19 04:14:03.503593325 +0000 UTC m=+22.686554474"
Sep 19 04:14:03.553781 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:03.553732    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-nwgjm" podStartSLOduration=4.068020857 podStartE2EDuration="22.553719339s" totalImagesPullingTime="18.485698482s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 04:13:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:14:03.55359489 +0000 UTC m=+22.736556041" watchObservedRunningTime="2026-09-19 04:14:03.553719339 +0000 UTC m=+22.736680484"
Sep 19 04:14:03.630849 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:03.630638    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-szpwc" podStartSLOduration=8.34458653 podStartE2EDuration="22.630620268s" totalImagesPullingTime="14.286033738s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 04:13:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:14:03.58850714 +0000 UTC m=+22.771468288" watchObservedRunningTime="2026-09-19 04:14:03.630620268 +0000 UTC m=+22.813581416"
Sep 19 04:14:03.672403 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:03.672345    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-5bzkh" podStartSLOduration=4.223305092 podStartE2EDuration="22.672327371s" totalImagesPullingTime="18.449022279s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 04:13:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:14:03.631190972 +0000 UTC m=+22.814152121" watchObservedRunningTime="2026-09-19 04:14:03.672327371 +0000 UTC m=+22.855288519"
Sep 19 04:14:03.747878 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:03.747745    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-dxttg" podStartSLOduration=4.306210388 podStartE2EDuration="22.747726896s" totalImagesPullingTime="18.441516508s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 04:13:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:14:03.672774152 +0000 UTC m=+22.855735300" watchObservedRunningTime="2026-09-19 04:14:03.747726896 +0000 UTC m=+22.930688045"
Sep 19 04:14:04.270440 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:04.270406    2579 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 19 04:14:04.279604 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:04.279519    2579 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-19T04:14:04.270433864Z","UUID":"e2da44e8-3f56-4207-95a7-b5d0b56fb9b1","Handler":null,"Name":"","Endpoint":""}
Sep 19 04:14:04.281071 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:04.281046    2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 19 04:14:04.281169 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:04.281081    2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 19 04:14:04.487123 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:04.487085    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gnm2x" event={"ID":"64c734ac-0f91-4e8b-83e9-a2e9307b3bfe","Type":"ContainerStarted","Data":"0c0d80d610d9cae2004ed8ea0d554259917e56027a3a0511cd774a9c7929c510"}
Sep 19 04:14:04.489674 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:04.489650    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kcb82" event={"ID":"4ece1d13-8010-47bb-8e97-40634dd06c5d","Type":"ContainerStarted","Data":"2e2889a421a5753bb52f923dd4febceae0001d20852aff91a4c0f060e2ca512c"}
Sep 19 04:14:04.521320 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:04.521270    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-kcb82" podStartSLOduration=5.083074229 podStartE2EDuration="23.521255641s" totalImagesPullingTime="18.438181412s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 04:13:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:14:04.521030307 +0000 UTC m=+23.703991456" watchObservedRunningTime="2026-09-19 04:14:04.521255641 +0000 UTC m=+23.704216788"
Sep 19 04:14:05.345555 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:05.345529    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7dgk"
Sep 19 04:14:05.345724 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:05.345530    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5t728"
Sep 19 04:14:05.345724 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:05.345654    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7dgk" podUID="f2fe8238-2a1d-4e7a-b4d7-4cc7d518f366"
Sep 19 04:14:05.345724 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:05.345662    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wmk74"
Sep 19 04:14:05.345878 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:05.345789    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5t728" podUID="64cb49fc-ede9-4369-aa8b-76a95d1edab6"
Sep 19 04:14:05.345932 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:05.345906    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wmk74" podUID="3738f1c7-67af-4201-8e8d-1d0406e0edb9"
Sep 19 04:14:05.495853 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:05.495809    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rtwn6" event={"ID":"1beab173-cbd6-4858-a590-c08e2d521ed9","Type":"ContainerStarted","Data":"0fe6b90a6bad7865caadbfad0b9fb03006531dc3ca3d21dd5e80222bda1ba8ea"}
Sep 19 04:14:05.498181 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:05.498138    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gnm2x" event={"ID":"64c734ac-0f91-4e8b-83e9-a2e9307b3bfe","Type":"ContainerStarted","Data":"898dd320a5ee04948004b81fee2814abfa786d455513b29a4f3a383838d9624c"}
Sep 19 04:14:06.457430 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:06.457396    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-szpwc"
Sep 19 04:14:06.458018 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:06.457995    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-szpwc"
Sep 19 04:14:06.501191 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:06.501155    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-szpwc"
Sep 19 04:14:06.501684 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:06.501666    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-szpwc"
Sep 19 04:14:06.520012 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:06.518513    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gnm2x" podStartSLOduration=4.940835349 podStartE2EDuration="25.518488654s" totalImagesPullingTime="20.577653305s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 04:13:41 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:14:05.525944485 +0000 UTC m=+24.708905634" watchObservedRunningTime="2026-09-19 04:14:06.518488654 +0000 UTC m=+25.701449804"
Sep 19 04:14:07.345100 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:07.345067    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wmk74"
Sep 19 04:14:07.345100 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:07.345089    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7dgk"
Sep 19 04:14:07.345318 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:07.345067    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5t728"
Sep 19 04:14:07.345318 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:07.345192    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wmk74" podUID="3738f1c7-67af-4201-8e8d-1d0406e0edb9"
Sep 19 04:14:07.345318 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:07.345302    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5t728" podUID="64cb49fc-ede9-4369-aa8b-76a95d1edab6"
Sep 19 04:14:07.345420 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:07.345359    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7dgk" podUID="f2fe8238-2a1d-4e7a-b4d7-4cc7d518f366"
Sep 19 04:14:08.508952 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:08.508685    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rtwn6" event={"ID":"1beab173-cbd6-4858-a590-c08e2d521ed9","Type":"ContainerStarted","Data":"d111a52b2b33b1abfaff2913949dde08fac8964e1d7125e3d7906627e6de1621"}
Sep 19 04:14:08.509839 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:08.509005    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rtwn6"
Sep 19 04:14:08.510831 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:08.510806    2579 generic.go:356] "Generic (PLEG): container finished" podID="bcd5da2a-a24f-40fa-80aa-03ef54cc22a6" containerID="379a1bb833f3d5046675542c995fcce8fdb64c8ac9f50f8e10d5fe971d131f8d" exitCode=0
Sep 19 04:14:08.510928 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:08.510835    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wzbjr" event={"ID":"bcd5da2a-a24f-40fa-80aa-03ef54cc22a6","Type":"ContainerDied","Data":"379a1bb833f3d5046675542c995fcce8fdb64c8ac9f50f8e10d5fe971d131f8d"}
Sep 19 04:14:08.525282 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:08.525266    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rtwn6"
Sep 19 04:14:08.581453 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:08.581404    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-rtwn6" podStartSLOduration=9.080806933 podStartE2EDuration="27.581390312s" totalImagesPullingTime="18.500583379s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 04:13:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:14:08.581176023 +0000 UTC m=+27.764137172" watchObservedRunningTime="2026-09-19 04:14:08.581390312 +0000 UTC m=+27.764351460"
Sep 19 04:14:09.345087 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:09.345046    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7dgk"
Sep 19 04:14:09.345326 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:09.345046    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wmk74"
Sep 19 04:14:09.345326 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:09.345179    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7dgk" podUID="f2fe8238-2a1d-4e7a-b4d7-4cc7d518f366"
Sep 19 04:14:09.345326 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:09.345046    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5t728"
Sep 19 04:14:09.345326 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:09.345265    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wmk74" podUID="3738f1c7-67af-4201-8e8d-1d0406e0edb9"
Sep 19 04:14:09.345326 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:09.345312    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5t728" podUID="64cb49fc-ede9-4369-aa8b-76a95d1edab6"
Sep 19 04:14:09.513173 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:09.513145    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rtwn6"
Sep 19 04:14:09.513610 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:09.513229    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rtwn6"
Sep 19 04:14:09.529401 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:09.529376    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rtwn6"
Sep 19 04:14:10.089515 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:10.089322    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wmk74"]
Sep 19 04:14:10.089677 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:10.089602    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wmk74"
Sep 19 04:14:10.089733 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:10.089711    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wmk74" podUID="3738f1c7-67af-4201-8e8d-1d0406e0edb9"
Sep 19 04:14:10.097196 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:10.097170    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5t728"]
Sep 19 04:14:10.097319 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:10.097253    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5t728"
Sep 19 04:14:10.097371 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:10.097320    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5t728" podUID="64cb49fc-ede9-4369-aa8b-76a95d1edab6"
Sep 19 04:14:10.123134 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:10.123049    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-z7dgk"]
Sep 19 04:14:10.123255 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:10.123154    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7dgk"
Sep 19 04:14:10.123304 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:10.123256    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7dgk" podUID="f2fe8238-2a1d-4e7a-b4d7-4cc7d518f366"
Sep 19 04:14:10.516825 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:10.516792    2579 generic.go:356] "Generic (PLEG): container finished" podID="bcd5da2a-a24f-40fa-80aa-03ef54cc22a6" containerID="dbc9cd146c5601f006c29ce3da36a44c63fc9908d5de464373b0557a13b57876" exitCode=0
Sep 19 04:14:10.517191 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:10.516883    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wzbjr" event={"ID":"bcd5da2a-a24f-40fa-80aa-03ef54cc22a6","Type":"ContainerDied","Data":"dbc9cd146c5601f006c29ce3da36a44c63fc9908d5de464373b0557a13b57876"}
Sep 19 04:14:11.348316 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:11.348287    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5t728"
Sep 19 04:14:11.348513 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:11.348384    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5t728" podUID="64cb49fc-ede9-4369-aa8b-76a95d1edab6"
Sep 19 04:14:11.348513 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:11.348437    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wmk74"
Sep 19 04:14:11.348596 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:11.348532    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wmk74" podUID="3738f1c7-67af-4201-8e8d-1d0406e0edb9"
Sep 19 04:14:12.345772 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:12.345742    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7dgk"
Sep 19 04:14:12.346116 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:12.345841    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7dgk" podUID="f2fe8238-2a1d-4e7a-b4d7-4cc7d518f366"
Sep 19 04:14:12.524992 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:12.524899    2579 generic.go:356] "Generic (PLEG): container finished" podID="bcd5da2a-a24f-40fa-80aa-03ef54cc22a6" containerID="b0471483238fe75e276e90ad1b4013e0106ec24d82b8e600df272c82422ff983" exitCode=0
Sep 19 04:14:12.524992 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:12.524954    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wzbjr" event={"ID":"bcd5da2a-a24f-40fa-80aa-03ef54cc22a6","Type":"ContainerDied","Data":"b0471483238fe75e276e90ad1b4013e0106ec24d82b8e600df272c82422ff983"}
Sep 19 04:14:13.345749 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:13.345714    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5t728"
Sep 19 04:14:13.345749 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:13.345741    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wmk74"
Sep 19 04:14:13.346235 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:13.345838    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5t728" podUID="64cb49fc-ede9-4369-aa8b-76a95d1edab6"
Sep 19 04:14:13.346235 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:13.345956    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wmk74" podUID="3738f1c7-67af-4201-8e8d-1d0406e0edb9"
Sep 19 04:14:14.345197 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:14.345119    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7dgk"
Sep 19 04:14:14.345345 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:14.345253    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7dgk" podUID="f2fe8238-2a1d-4e7a-b4d7-4cc7d518f366"
Sep 19 04:14:14.977199 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:14.977161    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3738f1c7-67af-4201-8e8d-1d0406e0edb9-metrics-certs\") pod \"network-metrics-daemon-wmk74\" (UID: \"3738f1c7-67af-4201-8e8d-1d0406e0edb9\") " pod="openshift-multus/network-metrics-daemon-wmk74"
Sep 19 04:14:14.977728 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:14.977231    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/64cb49fc-ede9-4369-aa8b-76a95d1edab6-original-pull-secret\") pod \"global-pull-secret-syncer-5t728\" (UID: \"64cb49fc-ede9-4369-aa8b-76a95d1edab6\") " pod="kube-system/global-pull-secret-syncer-5t728"
Sep 19 04:14:14.977728 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:14.977296    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 04:14:14.977728 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:14.977363    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 19 04:14:14.977728 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:14.977368    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3738f1c7-67af-4201-8e8d-1d0406e0edb9-metrics-certs podName:3738f1c7-67af-4201-8e8d-1d0406e0edb9 nodeName:}" failed. No retries permitted until 2026-09-19 04:14:46.977346829 +0000 UTC m=+66.160307973 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3738f1c7-67af-4201-8e8d-1d0406e0edb9-metrics-certs") pod "network-metrics-daemon-wmk74" (UID: "3738f1c7-67af-4201-8e8d-1d0406e0edb9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 04:14:14.977728 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:14.977435    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64cb49fc-ede9-4369-aa8b-76a95d1edab6-original-pull-secret podName:64cb49fc-ede9-4369-aa8b-76a95d1edab6 nodeName:}" failed. No retries permitted until 2026-09-19 04:14:46.977416373 +0000 UTC m=+66.160377503 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/64cb49fc-ede9-4369-aa8b-76a95d1edab6-original-pull-secret") pod "global-pull-secret-syncer-5t728" (UID: "64cb49fc-ede9-4369-aa8b-76a95d1edab6") : object "kube-system"/"original-pull-secret" not registered
Sep 19 04:14:15.077974 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:15.077941    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j856z\" (UniqueName: \"kubernetes.io/projected/f2fe8238-2a1d-4e7a-b4d7-4cc7d518f366-kube-api-access-j856z\") pod \"network-check-target-z7dgk\" (UID: \"f2fe8238-2a1d-4e7a-b4d7-4cc7d518f366\") " pod="openshift-network-diagnostics/network-check-target-z7dgk"
Sep 19 04:14:15.078172 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:15.078146    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 19 04:14:15.078238 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:15.078182    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 19 04:14:15.078238 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:15.078196    2579 projected.go:196] Error preparing data for projected volume kube-api-access-j856z for pod openshift-network-diagnostics/network-check-target-z7dgk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 04:14:15.078329 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:15.078252    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f2fe8238-2a1d-4e7a-b4d7-4cc7d518f366-kube-api-access-j856z podName:f2fe8238-2a1d-4e7a-b4d7-4cc7d518f366 nodeName:}" failed. No retries permitted until 2026-09-19 04:14:47.078239324 +0000 UTC m=+66.261200462 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-j856z" (UniqueName: "kubernetes.io/projected/f2fe8238-2a1d-4e7a-b4d7-4cc7d518f366-kube-api-access-j856z") pod "network-check-target-z7dgk" (UID: "f2fe8238-2a1d-4e7a-b4d7-4cc7d518f366") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 04:14:15.346924 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:15.344964    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wmk74"
Sep 19 04:14:15.346924 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:15.345112    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wmk74" podUID="3738f1c7-67af-4201-8e8d-1d0406e0edb9"
Sep 19 04:14:15.346924 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:15.345168    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5t728"
Sep 19 04:14:15.346924 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:15.345290    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5t728" podUID="64cb49fc-ede9-4369-aa8b-76a95d1edab6"
Sep 19 04:14:15.617608 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:15.617536    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-215.ec2.internal" event="NodeReady"
Sep 19 04:14:15.617800 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:15.617675    2579 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 19 04:14:15.717446 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:15.717414    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-8759c"]
Sep 19 04:14:15.722036 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:15.722016    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8759c"
Sep 19 04:14:15.729064 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:15.729038    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-n8jb9"]
Sep 19 04:14:15.732630 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:15.732606    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-n8jb9"
Sep 19 04:14:15.740959 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:15.740938    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-q6xzq\""
Sep 19 04:14:15.740959 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:15.740950    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 19 04:14:15.741099 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:15.740939    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 19 04:14:15.756562 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:15.756540    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 19 04:14:15.756562 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:15.756556    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 19 04:14:15.756731 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:15.756572    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 19 04:14:15.756837 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:15.756825    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5s7cn\""
Sep 19 04:14:15.765048 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:15.765028    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8759c"]
Sep 19 04:14:15.776103 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:15.775921    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-n8jb9"]
Sep 19 04:14:15.784187 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:15.784168    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e9d28ff6-a630-4517-acd6-2c596abcef2a-cert\") pod \"ingress-canary-8759c\" (UID: \"e9d28ff6-a630-4517-acd6-2c596abcef2a\") " pod="openshift-ingress-canary/ingress-canary-8759c"
Sep 19 04:14:15.784285 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:15.784197    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sflk5\" (UniqueName: \"kubernetes.io/projected/e9d28ff6-a630-4517-acd6-2c596abcef2a-kube-api-access-sflk5\") pod \"ingress-canary-8759c\" (UID: \"e9d28ff6-a630-4517-acd6-2c596abcef2a\") " pod="openshift-ingress-canary/ingress-canary-8759c"
Sep 19 04:14:15.884988 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:15.884905    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sflk5\" (UniqueName: \"kubernetes.io/projected/e9d28ff6-a630-4517-acd6-2c596abcef2a-kube-api-access-sflk5\") pod \"ingress-canary-8759c\" (UID: \"e9d28ff6-a630-4517-acd6-2c596abcef2a\") " pod="openshift-ingress-canary/ingress-canary-8759c"
Sep 19 04:14:15.884988 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:15.884954    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rct5n\" (UniqueName: \"kubernetes.io/projected/7a6cf6e3-7960-4902-9c40-375a4de502ac-kube-api-access-rct5n\") pod \"dns-default-n8jb9\" (UID: \"7a6cf6e3-7960-4902-9c40-375a4de502ac\") " pod="openshift-dns/dns-default-n8jb9"
Sep 19 04:14:15.885216 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:15.885099    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7a6cf6e3-7960-4902-9c40-375a4de502ac-metrics-tls\") pod \"dns-default-n8jb9\" (UID: \"7a6cf6e3-7960-4902-9c40-375a4de502ac\") " pod="openshift-dns/dns-default-n8jb9"
Sep 19 04:14:15.885216 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:15.885134    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a6cf6e3-7960-4902-9c40-375a4de502ac-tmp-dir\") pod \"dns-default-n8jb9\" (UID: \"7a6cf6e3-7960-4902-9c40-375a4de502ac\") " pod="openshift-dns/dns-default-n8jb9"
Sep 19 04:14:15.885298 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:15.885257    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7a6cf6e3-7960-4902-9c40-375a4de502ac-config-volume\") pod \"dns-default-n8jb9\" (UID: \"7a6cf6e3-7960-4902-9c40-375a4de502ac\") " pod="openshift-dns/dns-default-n8jb9"
Sep 19 04:14:15.885298 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:15.885281    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e9d28ff6-a630-4517-acd6-2c596abcef2a-cert\") pod \"ingress-canary-8759c\" (UID: \"e9d28ff6-a630-4517-acd6-2c596abcef2a\") " pod="openshift-ingress-canary/ingress-canary-8759c"
Sep 19 04:14:15.889990 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:15.889965    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e9d28ff6-a630-4517-acd6-2c596abcef2a-cert\") pod \"ingress-canary-8759c\" (UID: \"e9d28ff6-a630-4517-acd6-2c596abcef2a\") " pod="openshift-ingress-canary/ingress-canary-8759c"
Sep 19 04:14:15.906587 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:15.906564    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sflk5\" (UniqueName: \"kubernetes.io/projected/e9d28ff6-a630-4517-acd6-2c596abcef2a-kube-api-access-sflk5\") pod \"ingress-canary-8759c\" (UID: \"e9d28ff6-a630-4517-acd6-2c596abcef2a\") " pod="openshift-ingress-canary/ingress-canary-8759c"
Sep 19 04:14:15.986297 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:15.986251    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7a6cf6e3-7960-4902-9c40-375a4de502ac-metrics-tls\") pod \"dns-default-n8jb9\" (UID: \"7a6cf6e3-7960-4902-9c40-375a4de502ac\") " pod="openshift-dns/dns-default-n8jb9"
Sep 19 04:14:15.986297 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:15.986303    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a6cf6e3-7960-4902-9c40-375a4de502ac-tmp-dir\") pod \"dns-default-n8jb9\" (UID: \"7a6cf6e3-7960-4902-9c40-375a4de502ac\") " pod="openshift-dns/dns-default-n8jb9"
Sep 19 04:14:15.986818 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:15.986393    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7a6cf6e3-7960-4902-9c40-375a4de502ac-config-volume\") pod \"dns-default-n8jb9\" (UID: \"7a6cf6e3-7960-4902-9c40-375a4de502ac\") " pod="openshift-dns/dns-default-n8jb9"
Sep 19 04:14:15.986818 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:15.986430    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rct5n\" (UniqueName: \"kubernetes.io/projected/7a6cf6e3-7960-4902-9c40-375a4de502ac-kube-api-access-rct5n\") pod \"dns-default-n8jb9\" (UID: \"7a6cf6e3-7960-4902-9c40-375a4de502ac\") " pod="openshift-dns/dns-default-n8jb9"
Sep 19 04:14:15.986818 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:15.986429    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 19 04:14:15.986818 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:15.986540    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a6cf6e3-7960-4902-9c40-375a4de502ac-metrics-tls podName:7a6cf6e3-7960-4902-9c40-375a4de502ac nodeName:}" failed. No retries permitted until 2026-09-19 04:14:16.486506452 +0000 UTC m=+35.669467593 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7a6cf6e3-7960-4902-9c40-375a4de502ac-metrics-tls") pod "dns-default-n8jb9" (UID: "7a6cf6e3-7960-4902-9c40-375a4de502ac") : secret "dns-default-metrics-tls" not found
Sep 19 04:14:15.986818 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:15.986689    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a6cf6e3-7960-4902-9c40-375a4de502ac-tmp-dir\") pod \"dns-default-n8jb9\" (UID: \"7a6cf6e3-7960-4902-9c40-375a4de502ac\") " pod="openshift-dns/dns-default-n8jb9"
Sep 19 04:14:15.987049 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:15.986966    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7a6cf6e3-7960-4902-9c40-375a4de502ac-config-volume\") pod \"dns-default-n8jb9\" (UID: \"7a6cf6e3-7960-4902-9c40-375a4de502ac\") " pod="openshift-dns/dns-default-n8jb9"
Sep 19 04:14:16.006179 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:16.006155    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rct5n\" (UniqueName: \"kubernetes.io/projected/7a6cf6e3-7960-4902-9c40-375a4de502ac-kube-api-access-rct5n\") pod \"dns-default-n8jb9\" (UID: \"7a6cf6e3-7960-4902-9c40-375a4de502ac\") " pod="openshift-dns/dns-default-n8jb9"
Sep 19 04:14:16.033208 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:16.033181    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8759c"
Sep 19 04:14:16.344832 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:16.344800    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7dgk"
Sep 19 04:14:16.348651 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:16.348621    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 19 04:14:16.348991 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:16.348975    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-rwkvg\""
Sep 19 04:14:16.349355 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:16.349332    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 19 04:14:16.491388 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:16.491344    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7a6cf6e3-7960-4902-9c40-375a4de502ac-metrics-tls\") pod \"dns-default-n8jb9\" (UID: \"7a6cf6e3-7960-4902-9c40-375a4de502ac\") " pod="openshift-dns/dns-default-n8jb9"
Sep 19 04:14:16.491497 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:16.491482    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 19 04:14:16.491555 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:16.491544    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a6cf6e3-7960-4902-9c40-375a4de502ac-metrics-tls podName:7a6cf6e3-7960-4902-9c40-375a4de502ac nodeName:}" failed. No retries permitted until 2026-09-19 04:14:17.491526772 +0000 UTC m=+36.674487903 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7a6cf6e3-7960-4902-9c40-375a4de502ac-metrics-tls") pod "dns-default-n8jb9" (UID: "7a6cf6e3-7960-4902-9c40-375a4de502ac") : secret "dns-default-metrics-tls" not found
Sep 19 04:14:16.771586 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:16.771558    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8759c"]
Sep 19 04:14:16.784572 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:14:16.784533    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9d28ff6_a630_4517_acd6_2c596abcef2a.slice/crio-3750c34ffa7f03a4435b1450274b089dfab98d66ca2c97ffa386381c42889c98 WatchSource:0}: Error finding container 3750c34ffa7f03a4435b1450274b089dfab98d66ca2c97ffa386381c42889c98: Status 404 returned error can't find the container with id 3750c34ffa7f03a4435b1450274b089dfab98d66ca2c97ffa386381c42889c98
Sep 19 04:14:17.344777 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:17.344742    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wmk74"
Sep 19 04:14:17.345634 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:17.344784    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5t728"
Sep 19 04:14:17.348051 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:17.348030    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 19 04:14:17.349361 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:17.349339    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zg2ns\""
Sep 19 04:14:17.349483 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:17.349343    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 19 04:14:17.502459 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:17.502425    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7a6cf6e3-7960-4902-9c40-375a4de502ac-metrics-tls\") pod \"dns-default-n8jb9\" (UID: \"7a6cf6e3-7960-4902-9c40-375a4de502ac\") " pod="openshift-dns/dns-default-n8jb9"
Sep 19 04:14:17.502629 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:17.502591    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 19 04:14:17.502677 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:17.502653    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a6cf6e3-7960-4902-9c40-375a4de502ac-metrics-tls podName:7a6cf6e3-7960-4902-9c40-375a4de502ac nodeName:}" failed. No retries permitted until 2026-09-19 04:14:19.502636166 +0000 UTC m=+38.685597304 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7a6cf6e3-7960-4902-9c40-375a4de502ac-metrics-tls") pod "dns-default-n8jb9" (UID: "7a6cf6e3-7960-4902-9c40-375a4de502ac") : secret "dns-default-metrics-tls" not found
Sep 19 04:14:17.539801 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:17.539712    2579 generic.go:356] "Generic (PLEG): container finished" podID="bcd5da2a-a24f-40fa-80aa-03ef54cc22a6" containerID="f7c14dc6d1d5be6c636a3393c2b81b463623688e071a57e759a0c343ad017de1" exitCode=0
Sep 19 04:14:17.539943 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:17.539799    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wzbjr" event={"ID":"bcd5da2a-a24f-40fa-80aa-03ef54cc22a6","Type":"ContainerDied","Data":"f7c14dc6d1d5be6c636a3393c2b81b463623688e071a57e759a0c343ad017de1"}
Sep 19 04:14:17.541384 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:17.541355    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8759c" event={"ID":"e9d28ff6-a630-4517-acd6-2c596abcef2a","Type":"ContainerStarted","Data":"3750c34ffa7f03a4435b1450274b089dfab98d66ca2c97ffa386381c42889c98"}
Sep 19 04:14:18.548267 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:18.547962    2579 generic.go:356] "Generic (PLEG): container finished" podID="bcd5da2a-a24f-40fa-80aa-03ef54cc22a6" containerID="2021f9a8455fbcc090bd20ee8600fa9c57a9118e21b449bf417b6c81d8c16445" exitCode=0
Sep 19 04:14:18.548267 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:18.548043    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wzbjr" event={"ID":"bcd5da2a-a24f-40fa-80aa-03ef54cc22a6","Type":"ContainerDied","Data":"2021f9a8455fbcc090bd20ee8600fa9c57a9118e21b449bf417b6c81d8c16445"}
Sep 19 04:14:19.522566 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:19.522532    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7a6cf6e3-7960-4902-9c40-375a4de502ac-metrics-tls\") pod \"dns-default-n8jb9\" (UID: \"7a6cf6e3-7960-4902-9c40-375a4de502ac\") " pod="openshift-dns/dns-default-n8jb9"
Sep 19 04:14:19.522741 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:19.522641    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 19 04:14:19.522741 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:19.522716    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a6cf6e3-7960-4902-9c40-375a4de502ac-metrics-tls podName:7a6cf6e3-7960-4902-9c40-375a4de502ac nodeName:}" failed. No retries permitted until 2026-09-19 04:14:23.522684291 +0000 UTC m=+42.705645431 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7a6cf6e3-7960-4902-9c40-375a4de502ac-metrics-tls") pod "dns-default-n8jb9" (UID: "7a6cf6e3-7960-4902-9c40-375a4de502ac") : secret "dns-default-metrics-tls" not found
Sep 19 04:14:19.553989 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:19.553954    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wzbjr" event={"ID":"bcd5da2a-a24f-40fa-80aa-03ef54cc22a6","Type":"ContainerStarted","Data":"d24035e892daf6afe7b81db4972ddd5d28d9513ef5b9419447d5a066beb45165"}
Sep 19 04:14:19.558653 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:19.558624    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8759c" event={"ID":"e9d28ff6-a630-4517-acd6-2c596abcef2a","Type":"ContainerStarted","Data":"d539fa0c3b6c7092e51893b2b8bb3faad4a2c5e3e10ccd6b3b8b83b965893854"}
Sep 19 04:14:19.594135 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:19.594088    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-wzbjr" podStartSLOduration=9.604955495 podStartE2EDuration="38.594075587s" totalImagesPullingTime="28.989120092s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 04:13:41 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:14:19.593430545 +0000 UTC m=+38.776391706" watchObservedRunningTime="2026-09-19 04:14:19.594075587 +0000 UTC m=+38.777036734"
Sep 19 04:14:19.627101 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:19.626991    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-8759c" podStartSLOduration=2.38391526 podStartE2EDuration="4.626973412s" totalImagesPullingTime="2.243058152s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 04:14:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:14:19.626267384 +0000 UTC m=+38.809228531" watchObservedRunningTime="2026-09-19 04:14:19.626973412 +0000 UTC m=+38.809934551"
Sep 19 04:14:23.554308 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:23.554265    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7a6cf6e3-7960-4902-9c40-375a4de502ac-metrics-tls\") pod \"dns-default-n8jb9\" (UID: \"7a6cf6e3-7960-4902-9c40-375a4de502ac\") " pod="openshift-dns/dns-default-n8jb9"
Sep 19 04:14:23.554737 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:23.554434    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 19 04:14:23.554737 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:23.554502    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a6cf6e3-7960-4902-9c40-375a4de502ac-metrics-tls podName:7a6cf6e3-7960-4902-9c40-375a4de502ac nodeName:}" failed. No retries permitted until 2026-09-19 04:14:31.554484408 +0000 UTC m=+50.737445534 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7a6cf6e3-7960-4902-9c40-375a4de502ac-metrics-tls") pod "dns-default-n8jb9" (UID: "7a6cf6e3-7960-4902-9c40-375a4de502ac") : secret "dns-default-metrics-tls" not found
Sep 19 04:14:25.543921 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:25.543885    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-prdpb"]
Sep 19 04:14:25.586131 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:25.586098    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-6cddf8756d-5pc66"]
Sep 19 04:14:25.586292 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:25.586218    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-prdpb"
Sep 19 04:14:25.591490 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:25.591469    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 19 04:14:25.591736 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:25.591715    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 19 04:14:25.594055 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:25.594034    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 19 04:14:25.599520 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:25.599502    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-hlr7v\""
Sep 19 04:14:25.599633 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:25.599617    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 19 04:14:25.603575 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:25.603557    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-prdpb"]
Sep 19 04:14:25.603753 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:25.603585    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6cddf8756d-5pc66"]
Sep 19 04:14:25.603753 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:25.603683    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6cddf8756d-5pc66"
Sep 19 04:14:25.609295 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:25.609247    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 19 04:14:25.611938 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:25.611922    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 19 04:14:25.612906 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:25.612890    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 19 04:14:25.613889 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:25.613876    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-2tj52\""
Sep 19 04:14:25.614804 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:25.614791    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 19 04:14:25.615263 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:25.615251    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 19 04:14:25.626052 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:25.626031    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 19 04:14:25.668949 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:25.668917    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/1633a682-5c86-4a59-b25a-2d049b6f3577-snapshots\") pod \"insights-operator-6f9c644dd9-prdpb\" (UID: \"1633a682-5c86-4a59-b25a-2d049b6f3577\") " pod="openshift-insights/insights-operator-6f9c644dd9-prdpb"
Sep 19 04:14:25.669131 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:25.668979    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1633a682-5c86-4a59-b25a-2d049b6f3577-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-prdpb\" (UID: \"1633a682-5c86-4a59-b25a-2d049b6f3577\") " pod="openshift-insights/insights-operator-6f9c644dd9-prdpb"
Sep 19 04:14:25.669131 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:25.669026    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1633a682-5c86-4a59-b25a-2d049b6f3577-serving-cert\") pod \"insights-operator-6f9c644dd9-prdpb\" (UID: \"1633a682-5c86-4a59-b25a-2d049b6f3577\") " pod="openshift-insights/insights-operator-6f9c644dd9-prdpb"
Sep 19 04:14:25.669131 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:25.669107    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1633a682-5c86-4a59-b25a-2d049b6f3577-tmp\") pod \"insights-operator-6f9c644dd9-prdpb\" (UID: \"1633a682-5c86-4a59-b25a-2d049b6f3577\") " pod="openshift-insights/insights-operator-6f9c644dd9-prdpb"
Sep 19 04:14:25.669279 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:25.669160    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1633a682-5c86-4a59-b25a-2d049b6f3577-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-prdpb\" (UID: \"1633a682-5c86-4a59-b25a-2d049b6f3577\") " pod="openshift-insights/insights-operator-6f9c644dd9-prdpb"
Sep 19 04:14:25.669279 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:25.669208    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rltfp\" (UniqueName: \"kubernetes.io/projected/1633a682-5c86-4a59-b25a-2d049b6f3577-kube-api-access-rltfp\") pod \"insights-operator-6f9c644dd9-prdpb\" (UID: \"1633a682-5c86-4a59-b25a-2d049b6f3577\") " pod="openshift-insights/insights-operator-6f9c644dd9-prdpb"
Sep 19 04:14:25.710825 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:25.710798    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 19 04:14:25.770281 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:25.770250    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1ebe82f-7422-4a2c-9f72-7eef8335ffb2-service-ca-bundle\") pod \"router-default-6cddf8756d-5pc66\" (UID: \"c1ebe82f-7422-4a2c-9f72-7eef8335ffb2\") " pod="openshift-ingress/router-default-6cddf8756d-5pc66"
Sep 19 04:14:25.770281 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:25.770285    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/c1ebe82f-7422-4a2c-9f72-7eef8335ffb2-stats-auth\") pod \"router-default-6cddf8756d-5pc66\" (UID: \"c1ebe82f-7422-4a2c-9f72-7eef8335ffb2\") " pod="openshift-ingress/router-default-6cddf8756d-5pc66"
Sep 19 04:14:25.770467 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:25.770312    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1633a682-5c86-4a59-b25a-2d049b6f3577-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-prdpb\" (UID: \"1633a682-5c86-4a59-b25a-2d049b6f3577\") " pod="openshift-insights/insights-operator-6f9c644dd9-prdpb"
Sep 19 04:14:25.770467 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:25.770335    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rltfp\" (UniqueName: \"kubernetes.io/projected/1633a682-5c86-4a59-b25a-2d049b6f3577-kube-api-access-rltfp\") pod \"insights-operator-6f9c644dd9-prdpb\" (UID: \"1633a682-5c86-4a59-b25a-2d049b6f3577\") " pod="openshift-insights/insights-operator-6f9c644dd9-prdpb"
Sep 19 04:14:25.770467 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:25.770420    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/c1ebe82f-7422-4a2c-9f72-7eef8335ffb2-default-certificate\") pod \"router-default-6cddf8756d-5pc66\" (UID: \"c1ebe82f-7422-4a2c-9f72-7eef8335ffb2\") " pod="openshift-ingress/router-default-6cddf8756d-5pc66"
Sep 19 04:14:25.770467 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:25.770448    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4npqz\" (UniqueName: \"kubernetes.io/projected/c1ebe82f-7422-4a2c-9f72-7eef8335ffb2-kube-api-access-4npqz\") pod \"router-default-6cddf8756d-5pc66\" (UID: \"c1ebe82f-7422-4a2c-9f72-7eef8335ffb2\") " pod="openshift-ingress/router-default-6cddf8756d-5pc66"
Sep 19 04:14:25.770599 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:25.770481    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/1633a682-5c86-4a59-b25a-2d049b6f3577-snapshots\") pod \"insights-operator-6f9c644dd9-prdpb\" (UID: \"1633a682-5c86-4a59-b25a-2d049b6f3577\") " pod="openshift-insights/insights-operator-6f9c644dd9-prdpb"
Sep 19 04:14:25.770599 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:25.770531    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1633a682-5c86-4a59-b25a-2d049b6f3577-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-prdpb\" (UID: \"1633a682-5c86-4a59-b25a-2d049b6f3577\") " pod="openshift-insights/insights-operator-6f9c644dd9-prdpb"
Sep 19 04:14:25.770599 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:25.770553    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1633a682-5c86-4a59-b25a-2d049b6f3577-serving-cert\") pod \"insights-operator-6f9c644dd9-prdpb\" (UID: \"1633a682-5c86-4a59-b25a-2d049b6f3577\") " pod="openshift-insights/insights-operator-6f9c644dd9-prdpb"
Sep 19 04:14:25.770599 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:25.770584    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1ebe82f-7422-4a2c-9f72-7eef8335ffb2-metrics-certs\") pod \"router-default-6cddf8756d-5pc66\" (UID: \"c1ebe82f-7422-4a2c-9f72-7eef8335ffb2\") " pod="openshift-ingress/router-default-6cddf8756d-5pc66"
Sep 19 04:14:25.770811 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:25.770619    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1633a682-5c86-4a59-b25a-2d049b6f3577-tmp\") pod \"insights-operator-6f9c644dd9-prdpb\" (UID: \"1633a682-5c86-4a59-b25a-2d049b6f3577\") " pod="openshift-insights/insights-operator-6f9c644dd9-prdpb"
Sep 19 04:14:25.771095 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:25.771076    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1633a682-5c86-4a59-b25a-2d049b6f3577-tmp\") pod \"insights-operator-6f9c644dd9-prdpb\" (UID: \"1633a682-5c86-4a59-b25a-2d049b6f3577\") " pod="openshift-insights/insights-operator-6f9c644dd9-prdpb"
Sep 19 04:14:25.771165 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:25.771130    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1633a682-5c86-4a59-b25a-2d049b6f3577-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-prdpb\" (UID: \"1633a682-5c86-4a59-b25a-2d049b6f3577\") " pod="openshift-insights/insights-operator-6f9c644dd9-prdpb"
Sep 19 04:14:25.771203 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:25.771167    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/1633a682-5c86-4a59-b25a-2d049b6f3577-snapshots\") pod \"insights-operator-6f9c644dd9-prdpb\" (UID: \"1633a682-5c86-4a59-b25a-2d049b6f3577\") " pod="openshift-insights/insights-operator-6f9c644dd9-prdpb"
Sep 19 04:14:25.771236 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:25.771218    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1633a682-5c86-4a59-b25a-2d049b6f3577-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-prdpb\" (UID: \"1633a682-5c86-4a59-b25a-2d049b6f3577\") " pod="openshift-insights/insights-operator-6f9c644dd9-prdpb"
Sep 19 04:14:25.774130 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:25.774107    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1633a682-5c86-4a59-b25a-2d049b6f3577-serving-cert\") pod \"insights-operator-6f9c644dd9-prdpb\" (UID: \"1633a682-5c86-4a59-b25a-2d049b6f3577\") " pod="openshift-insights/insights-operator-6f9c644dd9-prdpb"
Sep 19 04:14:25.801825 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:25.801761    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rltfp\" (UniqueName: \"kubernetes.io/projected/1633a682-5c86-4a59-b25a-2d049b6f3577-kube-api-access-rltfp\") pod \"insights-operator-6f9c644dd9-prdpb\" (UID: \"1633a682-5c86-4a59-b25a-2d049b6f3577\") " pod="openshift-insights/insights-operator-6f9c644dd9-prdpb"
Sep 19 04:14:25.871784 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:25.871747    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1ebe82f-7422-4a2c-9f72-7eef8335ffb2-metrics-certs\") pod \"router-default-6cddf8756d-5pc66\" (UID: \"c1ebe82f-7422-4a2c-9f72-7eef8335ffb2\") " pod="openshift-ingress/router-default-6cddf8756d-5pc66"
Sep 19 04:14:25.871944 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:25.871798    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1ebe82f-7422-4a2c-9f72-7eef8335ffb2-service-ca-bundle\") pod \"router-default-6cddf8756d-5pc66\" (UID: \"c1ebe82f-7422-4a2c-9f72-7eef8335ffb2\") " pod="openshift-ingress/router-default-6cddf8756d-5pc66"
Sep 19 04:14:25.871944 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:25.871894    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 19 04:14:25.871944 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:25.871909    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/c1ebe82f-7422-4a2c-9f72-7eef8335ffb2-stats-auth\") pod \"router-default-6cddf8756d-5pc66\" (UID: \"c1ebe82f-7422-4a2c-9f72-7eef8335ffb2\") " pod="openshift-ingress/router-default-6cddf8756d-5pc66"
Sep 19 04:14:25.871944 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:25.871918    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c1ebe82f-7422-4a2c-9f72-7eef8335ffb2-service-ca-bundle podName:c1ebe82f-7422-4a2c-9f72-7eef8335ffb2 nodeName:}" failed. No retries permitted until 2026-09-19 04:14:26.371901121 +0000 UTC m=+45.554862274 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c1ebe82f-7422-4a2c-9f72-7eef8335ffb2-service-ca-bundle") pod "router-default-6cddf8756d-5pc66" (UID: "c1ebe82f-7422-4a2c-9f72-7eef8335ffb2") : configmap references non-existent config key: service-ca.crt
Sep 19 04:14:25.872229 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:25.871952    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1ebe82f-7422-4a2c-9f72-7eef8335ffb2-metrics-certs podName:c1ebe82f-7422-4a2c-9f72-7eef8335ffb2 nodeName:}" failed. No retries permitted until 2026-09-19 04:14:26.371939221 +0000 UTC m=+45.554900348 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c1ebe82f-7422-4a2c-9f72-7eef8335ffb2-metrics-certs") pod "router-default-6cddf8756d-5pc66" (UID: "c1ebe82f-7422-4a2c-9f72-7eef8335ffb2") : secret "router-metrics-certs-default" not found
Sep 19 04:14:25.872229 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:25.872023    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/c1ebe82f-7422-4a2c-9f72-7eef8335ffb2-default-certificate\") pod \"router-default-6cddf8756d-5pc66\" (UID: \"c1ebe82f-7422-4a2c-9f72-7eef8335ffb2\") " pod="openshift-ingress/router-default-6cddf8756d-5pc66"
Sep 19 04:14:25.872229 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:25.872041    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4npqz\" (UniqueName: \"kubernetes.io/projected/c1ebe82f-7422-4a2c-9f72-7eef8335ffb2-kube-api-access-4npqz\") pod \"router-default-6cddf8756d-5pc66\" (UID: \"c1ebe82f-7422-4a2c-9f72-7eef8335ffb2\") " pod="openshift-ingress/router-default-6cddf8756d-5pc66"
Sep 19 04:14:25.874438 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:25.874411    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/c1ebe82f-7422-4a2c-9f72-7eef8335ffb2-default-certificate\") pod \"router-default-6cddf8756d-5pc66\" (UID: \"c1ebe82f-7422-4a2c-9f72-7eef8335ffb2\") " pod="openshift-ingress/router-default-6cddf8756d-5pc66"
Sep 19 04:14:25.874527 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:25.874450    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/c1ebe82f-7422-4a2c-9f72-7eef8335ffb2-stats-auth\") pod \"router-default-6cddf8756d-5pc66\" (UID: \"c1ebe82f-7422-4a2c-9f72-7eef8335ffb2\") " pod="openshift-ingress/router-default-6cddf8756d-5pc66"
Sep 19 04:14:25.895793 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:25.895777    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-prdpb"
Sep 19 04:14:25.899066 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:25.899047    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4npqz\" (UniqueName: \"kubernetes.io/projected/c1ebe82f-7422-4a2c-9f72-7eef8335ffb2-kube-api-access-4npqz\") pod \"router-default-6cddf8756d-5pc66\" (UID: \"c1ebe82f-7422-4a2c-9f72-7eef8335ffb2\") " pod="openshift-ingress/router-default-6cddf8756d-5pc66"
Sep 19 04:14:26.196452 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:26.196415    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-prdpb"]
Sep 19 04:14:26.201517 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:14:26.201490    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1633a682_5c86_4a59_b25a_2d049b6f3577.slice/crio-c088a3410589b669032d24263173ae439e50e427a7309de0066e4a9799b5574a WatchSource:0}: Error finding container c088a3410589b669032d24263173ae439e50e427a7309de0066e4a9799b5574a: Status 404 returned error can't find the container with id c088a3410589b669032d24263173ae439e50e427a7309de0066e4a9799b5574a
Sep 19 04:14:26.375804 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:26.375768    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1ebe82f-7422-4a2c-9f72-7eef8335ffb2-metrics-certs\") pod \"router-default-6cddf8756d-5pc66\" (UID: \"c1ebe82f-7422-4a2c-9f72-7eef8335ffb2\") " pod="openshift-ingress/router-default-6cddf8756d-5pc66"
Sep 19 04:14:26.375947 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:26.375816    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1ebe82f-7422-4a2c-9f72-7eef8335ffb2-service-ca-bundle\") pod \"router-default-6cddf8756d-5pc66\" (UID: \"c1ebe82f-7422-4a2c-9f72-7eef8335ffb2\") " pod="openshift-ingress/router-default-6cddf8756d-5pc66"
Sep 19 04:14:26.375947 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:26.375928    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 19 04:14:26.376037 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:26.375988    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1ebe82f-7422-4a2c-9f72-7eef8335ffb2-metrics-certs podName:c1ebe82f-7422-4a2c-9f72-7eef8335ffb2 nodeName:}" failed. No retries permitted until 2026-09-19 04:14:27.375972995 +0000 UTC m=+46.558934122 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c1ebe82f-7422-4a2c-9f72-7eef8335ffb2-metrics-certs") pod "router-default-6cddf8756d-5pc66" (UID: "c1ebe82f-7422-4a2c-9f72-7eef8335ffb2") : secret "router-metrics-certs-default" not found
Sep 19 04:14:26.376037 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:26.376004    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c1ebe82f-7422-4a2c-9f72-7eef8335ffb2-service-ca-bundle podName:c1ebe82f-7422-4a2c-9f72-7eef8335ffb2 nodeName:}" failed. No retries permitted until 2026-09-19 04:14:27.375997938 +0000 UTC m=+46.558959063 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c1ebe82f-7422-4a2c-9f72-7eef8335ffb2-service-ca-bundle") pod "router-default-6cddf8756d-5pc66" (UID: "c1ebe82f-7422-4a2c-9f72-7eef8335ffb2") : configmap references non-existent config key: service-ca.crt
Sep 19 04:14:26.578907 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:26.578813    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-prdpb" event={"ID":"1633a682-5c86-4a59-b25a-2d049b6f3577","Type":"ContainerStarted","Data":"c088a3410589b669032d24263173ae439e50e427a7309de0066e4a9799b5574a"}
Sep 19 04:14:27.384378 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:27.384340    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1ebe82f-7422-4a2c-9f72-7eef8335ffb2-metrics-certs\") pod \"router-default-6cddf8756d-5pc66\" (UID: \"c1ebe82f-7422-4a2c-9f72-7eef8335ffb2\") " pod="openshift-ingress/router-default-6cddf8756d-5pc66"
Sep 19 04:14:27.384378 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:27.384385    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1ebe82f-7422-4a2c-9f72-7eef8335ffb2-service-ca-bundle\") pod \"router-default-6cddf8756d-5pc66\" (UID: \"c1ebe82f-7422-4a2c-9f72-7eef8335ffb2\") " pod="openshift-ingress/router-default-6cddf8756d-5pc66"
Sep 19 04:14:27.384585 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:27.384514    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c1ebe82f-7422-4a2c-9f72-7eef8335ffb2-service-ca-bundle podName:c1ebe82f-7422-4a2c-9f72-7eef8335ffb2 nodeName:}" failed. No retries permitted until 2026-09-19 04:14:29.384498181 +0000 UTC m=+48.567459308 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c1ebe82f-7422-4a2c-9f72-7eef8335ffb2-service-ca-bundle") pod "router-default-6cddf8756d-5pc66" (UID: "c1ebe82f-7422-4a2c-9f72-7eef8335ffb2") : configmap references non-existent config key: service-ca.crt
Sep 19 04:14:27.384585 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:27.384512    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 19 04:14:27.384585 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:27.384579    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1ebe82f-7422-4a2c-9f72-7eef8335ffb2-metrics-certs podName:c1ebe82f-7422-4a2c-9f72-7eef8335ffb2 nodeName:}" failed. No retries permitted until 2026-09-19 04:14:29.384566013 +0000 UTC m=+48.567527232 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c1ebe82f-7422-4a2c-9f72-7eef8335ffb2-metrics-certs") pod "router-default-6cddf8756d-5pc66" (UID: "c1ebe82f-7422-4a2c-9f72-7eef8335ffb2") : secret "router-metrics-certs-default" not found
Sep 19 04:14:29.401925 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:29.401886    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1ebe82f-7422-4a2c-9f72-7eef8335ffb2-metrics-certs\") pod \"router-default-6cddf8756d-5pc66\" (UID: \"c1ebe82f-7422-4a2c-9f72-7eef8335ffb2\") " pod="openshift-ingress/router-default-6cddf8756d-5pc66"
Sep 19 04:14:29.402325 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:29.401943    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1ebe82f-7422-4a2c-9f72-7eef8335ffb2-service-ca-bundle\") pod \"router-default-6cddf8756d-5pc66\" (UID: \"c1ebe82f-7422-4a2c-9f72-7eef8335ffb2\") " pod="openshift-ingress/router-default-6cddf8756d-5pc66"
Sep 19 04:14:29.402325 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:29.402036    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 19 04:14:29.402325 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:29.402089    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c1ebe82f-7422-4a2c-9f72-7eef8335ffb2-service-ca-bundle podName:c1ebe82f-7422-4a2c-9f72-7eef8335ffb2 nodeName:}" failed. No retries permitted until 2026-09-19 04:14:33.402071855 +0000 UTC m=+52.585033000 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c1ebe82f-7422-4a2c-9f72-7eef8335ffb2-service-ca-bundle") pod "router-default-6cddf8756d-5pc66" (UID: "c1ebe82f-7422-4a2c-9f72-7eef8335ffb2") : configmap references non-existent config key: service-ca.crt
Sep 19 04:14:29.402325 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:29.402111    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1ebe82f-7422-4a2c-9f72-7eef8335ffb2-metrics-certs podName:c1ebe82f-7422-4a2c-9f72-7eef8335ffb2 nodeName:}" failed. No retries permitted until 2026-09-19 04:14:33.402101196 +0000 UTC m=+52.585062322 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c1ebe82f-7422-4a2c-9f72-7eef8335ffb2-metrics-certs") pod "router-default-6cddf8756d-5pc66" (UID: "c1ebe82f-7422-4a2c-9f72-7eef8335ffb2") : secret "router-metrics-certs-default" not found
Sep 19 04:14:29.586743 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:29.586715    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-prdpb" event={"ID":"1633a682-5c86-4a59-b25a-2d049b6f3577","Type":"ContainerStarted","Data":"42d579a27d981fba5895f8773777bfc8f2cbe9859c82bb4efc6f76597c995660"}
Sep 19 04:14:29.602340 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:29.602304    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-t7mz5"]
Sep 19 04:14:29.618564 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:29.618539    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-t7mz5"
Sep 19 04:14:29.622204 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:29.622183    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mt2jg\""
Sep 19 04:14:29.622315 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:29.622218    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 19 04:14:29.622908 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:29.622886    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 19 04:14:29.637148 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:29.637124    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-t7mz5"]
Sep 19 04:14:29.657970 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:29.657889    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-6f9c644dd9-prdpb" podStartSLOduration=2.302878048 podStartE2EDuration="4.657874855s" totalImagesPullingTime="2.354996807s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 04:14:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:14:29.656476835 +0000 UTC m=+48.839437983" watchObservedRunningTime="2026-09-19 04:14:29.657874855 +0000 UTC m=+48.840836003"
Sep 19 04:14:29.705111 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:29.705084    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7ef03827-5da3-44f3-84fd-7a9581528469-crio-socket\") pod \"insights-runtime-extractor-t7mz5\" (UID: \"7ef03827-5da3-44f3-84fd-7a9581528469\") " pod="openshift-insights/insights-runtime-extractor-t7mz5"
Sep 19 04:14:29.705273 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:29.705131    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7ef03827-5da3-44f3-84fd-7a9581528469-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t7mz5\" (UID: \"7ef03827-5da3-44f3-84fd-7a9581528469\") " pod="openshift-insights/insights-runtime-extractor-t7mz5"
Sep 19 04:14:29.705273 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:29.705170    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7ef03827-5da3-44f3-84fd-7a9581528469-data-volume\") pod \"insights-runtime-extractor-t7mz5\" (UID: \"7ef03827-5da3-44f3-84fd-7a9581528469\") " pod="openshift-insights/insights-runtime-extractor-t7mz5"
Sep 19 04:14:29.705273 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:29.705232    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7ef03827-5da3-44f3-84fd-7a9581528469-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t7mz5\" (UID: \"7ef03827-5da3-44f3-84fd-7a9581528469\") " pod="openshift-insights/insights-runtime-extractor-t7mz5"
Sep 19 04:14:29.705454 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:29.705321    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5ll5\" (UniqueName: \"kubernetes.io/projected/7ef03827-5da3-44f3-84fd-7a9581528469-kube-api-access-p5ll5\") pod \"insights-runtime-extractor-t7mz5\" (UID: \"7ef03827-5da3-44f3-84fd-7a9581528469\") " pod="openshift-insights/insights-runtime-extractor-t7mz5"
Sep 19 04:14:29.806482 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:29.806441    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7ef03827-5da3-44f3-84fd-7a9581528469-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t7mz5\" (UID: \"7ef03827-5da3-44f3-84fd-7a9581528469\") " pod="openshift-insights/insights-runtime-extractor-t7mz5"
Sep 19 04:14:29.806482 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:29.806488    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7ef03827-5da3-44f3-84fd-7a9581528469-data-volume\") pod \"insights-runtime-extractor-t7mz5\" (UID: \"7ef03827-5da3-44f3-84fd-7a9581528469\") " pod="openshift-insights/insights-runtime-extractor-t7mz5"
Sep 19 04:14:29.806674 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:29.806527    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7ef03827-5da3-44f3-84fd-7a9581528469-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t7mz5\" (UID: \"7ef03827-5da3-44f3-84fd-7a9581528469\") " pod="openshift-insights/insights-runtime-extractor-t7mz5"
Sep 19 04:14:29.806674 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:29.806571    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5ll5\" (UniqueName: \"kubernetes.io/projected/7ef03827-5da3-44f3-84fd-7a9581528469-kube-api-access-p5ll5\") pod \"insights-runtime-extractor-t7mz5\" (UID: \"7ef03827-5da3-44f3-84fd-7a9581528469\") " pod="openshift-insights/insights-runtime-extractor-t7mz5"
Sep 19 04:14:29.806674 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:29.806605    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7ef03827-5da3-44f3-84fd-7a9581528469-crio-socket\") pod \"insights-runtime-extractor-t7mz5\" (UID: \"7ef03827-5da3-44f3-84fd-7a9581528469\") " pod="openshift-insights/insights-runtime-extractor-t7mz5"
Sep 19 04:14:29.806674 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:29.806568    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 19 04:14:29.806830 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:29.806683    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 19 04:14:29.806830 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:29.806743    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7ef03827-5da3-44f3-84fd-7a9581528469-kube-rbac-proxy-cm podName:7ef03827-5da3-44f3-84fd-7a9581528469 nodeName:}" failed. No retries permitted until 2026-09-19 04:14:30.306719505 +0000 UTC m=+49.489680651 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7ef03827-5da3-44f3-84fd-7a9581528469-kube-rbac-proxy-cm") pod "insights-runtime-extractor-t7mz5" (UID: "7ef03827-5da3-44f3-84fd-7a9581528469") : configmap "kube-rbac-proxy" not found
Sep 19 04:14:29.806830 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:29.806762    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ef03827-5da3-44f3-84fd-7a9581528469-insights-runtime-extractor-tls podName:7ef03827-5da3-44f3-84fd-7a9581528469 nodeName:}" failed. No retries permitted until 2026-09-19 04:14:30.30675286 +0000 UTC m=+49.489713986 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7ef03827-5da3-44f3-84fd-7a9581528469-insights-runtime-extractor-tls") pod "insights-runtime-extractor-t7mz5" (UID: "7ef03827-5da3-44f3-84fd-7a9581528469") : secret "insights-runtime-extractor-tls" not found
Sep 19 04:14:29.806830 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:29.806801    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7ef03827-5da3-44f3-84fd-7a9581528469-crio-socket\") pod \"insights-runtime-extractor-t7mz5\" (UID: \"7ef03827-5da3-44f3-84fd-7a9581528469\") " pod="openshift-insights/insights-runtime-extractor-t7mz5"
Sep 19 04:14:29.806957 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:29.806904    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7ef03827-5da3-44f3-84fd-7a9581528469-data-volume\") pod \"insights-runtime-extractor-t7mz5\" (UID: \"7ef03827-5da3-44f3-84fd-7a9581528469\") " pod="openshift-insights/insights-runtime-extractor-t7mz5"
Sep 19 04:14:29.816621 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:29.816596    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5ll5\" (UniqueName: \"kubernetes.io/projected/7ef03827-5da3-44f3-84fd-7a9581528469-kube-api-access-p5ll5\") pod \"insights-runtime-extractor-t7mz5\" (UID: \"7ef03827-5da3-44f3-84fd-7a9581528469\") " pod="openshift-insights/insights-runtime-extractor-t7mz5"
Sep 19 04:14:30.310944 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:30.310898    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7ef03827-5da3-44f3-84fd-7a9581528469-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t7mz5\" (UID: \"7ef03827-5da3-44f3-84fd-7a9581528469\") " pod="openshift-insights/insights-runtime-extractor-t7mz5"
Sep 19 04:14:30.311132 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:30.310973    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7ef03827-5da3-44f3-84fd-7a9581528469-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t7mz5\" (UID: \"7ef03827-5da3-44f3-84fd-7a9581528469\") " pod="openshift-insights/insights-runtime-extractor-t7mz5"
Sep 19 04:14:30.311132 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:30.311025    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 19 04:14:30.311132 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:30.311086    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 19 04:14:30.311132 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:30.311096    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7ef03827-5da3-44f3-84fd-7a9581528469-kube-rbac-proxy-cm podName:7ef03827-5da3-44f3-84fd-7a9581528469 nodeName:}" failed. No retries permitted until 2026-09-19 04:14:31.311078248 +0000 UTC m=+50.494039386 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7ef03827-5da3-44f3-84fd-7a9581528469-kube-rbac-proxy-cm") pod "insights-runtime-extractor-t7mz5" (UID: "7ef03827-5da3-44f3-84fd-7a9581528469") : configmap "kube-rbac-proxy" not found
Sep 19 04:14:30.311279 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:30.311136    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ef03827-5da3-44f3-84fd-7a9581528469-insights-runtime-extractor-tls podName:7ef03827-5da3-44f3-84fd-7a9581528469 nodeName:}" failed. No retries permitted until 2026-09-19 04:14:31.311124808 +0000 UTC m=+50.494085933 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7ef03827-5da3-44f3-84fd-7a9581528469-insights-runtime-extractor-tls") pod "insights-runtime-extractor-t7mz5" (UID: "7ef03827-5da3-44f3-84fd-7a9581528469") : secret "insights-runtime-extractor-tls" not found
Sep 19 04:14:31.255681 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:31.255651    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2ptq2_a5194aa7-dba0-46dc-b3e9-91a4ea69e79e/dns-node-resolver/0.log"
Sep 19 04:14:31.319288 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:31.319252    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7ef03827-5da3-44f3-84fd-7a9581528469-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t7mz5\" (UID: \"7ef03827-5da3-44f3-84fd-7a9581528469\") " pod="openshift-insights/insights-runtime-extractor-t7mz5"
Sep 19 04:14:31.319524 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:31.319337    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7ef03827-5da3-44f3-84fd-7a9581528469-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t7mz5\" (UID: \"7ef03827-5da3-44f3-84fd-7a9581528469\") " pod="openshift-insights/insights-runtime-extractor-t7mz5"
Sep 19 04:14:31.319524 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:31.319415    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 19 04:14:31.319524 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:31.319425    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 19 04:14:31.319524 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:31.319491    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ef03827-5da3-44f3-84fd-7a9581528469-insights-runtime-extractor-tls podName:7ef03827-5da3-44f3-84fd-7a9581528469 nodeName:}" failed. No retries permitted until 2026-09-19 04:14:33.319467993 +0000 UTC m=+52.502429132 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7ef03827-5da3-44f3-84fd-7a9581528469-insights-runtime-extractor-tls") pod "insights-runtime-extractor-t7mz5" (UID: "7ef03827-5da3-44f3-84fd-7a9581528469") : secret "insights-runtime-extractor-tls" not found
Sep 19 04:14:31.319524 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:31.319524    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7ef03827-5da3-44f3-84fd-7a9581528469-kube-rbac-proxy-cm podName:7ef03827-5da3-44f3-84fd-7a9581528469 nodeName:}" failed. No retries permitted until 2026-09-19 04:14:33.319511636 +0000 UTC m=+52.502472762 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7ef03827-5da3-44f3-84fd-7a9581528469-kube-rbac-proxy-cm") pod "insights-runtime-extractor-t7mz5" (UID: "7ef03827-5da3-44f3-84fd-7a9581528469") : configmap "kube-rbac-proxy" not found
Sep 19 04:14:31.622019 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:31.621924    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7a6cf6e3-7960-4902-9c40-375a4de502ac-metrics-tls\") pod \"dns-default-n8jb9\" (UID: \"7a6cf6e3-7960-4902-9c40-375a4de502ac\") " pod="openshift-dns/dns-default-n8jb9"
Sep 19 04:14:31.622181 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:31.622073    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 19 04:14:31.622181 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:31.622141    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a6cf6e3-7960-4902-9c40-375a4de502ac-metrics-tls podName:7a6cf6e3-7960-4902-9c40-375a4de502ac nodeName:}" failed. No retries permitted until 2026-09-19 04:14:47.622121874 +0000 UTC m=+66.805083017 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7a6cf6e3-7960-4902-9c40-375a4de502ac-metrics-tls") pod "dns-default-n8jb9" (UID: "7a6cf6e3-7960-4902-9c40-375a4de502ac") : secret "dns-default-metrics-tls" not found
Sep 19 04:14:32.255439 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:32.255410    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5bzkh_a8d1a2a7-a76e-4741-be3a-d87124254f5d/node-ca/0.log"
Sep 19 04:14:33.056738 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:33.056709    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8759c_e9d28ff6-a630-4517-acd6-2c596abcef2a/serve-healthcheck-canary/0.log"
Sep 19 04:14:33.334977 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:33.334881    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7ef03827-5da3-44f3-84fd-7a9581528469-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t7mz5\" (UID: \"7ef03827-5da3-44f3-84fd-7a9581528469\") " pod="openshift-insights/insights-runtime-extractor-t7mz5"
Sep 19 04:14:33.334977 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:33.334969    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7ef03827-5da3-44f3-84fd-7a9581528469-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t7mz5\" (UID: \"7ef03827-5da3-44f3-84fd-7a9581528469\") " pod="openshift-insights/insights-runtime-extractor-t7mz5"
Sep 19 04:14:33.335211 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:33.335014    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 19 04:14:33.335211 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:33.335123    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 19 04:14:33.335211 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:33.335129    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7ef03827-5da3-44f3-84fd-7a9581528469-kube-rbac-proxy-cm podName:7ef03827-5da3-44f3-84fd-7a9581528469 nodeName:}" failed. No retries permitted until 2026-09-19 04:14:37.335113435 +0000 UTC m=+56.518074565 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7ef03827-5da3-44f3-84fd-7a9581528469-kube-rbac-proxy-cm") pod "insights-runtime-extractor-t7mz5" (UID: "7ef03827-5da3-44f3-84fd-7a9581528469") : configmap "kube-rbac-proxy" not found
Sep 19 04:14:33.335211 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:33.335199    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ef03827-5da3-44f3-84fd-7a9581528469-insights-runtime-extractor-tls podName:7ef03827-5da3-44f3-84fd-7a9581528469 nodeName:}" failed. No retries permitted until 2026-09-19 04:14:37.335181892 +0000 UTC m=+56.518143023 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7ef03827-5da3-44f3-84fd-7a9581528469-insights-runtime-extractor-tls") pod "insights-runtime-extractor-t7mz5" (UID: "7ef03827-5da3-44f3-84fd-7a9581528469") : secret "insights-runtime-extractor-tls" not found
Sep 19 04:14:33.436377 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:33.436341    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1ebe82f-7422-4a2c-9f72-7eef8335ffb2-metrics-certs\") pod \"router-default-6cddf8756d-5pc66\" (UID: \"c1ebe82f-7422-4a2c-9f72-7eef8335ffb2\") " pod="openshift-ingress/router-default-6cddf8756d-5pc66"
Sep 19 04:14:33.436537 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:33.436389    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1ebe82f-7422-4a2c-9f72-7eef8335ffb2-service-ca-bundle\") pod \"router-default-6cddf8756d-5pc66\" (UID: \"c1ebe82f-7422-4a2c-9f72-7eef8335ffb2\") " pod="openshift-ingress/router-default-6cddf8756d-5pc66"
Sep 19 04:14:33.436537 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:33.436497    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 19 04:14:33.436537 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:33.436533    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c1ebe82f-7422-4a2c-9f72-7eef8335ffb2-service-ca-bundle podName:c1ebe82f-7422-4a2c-9f72-7eef8335ffb2 nodeName:}" failed. No retries permitted until 2026-09-19 04:14:41.43652038 +0000 UTC m=+60.619481518 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c1ebe82f-7422-4a2c-9f72-7eef8335ffb2-service-ca-bundle") pod "router-default-6cddf8756d-5pc66" (UID: "c1ebe82f-7422-4a2c-9f72-7eef8335ffb2") : configmap references non-existent config key: service-ca.crt
Sep 19 04:14:33.436648 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:33.436549    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1ebe82f-7422-4a2c-9f72-7eef8335ffb2-metrics-certs podName:c1ebe82f-7422-4a2c-9f72-7eef8335ffb2 nodeName:}" failed. No retries permitted until 2026-09-19 04:14:41.436543799 +0000 UTC m=+60.619504924 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c1ebe82f-7422-4a2c-9f72-7eef8335ffb2-metrics-certs") pod "router-default-6cddf8756d-5pc66" (UID: "c1ebe82f-7422-4a2c-9f72-7eef8335ffb2") : secret "router-metrics-certs-default" not found
Sep 19 04:14:37.365539 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:37.365494    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7ef03827-5da3-44f3-84fd-7a9581528469-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t7mz5\" (UID: \"7ef03827-5da3-44f3-84fd-7a9581528469\") " pod="openshift-insights/insights-runtime-extractor-t7mz5"
Sep 19 04:14:37.366005 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:37.365599    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7ef03827-5da3-44f3-84fd-7a9581528469-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t7mz5\" (UID: \"7ef03827-5da3-44f3-84fd-7a9581528469\") " pod="openshift-insights/insights-runtime-extractor-t7mz5"
Sep 19 04:14:37.366005 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:37.365631    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 19 04:14:37.366005 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:37.365677    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 19 04:14:37.366005 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:37.365689    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ef03827-5da3-44f3-84fd-7a9581528469-insights-runtime-extractor-tls podName:7ef03827-5da3-44f3-84fd-7a9581528469 nodeName:}" failed. No retries permitted until 2026-09-19 04:14:45.365675312 +0000 UTC m=+64.548636437 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7ef03827-5da3-44f3-84fd-7a9581528469-insights-runtime-extractor-tls") pod "insights-runtime-extractor-t7mz5" (UID: "7ef03827-5da3-44f3-84fd-7a9581528469") : secret "insights-runtime-extractor-tls" not found
Sep 19 04:14:37.366005 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:37.365737    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7ef03827-5da3-44f3-84fd-7a9581528469-kube-rbac-proxy-cm podName:7ef03827-5da3-44f3-84fd-7a9581528469 nodeName:}" failed. No retries permitted until 2026-09-19 04:14:45.365723568 +0000 UTC m=+64.548684693 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7ef03827-5da3-44f3-84fd-7a9581528469-kube-rbac-proxy-cm") pod "insights-runtime-extractor-t7mz5" (UID: "7ef03827-5da3-44f3-84fd-7a9581528469") : configmap "kube-rbac-proxy" not found
Sep 19 04:14:38.904364 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:38.904331    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-fbc69c679-f86tq"]
Sep 19 04:14:38.908446 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:38.908430    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-fbc69c679-f86tq"
Sep 19 04:14:38.913942 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:38.913920    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 19 04:14:38.914862 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:38.914843    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 19 04:14:38.914987 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:38.914968    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 19 04:14:38.915933 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:38.915916    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-tnxhx\""
Sep 19 04:14:38.920818 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:38.920800    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 19 04:14:38.928214 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:38.928193    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-fbc69c679-f86tq"]
Sep 19 04:14:38.977713 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:38.977673    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/91ea3d76-ba23-42cc-80d5-c55cecf6e264-image-registry-private-configuration\") pod \"image-registry-fbc69c679-f86tq\" (UID: \"91ea3d76-ba23-42cc-80d5-c55cecf6e264\") " pod="openshift-image-registry/image-registry-fbc69c679-f86tq"
Sep 19 04:14:38.977849 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:38.977726    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/91ea3d76-ba23-42cc-80d5-c55cecf6e264-installation-pull-secrets\") pod \"image-registry-fbc69c679-f86tq\" (UID: \"91ea3d76-ba23-42cc-80d5-c55cecf6e264\") " pod="openshift-image-registry/image-registry-fbc69c679-f86tq"
Sep 19 04:14:38.977849 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:38.977757    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/91ea3d76-ba23-42cc-80d5-c55cecf6e264-registry-tls\") pod \"image-registry-fbc69c679-f86tq\" (UID: \"91ea3d76-ba23-42cc-80d5-c55cecf6e264\") " pod="openshift-image-registry/image-registry-fbc69c679-f86tq"
Sep 19 04:14:38.977849 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:38.977804    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/91ea3d76-ba23-42cc-80d5-c55cecf6e264-trusted-ca\") pod \"image-registry-fbc69c679-f86tq\" (UID: \"91ea3d76-ba23-42cc-80d5-c55cecf6e264\") " pod="openshift-image-registry/image-registry-fbc69c679-f86tq"
Sep 19 04:14:38.977849 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:38.977835    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/91ea3d76-ba23-42cc-80d5-c55cecf6e264-ca-trust-extracted\") pod \"image-registry-fbc69c679-f86tq\" (UID: \"91ea3d76-ba23-42cc-80d5-c55cecf6e264\") " pod="openshift-image-registry/image-registry-fbc69c679-f86tq"
Sep 19 04:14:38.977996 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:38.977876    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/91ea3d76-ba23-42cc-80d5-c55cecf6e264-registry-certificates\") pod \"image-registry-fbc69c679-f86tq\" (UID: \"91ea3d76-ba23-42cc-80d5-c55cecf6e264\") " pod="openshift-image-registry/image-registry-fbc69c679-f86tq"
Sep 19 04:14:38.977996 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:38.977938    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/91ea3d76-ba23-42cc-80d5-c55cecf6e264-bound-sa-token\") pod \"image-registry-fbc69c679-f86tq\" (UID: \"91ea3d76-ba23-42cc-80d5-c55cecf6e264\") " pod="openshift-image-registry/image-registry-fbc69c679-f86tq"
Sep 19 04:14:38.977996 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:38.977991    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqlqk\" (UniqueName: \"kubernetes.io/projected/91ea3d76-ba23-42cc-80d5-c55cecf6e264-kube-api-access-nqlqk\") pod \"image-registry-fbc69c679-f86tq\" (UID: \"91ea3d76-ba23-42cc-80d5-c55cecf6e264\") " pod="openshift-image-registry/image-registry-fbc69c679-f86tq"
Sep 19 04:14:39.079042 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:39.079009    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/91ea3d76-ba23-42cc-80d5-c55cecf6e264-image-registry-private-configuration\") pod \"image-registry-fbc69c679-f86tq\" (UID: \"91ea3d76-ba23-42cc-80d5-c55cecf6e264\") " pod="openshift-image-registry/image-registry-fbc69c679-f86tq"
Sep 19 04:14:39.079042 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:39.079042    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/91ea3d76-ba23-42cc-80d5-c55cecf6e264-installation-pull-secrets\") pod \"image-registry-fbc69c679-f86tq\" (UID: \"91ea3d76-ba23-42cc-80d5-c55cecf6e264\") " pod="openshift-image-registry/image-registry-fbc69c679-f86tq"
Sep 19 04:14:39.079275 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:39.079063    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/91ea3d76-ba23-42cc-80d5-c55cecf6e264-registry-tls\") pod \"image-registry-fbc69c679-f86tq\" (UID: \"91ea3d76-ba23-42cc-80d5-c55cecf6e264\") " pod="openshift-image-registry/image-registry-fbc69c679-f86tq"
Sep 19 04:14:39.079275 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:39.079105    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/91ea3d76-ba23-42cc-80d5-c55cecf6e264-trusted-ca\") pod \"image-registry-fbc69c679-f86tq\" (UID: \"91ea3d76-ba23-42cc-80d5-c55cecf6e264\") " pod="openshift-image-registry/image-registry-fbc69c679-f86tq"
Sep 19 04:14:39.079275 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:39.079134    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/91ea3d76-ba23-42cc-80d5-c55cecf6e264-ca-trust-extracted\") pod \"image-registry-fbc69c679-f86tq\" (UID: \"91ea3d76-ba23-42cc-80d5-c55cecf6e264\") " pod="openshift-image-registry/image-registry-fbc69c679-f86tq"
Sep 19 04:14:39.079275 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:39.079174    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/91ea3d76-ba23-42cc-80d5-c55cecf6e264-registry-certificates\") pod \"image-registry-fbc69c679-f86tq\" (UID: \"91ea3d76-ba23-42cc-80d5-c55cecf6e264\") " pod="openshift-image-registry/image-registry-fbc69c679-f86tq"
Sep 19 04:14:39.079275 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:39.079185    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 19 04:14:39.079275 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:39.079206    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-fbc69c679-f86tq: secret "image-registry-tls" not found
Sep 19 04:14:39.079578 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:39.079290    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/91ea3d76-ba23-42cc-80d5-c55cecf6e264-registry-tls podName:91ea3d76-ba23-42cc-80d5-c55cecf6e264 nodeName:}" failed. No retries permitted until 2026-09-19 04:14:39.579265279 +0000 UTC m=+58.762226404 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/91ea3d76-ba23-42cc-80d5-c55cecf6e264-registry-tls") pod "image-registry-fbc69c679-f86tq" (UID: "91ea3d76-ba23-42cc-80d5-c55cecf6e264") : secret "image-registry-tls" not found
Sep 19 04:14:39.079578 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:39.079405    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/91ea3d76-ba23-42cc-80d5-c55cecf6e264-bound-sa-token\") pod \"image-registry-fbc69c679-f86tq\" (UID: \"91ea3d76-ba23-42cc-80d5-c55cecf6e264\") " pod="openshift-image-registry/image-registry-fbc69c679-f86tq"
Sep 19 04:14:39.079578 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:39.079472    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqlqk\" (UniqueName: \"kubernetes.io/projected/91ea3d76-ba23-42cc-80d5-c55cecf6e264-kube-api-access-nqlqk\") pod \"image-registry-fbc69c679-f86tq\" (UID: \"91ea3d76-ba23-42cc-80d5-c55cecf6e264\") " pod="openshift-image-registry/image-registry-fbc69c679-f86tq"
Sep 19 04:14:39.079764 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:39.079608    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/91ea3d76-ba23-42cc-80d5-c55cecf6e264-ca-trust-extracted\") pod \"image-registry-fbc69c679-f86tq\" (UID: \"91ea3d76-ba23-42cc-80d5-c55cecf6e264\") " pod="openshift-image-registry/image-registry-fbc69c679-f86tq"
Sep 19 04:14:39.080092 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:39.080068    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/91ea3d76-ba23-42cc-80d5-c55cecf6e264-registry-certificates\") pod \"image-registry-fbc69c679-f86tq\" (UID: \"91ea3d76-ba23-42cc-80d5-c55cecf6e264\") " pod="openshift-image-registry/image-registry-fbc69c679-f86tq"
Sep 19 04:14:39.080713 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:39.080673    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/91ea3d76-ba23-42cc-80d5-c55cecf6e264-trusted-ca\") pod \"image-registry-fbc69c679-f86tq\" (UID: \"91ea3d76-ba23-42cc-80d5-c55cecf6e264\") " pod="openshift-image-registry/image-registry-fbc69c679-f86tq"
Sep 19 04:14:39.081723 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:39.081673    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/91ea3d76-ba23-42cc-80d5-c55cecf6e264-installation-pull-secrets\") pod \"image-registry-fbc69c679-f86tq\" (UID: \"91ea3d76-ba23-42cc-80d5-c55cecf6e264\") " pod="openshift-image-registry/image-registry-fbc69c679-f86tq"
Sep 19 04:14:39.081817 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:39.081752    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/91ea3d76-ba23-42cc-80d5-c55cecf6e264-image-registry-private-configuration\") pod \"image-registry-fbc69c679-f86tq\" (UID: \"91ea3d76-ba23-42cc-80d5-c55cecf6e264\") " pod="openshift-image-registry/image-registry-fbc69c679-f86tq"
Sep 19 04:14:39.092496 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:39.092469    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/91ea3d76-ba23-42cc-80d5-c55cecf6e264-bound-sa-token\") pod \"image-registry-fbc69c679-f86tq\" (UID: \"91ea3d76-ba23-42cc-80d5-c55cecf6e264\") " pod="openshift-image-registry/image-registry-fbc69c679-f86tq"
Sep 19 04:14:39.094425 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:39.094405    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqlqk\" (UniqueName: \"kubernetes.io/projected/91ea3d76-ba23-42cc-80d5-c55cecf6e264-kube-api-access-nqlqk\") pod \"image-registry-fbc69c679-f86tq\" (UID: \"91ea3d76-ba23-42cc-80d5-c55cecf6e264\") " pod="openshift-image-registry/image-registry-fbc69c679-f86tq"
Sep 19 04:14:39.583124 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:39.583086    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/91ea3d76-ba23-42cc-80d5-c55cecf6e264-registry-tls\") pod \"image-registry-fbc69c679-f86tq\" (UID: \"91ea3d76-ba23-42cc-80d5-c55cecf6e264\") " pod="openshift-image-registry/image-registry-fbc69c679-f86tq"
Sep 19 04:14:39.583286 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:39.583197    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 19 04:14:39.583286 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:39.583208    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-fbc69c679-f86tq: secret "image-registry-tls" not found
Sep 19 04:14:39.583286 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:39.583257    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/91ea3d76-ba23-42cc-80d5-c55cecf6e264-registry-tls podName:91ea3d76-ba23-42cc-80d5-c55cecf6e264 nodeName:}" failed. No retries permitted until 2026-09-19 04:14:40.58324399 +0000 UTC m=+59.766205116 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/91ea3d76-ba23-42cc-80d5-c55cecf6e264-registry-tls") pod "image-registry-fbc69c679-f86tq" (UID: "91ea3d76-ba23-42cc-80d5-c55cecf6e264") : secret "image-registry-tls" not found
Sep 19 04:14:40.591198 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:40.591162    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/91ea3d76-ba23-42cc-80d5-c55cecf6e264-registry-tls\") pod \"image-registry-fbc69c679-f86tq\" (UID: \"91ea3d76-ba23-42cc-80d5-c55cecf6e264\") " pod="openshift-image-registry/image-registry-fbc69c679-f86tq"
Sep 19 04:14:40.591603 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:40.591282    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 19 04:14:40.591603 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:40.591295    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-fbc69c679-f86tq: secret "image-registry-tls" not found
Sep 19 04:14:40.591603 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:40.591356    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/91ea3d76-ba23-42cc-80d5-c55cecf6e264-registry-tls podName:91ea3d76-ba23-42cc-80d5-c55cecf6e264 nodeName:}" failed. No retries permitted until 2026-09-19 04:14:42.591341281 +0000 UTC m=+61.774302406 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/91ea3d76-ba23-42cc-80d5-c55cecf6e264-registry-tls") pod "image-registry-fbc69c679-f86tq" (UID: "91ea3d76-ba23-42cc-80d5-c55cecf6e264") : secret "image-registry-tls" not found
Sep 19 04:14:41.498512 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:41.498479    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1ebe82f-7422-4a2c-9f72-7eef8335ffb2-metrics-certs\") pod \"router-default-6cddf8756d-5pc66\" (UID: \"c1ebe82f-7422-4a2c-9f72-7eef8335ffb2\") " pod="openshift-ingress/router-default-6cddf8756d-5pc66"
Sep 19 04:14:41.498690 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:41.498524    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1ebe82f-7422-4a2c-9f72-7eef8335ffb2-service-ca-bundle\") pod \"router-default-6cddf8756d-5pc66\" (UID: \"c1ebe82f-7422-4a2c-9f72-7eef8335ffb2\") " pod="openshift-ingress/router-default-6cddf8756d-5pc66"
Sep 19 04:14:41.498690 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:41.498643    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 19 04:14:41.498690 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:41.498660    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c1ebe82f-7422-4a2c-9f72-7eef8335ffb2-service-ca-bundle podName:c1ebe82f-7422-4a2c-9f72-7eef8335ffb2 nodeName:}" failed. No retries permitted until 2026-09-19 04:14:57.498644445 +0000 UTC m=+76.681605594 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c1ebe82f-7422-4a2c-9f72-7eef8335ffb2-service-ca-bundle") pod "router-default-6cddf8756d-5pc66" (UID: "c1ebe82f-7422-4a2c-9f72-7eef8335ffb2") : configmap references non-existent config key: service-ca.crt
Sep 19 04:14:41.498829 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:41.498721    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1ebe82f-7422-4a2c-9f72-7eef8335ffb2-metrics-certs podName:c1ebe82f-7422-4a2c-9f72-7eef8335ffb2 nodeName:}" failed. No retries permitted until 2026-09-19 04:14:57.498687308 +0000 UTC m=+76.681648434 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c1ebe82f-7422-4a2c-9f72-7eef8335ffb2-metrics-certs") pod "router-default-6cddf8756d-5pc66" (UID: "c1ebe82f-7422-4a2c-9f72-7eef8335ffb2") : secret "router-metrics-certs-default" not found
Sep 19 04:14:41.530750 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:41.530717    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rtwn6"
Sep 19 04:14:42.607741 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:42.607689    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/91ea3d76-ba23-42cc-80d5-c55cecf6e264-registry-tls\") pod \"image-registry-fbc69c679-f86tq\" (UID: \"91ea3d76-ba23-42cc-80d5-c55cecf6e264\") " pod="openshift-image-registry/image-registry-fbc69c679-f86tq"
Sep 19 04:14:42.608102 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:42.607799    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 19 04:14:42.608102 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:42.607818    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-fbc69c679-f86tq: secret "image-registry-tls" not found
Sep 19 04:14:42.608102 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:42.607875    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/91ea3d76-ba23-42cc-80d5-c55cecf6e264-registry-tls podName:91ea3d76-ba23-42cc-80d5-c55cecf6e264 nodeName:}" failed. No retries permitted until 2026-09-19 04:14:46.607858541 +0000 UTC m=+65.790819667 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/91ea3d76-ba23-42cc-80d5-c55cecf6e264-registry-tls") pod "image-registry-fbc69c679-f86tq" (UID: "91ea3d76-ba23-42cc-80d5-c55cecf6e264") : secret "image-registry-tls" not found
Sep 19 04:14:45.429149 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:45.429097    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7ef03827-5da3-44f3-84fd-7a9581528469-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t7mz5\" (UID: \"7ef03827-5da3-44f3-84fd-7a9581528469\") " pod="openshift-insights/insights-runtime-extractor-t7mz5"
Sep 19 04:14:45.429552 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:45.429198    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7ef03827-5da3-44f3-84fd-7a9581528469-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t7mz5\" (UID: \"7ef03827-5da3-44f3-84fd-7a9581528469\") " pod="openshift-insights/insights-runtime-extractor-t7mz5"
Sep 19 04:14:45.429552 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:45.429253    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 19 04:14:45.429552 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:45.429328    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ef03827-5da3-44f3-84fd-7a9581528469-insights-runtime-extractor-tls podName:7ef03827-5da3-44f3-84fd-7a9581528469 nodeName:}" failed. No retries permitted until 2026-09-19 04:15:01.429312611 +0000 UTC m=+80.612273738 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7ef03827-5da3-44f3-84fd-7a9581528469-insights-runtime-extractor-tls") pod "insights-runtime-extractor-t7mz5" (UID: "7ef03827-5da3-44f3-84fd-7a9581528469") : secret "insights-runtime-extractor-tls" not found
Sep 19 04:14:45.429659 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:45.429649    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7ef03827-5da3-44f3-84fd-7a9581528469-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t7mz5\" (UID: \"7ef03827-5da3-44f3-84fd-7a9581528469\") " pod="openshift-insights/insights-runtime-extractor-t7mz5"
Sep 19 04:14:46.639963 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:46.639928    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/91ea3d76-ba23-42cc-80d5-c55cecf6e264-registry-tls\") pod \"image-registry-fbc69c679-f86tq\" (UID: \"91ea3d76-ba23-42cc-80d5-c55cecf6e264\") " pod="openshift-image-registry/image-registry-fbc69c679-f86tq"
Sep 19 04:14:46.640330 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:46.640039    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 19 04:14:46.640330 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:46.640050    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-fbc69c679-f86tq: secret "image-registry-tls" not found
Sep 19 04:14:46.640330 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:46.640111    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/91ea3d76-ba23-42cc-80d5-c55cecf6e264-registry-tls podName:91ea3d76-ba23-42cc-80d5-c55cecf6e264 nodeName:}" failed. No retries permitted until 2026-09-19 04:14:54.640095738 +0000 UTC m=+73.823056887 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/91ea3d76-ba23-42cc-80d5-c55cecf6e264-registry-tls") pod "image-registry-fbc69c679-f86tq" (UID: "91ea3d76-ba23-42cc-80d5-c55cecf6e264") : secret "image-registry-tls" not found
Sep 19 04:14:47.042403 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:47.042364    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/64cb49fc-ede9-4369-aa8b-76a95d1edab6-original-pull-secret\") pod \"global-pull-secret-syncer-5t728\" (UID: \"64cb49fc-ede9-4369-aa8b-76a95d1edab6\") " pod="kube-system/global-pull-secret-syncer-5t728"
Sep 19 04:14:47.042575 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:47.042488    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3738f1c7-67af-4201-8e8d-1d0406e0edb9-metrics-certs\") pod \"network-metrics-daemon-wmk74\" (UID: \"3738f1c7-67af-4201-8e8d-1d0406e0edb9\") " pod="openshift-multus/network-metrics-daemon-wmk74"
Sep 19 04:14:47.050953 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:47.050935    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 19 04:14:47.051504 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:47.051492    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 19 04:14:47.052633 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:47.052619    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 19 04:14:47.052675 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:47.052669    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3738f1c7-67af-4201-8e8d-1d0406e0edb9-metrics-certs podName:3738f1c7-67af-4201-8e8d-1d0406e0edb9 nodeName:}" failed. No retries permitted until 2026-09-19 04:15:51.05265484 +0000 UTC m=+130.235615965 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3738f1c7-67af-4201-8e8d-1d0406e0edb9-metrics-certs") pod "network-metrics-daemon-wmk74" (UID: "3738f1c7-67af-4201-8e8d-1d0406e0edb9") : secret "metrics-daemon-secret" not found
Sep 19 04:14:47.054941 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:47.054922    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/64cb49fc-ede9-4369-aa8b-76a95d1edab6-original-pull-secret\") pod \"global-pull-secret-syncer-5t728\" (UID: \"64cb49fc-ede9-4369-aa8b-76a95d1edab6\") " pod="kube-system/global-pull-secret-syncer-5t728"
Sep 19 04:14:47.062942 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:47.062920    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5t728"
Sep 19 04:14:47.143197 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:47.143168    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j856z\" (UniqueName: \"kubernetes.io/projected/f2fe8238-2a1d-4e7a-b4d7-4cc7d518f366-kube-api-access-j856z\") pod \"network-check-target-z7dgk\" (UID: \"f2fe8238-2a1d-4e7a-b4d7-4cc7d518f366\") " pod="openshift-network-diagnostics/network-check-target-z7dgk"
Sep 19 04:14:47.146549 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:47.146528    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 19 04:14:47.156839 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:47.156815    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 19 04:14:47.166367 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:47.166344    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j856z\" (UniqueName: \"kubernetes.io/projected/f2fe8238-2a1d-4e7a-b4d7-4cc7d518f366-kube-api-access-j856z\") pod \"network-check-target-z7dgk\" (UID: \"f2fe8238-2a1d-4e7a-b4d7-4cc7d518f366\") " pod="openshift-network-diagnostics/network-check-target-z7dgk"
Sep 19 04:14:47.257586 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:47.257567    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-rwkvg\""
Sep 19 04:14:47.264424 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:47.264409    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7dgk"
Sep 19 04:14:47.364282 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:47.364247    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5t728"]
Sep 19 04:14:47.367572 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:14:47.367551    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64cb49fc_ede9_4369_aa8b_76a95d1edab6.slice/crio-de142aa7d7e044c311ae7804f3c9a963fdcd39a7aece97a553217b03451f64ae WatchSource:0}: Error finding container de142aa7d7e044c311ae7804f3c9a963fdcd39a7aece97a553217b03451f64ae: Status 404 returned error can't find the container with id de142aa7d7e044c311ae7804f3c9a963fdcd39a7aece97a553217b03451f64ae
Sep 19 04:14:47.564275 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:47.564217    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-z7dgk"]
Sep 19 04:14:47.564847 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:14:47.564825    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2fe8238_2a1d_4e7a_b4d7_4cc7d518f366.slice/crio-d15d163dba4b68b2ce56090e3e9500ea9443265ae976591e7be04be2f2541f33 WatchSource:0}: Error finding container d15d163dba4b68b2ce56090e3e9500ea9443265ae976591e7be04be2f2541f33: Status 404 returned error can't find the container with id d15d163dba4b68b2ce56090e3e9500ea9443265ae976591e7be04be2f2541f33
Sep 19 04:14:47.632937 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:47.632842    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5t728" event={"ID":"64cb49fc-ede9-4369-aa8b-76a95d1edab6","Type":"ContainerStarted","Data":"de142aa7d7e044c311ae7804f3c9a963fdcd39a7aece97a553217b03451f64ae"}
Sep 19 04:14:47.633939 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:47.633912    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-z7dgk" event={"ID":"f2fe8238-2a1d-4e7a-b4d7-4cc7d518f366","Type":"ContainerStarted","Data":"d15d163dba4b68b2ce56090e3e9500ea9443265ae976591e7be04be2f2541f33"}
Sep 19 04:14:47.647357 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:47.647334    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7a6cf6e3-7960-4902-9c40-375a4de502ac-metrics-tls\") pod \"dns-default-n8jb9\" (UID: \"7a6cf6e3-7960-4902-9c40-375a4de502ac\") " pod="openshift-dns/dns-default-n8jb9"
Sep 19 04:14:47.647777 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:47.647512    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 19 04:14:47.647777 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:14:47.647596    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a6cf6e3-7960-4902-9c40-375a4de502ac-metrics-tls podName:7a6cf6e3-7960-4902-9c40-375a4de502ac nodeName:}" failed. No retries permitted until 2026-09-19 04:15:19.647573872 +0000 UTC m=+98.830535021 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7a6cf6e3-7960-4902-9c40-375a4de502ac-metrics-tls") pod "dns-default-n8jb9" (UID: "7a6cf6e3-7960-4902-9c40-375a4de502ac") : secret "dns-default-metrics-tls" not found
Sep 19 04:14:52.648849 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:52.648810    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5t728" event={"ID":"64cb49fc-ede9-4369-aa8b-76a95d1edab6","Type":"ContainerStarted","Data":"739f8f08566a1afc986fd2c0e7521f25bfbde946e04f93209003f0b2c2a4e4b5"}
Sep 19 04:14:52.650373 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:52.650352    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-z7dgk" event={"ID":"f2fe8238-2a1d-4e7a-b4d7-4cc7d518f366","Type":"ContainerStarted","Data":"c30d9e75510e10cca537d63394ff93267e333ab267a0e8e35ac9ec99c1f14d7c"}
Sep 19 04:14:52.650481 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:52.650395    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-z7dgk"
Sep 19 04:14:52.669254 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:52.669213    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-5t728" podStartSLOduration=67.147519253 podStartE2EDuration="1m11.669201107s" totalImagesPullingTime="4.521681854s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 04:13:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:14:52.668573429 +0000 UTC m=+71.851534578" watchObservedRunningTime="2026-09-19 04:14:52.669201107 +0000 UTC m=+71.852162256"
Sep 19 04:14:52.691627 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:52.691592    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-z7dgk" podStartSLOduration=67.371996263 podStartE2EDuration="1m11.691583106s" totalImagesPullingTime="4.319586843s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 04:13:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:14:52.691253731 +0000 UTC m=+71.874214890" watchObservedRunningTime="2026-09-19 04:14:52.691583106 +0000 UTC m=+71.874544254"
Sep 19 04:14:54.706934 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:54.706896    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/91ea3d76-ba23-42cc-80d5-c55cecf6e264-registry-tls\") pod \"image-registry-fbc69c679-f86tq\" (UID: \"91ea3d76-ba23-42cc-80d5-c55cecf6e264\") " pod="openshift-image-registry/image-registry-fbc69c679-f86tq"
Sep 19 04:14:54.709316 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:54.709296    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/91ea3d76-ba23-42cc-80d5-c55cecf6e264-registry-tls\") pod \"image-registry-fbc69c679-f86tq\" (UID: \"91ea3d76-ba23-42cc-80d5-c55cecf6e264\") " pod="openshift-image-registry/image-registry-fbc69c679-f86tq"
Sep 19 04:14:54.820807 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:54.820780    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-tnxhx\""
Sep 19 04:14:54.828229 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:54.828194    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-fbc69c679-f86tq"
Sep 19 04:14:55.108259 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:55.108181    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-fbc69c679-f86tq"]
Sep 19 04:14:55.111495 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:14:55.111473    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91ea3d76_ba23_42cc_80d5_c55cecf6e264.slice/crio-d11fd4a6dbf04d77ae4316b1499443fdc120d567637304cbdcb47f74d1978e0c WatchSource:0}: Error finding container d11fd4a6dbf04d77ae4316b1499443fdc120d567637304cbdcb47f74d1978e0c: Status 404 returned error can't find the container with id d11fd4a6dbf04d77ae4316b1499443fdc120d567637304cbdcb47f74d1978e0c
Sep 19 04:14:55.660168 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:55.660127    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-fbc69c679-f86tq" event={"ID":"91ea3d76-ba23-42cc-80d5-c55cecf6e264","Type":"ContainerStarted","Data":"d808a368b62b84c99c99b2a1b6b4f7b4394669daf3e028ddf870afe6dc3d4b54"}
Sep 19 04:14:55.660361 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:55.660187    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-fbc69c679-f86tq" event={"ID":"91ea3d76-ba23-42cc-80d5-c55cecf6e264","Type":"ContainerStarted","Data":"d11fd4a6dbf04d77ae4316b1499443fdc120d567637304cbdcb47f74d1978e0c"}
Sep 19 04:14:55.660361 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:55.660272    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-fbc69c679-f86tq"
Sep 19 04:14:55.690301 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:55.690258    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-fbc69c679-f86tq" podStartSLOduration=17.690243996 podStartE2EDuration="17.690243996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 04:14:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:14:55.689236134 +0000 UTC m=+74.872197282" watchObservedRunningTime="2026-09-19 04:14:55.690243996 +0000 UTC m=+74.873205145"
Sep 19 04:14:57.528509 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:57.528465    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1ebe82f-7422-4a2c-9f72-7eef8335ffb2-metrics-certs\") pod \"router-default-6cddf8756d-5pc66\" (UID: \"c1ebe82f-7422-4a2c-9f72-7eef8335ffb2\") " pod="openshift-ingress/router-default-6cddf8756d-5pc66"
Sep 19 04:14:57.528509 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:57.528509    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1ebe82f-7422-4a2c-9f72-7eef8335ffb2-service-ca-bundle\") pod \"router-default-6cddf8756d-5pc66\" (UID: \"c1ebe82f-7422-4a2c-9f72-7eef8335ffb2\") " pod="openshift-ingress/router-default-6cddf8756d-5pc66"
Sep 19 04:14:57.529147 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:57.529129    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1ebe82f-7422-4a2c-9f72-7eef8335ffb2-service-ca-bundle\") pod \"router-default-6cddf8756d-5pc66\" (UID: \"c1ebe82f-7422-4a2c-9f72-7eef8335ffb2\") " pod="openshift-ingress/router-default-6cddf8756d-5pc66"
Sep 19 04:14:57.530979 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:57.530951    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1ebe82f-7422-4a2c-9f72-7eef8335ffb2-metrics-certs\") pod \"router-default-6cddf8756d-5pc66\" (UID: \"c1ebe82f-7422-4a2c-9f72-7eef8335ffb2\") " pod="openshift-ingress/router-default-6cddf8756d-5pc66"
Sep 19 04:14:57.714450 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:57.714419    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-2tj52\""
Sep 19 04:14:57.722866 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:57.722838    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6cddf8756d-5pc66"
Sep 19 04:14:57.997056 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:57.997027    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6cddf8756d-5pc66"]
Sep 19 04:14:58.000528 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:14:58.000498    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1ebe82f_7422_4a2c_9f72_7eef8335ffb2.slice/crio-0cd75043f8314bb0adae009520f908e3b660e95f3060bc99383a6a44f21977b7 WatchSource:0}: Error finding container 0cd75043f8314bb0adae009520f908e3b660e95f3060bc99383a6a44f21977b7: Status 404 returned error can't find the container with id 0cd75043f8314bb0adae009520f908e3b660e95f3060bc99383a6a44f21977b7
Sep 19 04:14:58.669991 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:58.669955    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6cddf8756d-5pc66" event={"ID":"c1ebe82f-7422-4a2c-9f72-7eef8335ffb2","Type":"ContainerStarted","Data":"887be104f9831e6d0f000d3e84785ade2b3964f88bb59e402a5c4b769013e676"}
Sep 19 04:14:58.670374 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:58.670007    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6cddf8756d-5pc66" event={"ID":"c1ebe82f-7422-4a2c-9f72-7eef8335ffb2","Type":"ContainerStarted","Data":"0cd75043f8314bb0adae009520f908e3b660e95f3060bc99383a6a44f21977b7"}
Sep 19 04:14:58.695634 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:58.695585    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-6cddf8756d-5pc66" podStartSLOduration=33.695569822 podStartE2EDuration="33.695569822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 04:14:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:14:58.694325697 +0000 UTC m=+77.877286852" watchObservedRunningTime="2026-09-19 04:14:58.695569822 +0000 UTC m=+77.878530969"
Sep 19 04:14:58.723179 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:58.723144    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-6cddf8756d-5pc66"
Sep 19 04:14:58.725968 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:58.725945    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-6cddf8756d-5pc66"
Sep 19 04:14:59.673360 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:59.673327    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-6cddf8756d-5pc66"
Sep 19 04:14:59.674753 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:14:59.674732    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-6cddf8756d-5pc66"
Sep 19 04:15:01.456350 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:01.456308    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7ef03827-5da3-44f3-84fd-7a9581528469-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t7mz5\" (UID: \"7ef03827-5da3-44f3-84fd-7a9581528469\") " pod="openshift-insights/insights-runtime-extractor-t7mz5"
Sep 19 04:15:01.458681 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:01.458662    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7ef03827-5da3-44f3-84fd-7a9581528469-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t7mz5\" (UID: \"7ef03827-5da3-44f3-84fd-7a9581528469\") " pod="openshift-insights/insights-runtime-extractor-t7mz5"
Sep 19 04:15:01.731561 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:01.731475    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mt2jg\""
Sep 19 04:15:01.738301 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:01.738274    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-t7mz5"
Sep 19 04:15:02.030930 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:02.030837    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-t7mz5"]
Sep 19 04:15:02.035020 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:15:02.034999    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ef03827_5da3_44f3_84fd_7a9581528469.slice/crio-53e9173244094a6755d93e9d5d478d79e5326d35a19234ef71fcb5d4602cbb85 WatchSource:0}: Error finding container 53e9173244094a6755d93e9d5d478d79e5326d35a19234ef71fcb5d4602cbb85: Status 404 returned error can't find the container with id 53e9173244094a6755d93e9d5d478d79e5326d35a19234ef71fcb5d4602cbb85
Sep 19 04:15:02.689616 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:02.689578    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-t7mz5" event={"ID":"7ef03827-5da3-44f3-84fd-7a9581528469","Type":"ContainerStarted","Data":"6566681126d216b5a3d014e66591e2ade85bded322bb6bda6627c8c818d85374"}
Sep 19 04:15:02.690108 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:02.689648    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-t7mz5" event={"ID":"7ef03827-5da3-44f3-84fd-7a9581528469","Type":"ContainerStarted","Data":"53e9173244094a6755d93e9d5d478d79e5326d35a19234ef71fcb5d4602cbb85"}
Sep 19 04:15:03.694811 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:03.694772    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-t7mz5" event={"ID":"7ef03827-5da3-44f3-84fd-7a9581528469","Type":"ContainerStarted","Data":"43ce8d4e6fbffb36d02742560b5bbdc7bbee3735e687d4a30ac37c61d5d857cb"}
Sep 19 04:15:04.699844 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:04.699813    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-t7mz5" event={"ID":"7ef03827-5da3-44f3-84fd-7a9581528469","Type":"ContainerStarted","Data":"04838f6b2a2af4f229030c7a77628a384f15cd9bc462cb03f4d73e36350c6e62"}
Sep 19 04:15:04.723561 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:04.723512    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-t7mz5" podStartSLOduration=33.727911705 podStartE2EDuration="35.723497668s" totalImagesPullingTime="1.995585963s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 04:14:29 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:15:04.72257524 +0000 UTC m=+83.905536388" watchObservedRunningTime="2026-09-19 04:15:04.723497668 +0000 UTC m=+83.906458815"
Sep 19 04:15:06.807998 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:06.807969    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb4d9c6bc-5lrpf"]
Sep 19 04:15:06.811038 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:06.811016    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb4d9c6bc-5lrpf"
Sep 19 04:15:06.815659 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:06.815634    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 19 04:15:06.816070 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:06.816051    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 19 04:15:06.816153 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:06.816070    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 19 04:15:06.816540 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:06.816526    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 19 04:15:06.816540 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:06.816534    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 19 04:15:06.816679 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:06.816660    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 19 04:15:06.816763 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:06.816674    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 19 04:15:06.834885 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:06.834851    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb4d9c6bc-5lrpf"]
Sep 19 04:15:06.900559 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:06.900510    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9741183-e1ad-425d-b44c-26aa964acd73-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6fb4d9c6bc-5lrpf\" (UID: \"c9741183-e1ad-425d-b44c-26aa964acd73\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb4d9c6bc-5lrpf"
Sep 19 04:15:06.900739 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:06.900576    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/c9741183-e1ad-425d-b44c-26aa964acd73-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6fb4d9c6bc-5lrpf\" (UID: \"c9741183-e1ad-425d-b44c-26aa964acd73\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb4d9c6bc-5lrpf"
Sep 19 04:15:06.900739 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:06.900665    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsfqm\" (UniqueName: \"kubernetes.io/projected/c9741183-e1ad-425d-b44c-26aa964acd73-kube-api-access-rsfqm\") pod \"cluster-proxy-proxy-agent-6fb4d9c6bc-5lrpf\" (UID: \"c9741183-e1ad-425d-b44c-26aa964acd73\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb4d9c6bc-5lrpf"
Sep 19 04:15:06.900739 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:06.900719    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c9741183-e1ad-425d-b44c-26aa964acd73-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6fb4d9c6bc-5lrpf\" (UID: \"c9741183-e1ad-425d-b44c-26aa964acd73\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb4d9c6bc-5lrpf"
Sep 19 04:15:06.900868 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:06.900803    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/c9741183-e1ad-425d-b44c-26aa964acd73-hub\") pod \"cluster-proxy-proxy-agent-6fb4d9c6bc-5lrpf\" (UID: \"c9741183-e1ad-425d-b44c-26aa964acd73\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb4d9c6bc-5lrpf"
Sep 19 04:15:06.900868 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:06.900825    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/c9741183-e1ad-425d-b44c-26aa964acd73-ca\") pod \"cluster-proxy-proxy-agent-6fb4d9c6bc-5lrpf\" (UID: \"c9741183-e1ad-425d-b44c-26aa964acd73\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb4d9c6bc-5lrpf"
Sep 19 04:15:07.002097 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:07.002064    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsfqm\" (UniqueName: \"kubernetes.io/projected/c9741183-e1ad-425d-b44c-26aa964acd73-kube-api-access-rsfqm\") pod \"cluster-proxy-proxy-agent-6fb4d9c6bc-5lrpf\" (UID: \"c9741183-e1ad-425d-b44c-26aa964acd73\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb4d9c6bc-5lrpf"
Sep 19 04:15:07.002245 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:07.002100    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c9741183-e1ad-425d-b44c-26aa964acd73-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6fb4d9c6bc-5lrpf\" (UID: \"c9741183-e1ad-425d-b44c-26aa964acd73\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb4d9c6bc-5lrpf"
Sep 19 04:15:07.002245 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:07.002147    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/c9741183-e1ad-425d-b44c-26aa964acd73-hub\") pod \"cluster-proxy-proxy-agent-6fb4d9c6bc-5lrpf\" (UID: \"c9741183-e1ad-425d-b44c-26aa964acd73\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb4d9c6bc-5lrpf"
Sep 19 04:15:07.002245 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:07.002165    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/c9741183-e1ad-425d-b44c-26aa964acd73-ca\") pod \"cluster-proxy-proxy-agent-6fb4d9c6bc-5lrpf\" (UID: \"c9741183-e1ad-425d-b44c-26aa964acd73\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb4d9c6bc-5lrpf"
Sep 19 04:15:07.002245 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:07.002206    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9741183-e1ad-425d-b44c-26aa964acd73-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6fb4d9c6bc-5lrpf\" (UID: \"c9741183-e1ad-425d-b44c-26aa964acd73\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb4d9c6bc-5lrpf"
Sep 19 04:15:07.002454 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:07.002233    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/c9741183-e1ad-425d-b44c-26aa964acd73-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6fb4d9c6bc-5lrpf\" (UID: \"c9741183-e1ad-425d-b44c-26aa964acd73\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb4d9c6bc-5lrpf"
Sep 19 04:15:07.003609 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:07.003567    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9741183-e1ad-425d-b44c-26aa964acd73-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6fb4d9c6bc-5lrpf\" (UID: \"c9741183-e1ad-425d-b44c-26aa964acd73\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb4d9c6bc-5lrpf"
Sep 19 04:15:07.007949 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:07.005819    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/c9741183-e1ad-425d-b44c-26aa964acd73-ca\") pod \"cluster-proxy-proxy-agent-6fb4d9c6bc-5lrpf\" (UID: \"c9741183-e1ad-425d-b44c-26aa964acd73\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb4d9c6bc-5lrpf"
Sep 19 04:15:07.007949 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:07.005842    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c9741183-e1ad-425d-b44c-26aa964acd73-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6fb4d9c6bc-5lrpf\" (UID: \"c9741183-e1ad-425d-b44c-26aa964acd73\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb4d9c6bc-5lrpf"
Sep 19 04:15:07.007949 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:07.005832    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/c9741183-e1ad-425d-b44c-26aa964acd73-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6fb4d9c6bc-5lrpf\" (UID: \"c9741183-e1ad-425d-b44c-26aa964acd73\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb4d9c6bc-5lrpf"
Sep 19 04:15:07.007949 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:07.005924    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/c9741183-e1ad-425d-b44c-26aa964acd73-hub\") pod \"cluster-proxy-proxy-agent-6fb4d9c6bc-5lrpf\" (UID: \"c9741183-e1ad-425d-b44c-26aa964acd73\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb4d9c6bc-5lrpf"
Sep 19 04:15:07.058124 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:07.058065    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsfqm\" (UniqueName: \"kubernetes.io/projected/c9741183-e1ad-425d-b44c-26aa964acd73-kube-api-access-rsfqm\") pod \"cluster-proxy-proxy-agent-6fb4d9c6bc-5lrpf\" (UID: \"c9741183-e1ad-425d-b44c-26aa964acd73\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb4d9c6bc-5lrpf"
Sep 19 04:15:07.114641 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:07.114607    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-fbc69c679-f86tq"]
Sep 19 04:15:07.135110 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:07.135088    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb4d9c6bc-5lrpf"
Sep 19 04:15:07.446520 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:07.446490    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb4d9c6bc-5lrpf"]
Sep 19 04:15:07.449096 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:15:07.449063    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc9741183_e1ad_425d_b44c_26aa964acd73.slice/crio-ff754c36fd7e9dffca6cf854688b2602bf8c0061dd7e6b5adc22005905b43e54 WatchSource:0}: Error finding container ff754c36fd7e9dffca6cf854688b2602bf8c0061dd7e6b5adc22005905b43e54: Status 404 returned error can't find the container with id ff754c36fd7e9dffca6cf854688b2602bf8c0061dd7e6b5adc22005905b43e54
Sep 19 04:15:07.710430 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:07.710346    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb4d9c6bc-5lrpf" event={"ID":"c9741183-e1ad-425d-b44c-26aa964acd73","Type":"ContainerStarted","Data":"ff754c36fd7e9dffca6cf854688b2602bf8c0061dd7e6b5adc22005905b43e54"}
Sep 19 04:15:11.726950 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:11.726916    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb4d9c6bc-5lrpf" event={"ID":"c9741183-e1ad-425d-b44c-26aa964acd73","Type":"ContainerStarted","Data":"f2641fa5da36e9cb35526ad9d487f204cdb9e479a12bc7ef5352b3457d3493ac"}
Sep 19 04:15:11.727316 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:11.726977    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb4d9c6bc-5lrpf" event={"ID":"c9741183-e1ad-425d-b44c-26aa964acd73","Type":"ContainerStarted","Data":"ff28c1d9a0e6ef09513cc14c13085b0d09bc0c181521d3926fb25447e37b8067"}
Sep 19 04:15:11.727316 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:11.727000    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb4d9c6bc-5lrpf" event={"ID":"c9741183-e1ad-425d-b44c-26aa964acd73","Type":"ContainerStarted","Data":"c4b7193db38174d1ffd5b069b2b0b64f734d448c184aacff29a009b3ebd232ed"}
Sep 19 04:15:11.751077 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:11.751028    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb4d9c6bc-5lrpf" podStartSLOduration=1.90857717 podStartE2EDuration="5.75101382s" totalImagesPullingTime="3.84243665s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 04:15:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:15:11.749711091 +0000 UTC m=+90.932672234" watchObservedRunningTime="2026-09-19 04:15:11.75101382 +0000 UTC m=+90.933974968"
Sep 19 04:15:17.120218 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:17.120181    2579 patch_prober.go:28] interesting pod/image-registry-fbc69c679-f86tq container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 19 04:15:17.120597 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:17.120233    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-fbc69c679-f86tq" podUID="91ea3d76-ba23-42cc-80d5-c55cecf6e264" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 04:15:17.624359 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:17.624327    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-h52kg"]
Sep 19 04:15:17.627629 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:17.627611    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-h52kg"
Sep 19 04:15:17.633186 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:17.633155    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 19 04:15:17.633615 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:17.633578    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 19 04:15:17.639171 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:17.639148    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 19 04:15:17.639288 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:17.639191    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 19 04:15:17.639288 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:17.639271    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 19 04:15:17.639954 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:17.639936    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rp7q2\""
Sep 19 04:15:17.653779 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:17.653760    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 19 04:15:17.787204 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:17.787164    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/71273c3e-c393-4009-945f-f7f42c668d75-node-exporter-textfile\") pod \"node-exporter-h52kg\" (UID: \"71273c3e-c393-4009-945f-f7f42c668d75\") " pod="openshift-monitoring/node-exporter-h52kg"
Sep 19 04:15:17.787204 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:17.787201    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/71273c3e-c393-4009-945f-f7f42c668d75-sys\") pod \"node-exporter-h52kg\" (UID: \"71273c3e-c393-4009-945f-f7f42c668d75\") " pod="openshift-monitoring/node-exporter-h52kg"
Sep 19 04:15:17.787408 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:17.787223    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/71273c3e-c393-4009-945f-f7f42c668d75-node-exporter-wtmp\") pod \"node-exporter-h52kg\" (UID: \"71273c3e-c393-4009-945f-f7f42c668d75\") " pod="openshift-monitoring/node-exporter-h52kg"
Sep 19 04:15:17.787408 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:17.787285    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpvtb\" (UniqueName: \"kubernetes.io/projected/71273c3e-c393-4009-945f-f7f42c668d75-kube-api-access-mpvtb\") pod \"node-exporter-h52kg\" (UID: \"71273c3e-c393-4009-945f-f7f42c668d75\") " pod="openshift-monitoring/node-exporter-h52kg"
Sep 19 04:15:17.787408 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:17.787356    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/71273c3e-c393-4009-945f-f7f42c668d75-root\") pod \"node-exporter-h52kg\" (UID: \"71273c3e-c393-4009-945f-f7f42c668d75\") " pod="openshift-monitoring/node-exporter-h52kg"
Sep 19 04:15:17.787408 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:17.787400    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/71273c3e-c393-4009-945f-f7f42c668d75-node-exporter-tls\") pod \"node-exporter-h52kg\" (UID: \"71273c3e-c393-4009-945f-f7f42c668d75\") " pod="openshift-monitoring/node-exporter-h52kg"
Sep 19 04:15:17.787530 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:17.787461    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/71273c3e-c393-4009-945f-f7f42c668d75-node-exporter-accelerators-collector-config\") pod \"node-exporter-h52kg\" (UID: \"71273c3e-c393-4009-945f-f7f42c668d75\") " pod="openshift-monitoring/node-exporter-h52kg"
Sep 19 04:15:17.787530 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:17.787483    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/71273c3e-c393-4009-945f-f7f42c668d75-metrics-client-ca\") pod \"node-exporter-h52kg\" (UID: \"71273c3e-c393-4009-945f-f7f42c668d75\") " pod="openshift-monitoring/node-exporter-h52kg"
Sep 19 04:15:17.787530 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:17.787500    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/71273c3e-c393-4009-945f-f7f42c668d75-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-h52kg\" (UID: \"71273c3e-c393-4009-945f-f7f42c668d75\") " pod="openshift-monitoring/node-exporter-h52kg"
Sep 19 04:15:17.888129 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:17.888042    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/71273c3e-c393-4009-945f-f7f42c668d75-node-exporter-wtmp\") pod \"node-exporter-h52kg\" (UID: \"71273c3e-c393-4009-945f-f7f42c668d75\") " pod="openshift-monitoring/node-exporter-h52kg"
Sep 19 04:15:17.888129 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:17.888081    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpvtb\" (UniqueName: \"kubernetes.io/projected/71273c3e-c393-4009-945f-f7f42c668d75-kube-api-access-mpvtb\") pod \"node-exporter-h52kg\" (UID: \"71273c3e-c393-4009-945f-f7f42c668d75\") " pod="openshift-monitoring/node-exporter-h52kg"
Sep 19 04:15:17.888129 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:17.888113    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/71273c3e-c393-4009-945f-f7f42c668d75-root\") pod \"node-exporter-h52kg\" (UID: \"71273c3e-c393-4009-945f-f7f42c668d75\") " pod="openshift-monitoring/node-exporter-h52kg"
Sep 19 04:15:17.888405 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:17.888158    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/71273c3e-c393-4009-945f-f7f42c668d75-node-exporter-tls\") pod \"node-exporter-h52kg\" (UID: \"71273c3e-c393-4009-945f-f7f42c668d75\") " pod="openshift-monitoring/node-exporter-h52kg"
Sep 19 04:15:17.888405 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:17.888191    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/71273c3e-c393-4009-945f-f7f42c668d75-node-exporter-accelerators-collector-config\") pod \"node-exporter-h52kg\" (UID: \"71273c3e-c393-4009-945f-f7f42c668d75\") " pod="openshift-monitoring/node-exporter-h52kg"
Sep 19 04:15:17.888405 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:17.888222    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/71273c3e-c393-4009-945f-f7f42c668d75-metrics-client-ca\") pod \"node-exporter-h52kg\" (UID: \"71273c3e-c393-4009-945f-f7f42c668d75\") " pod="openshift-monitoring/node-exporter-h52kg"
Sep 19 04:15:17.888405 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:17.888244    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/71273c3e-c393-4009-945f-f7f42c668d75-node-exporter-wtmp\") pod \"node-exporter-h52kg\" (UID: \"71273c3e-c393-4009-945f-f7f42c668d75\") " pod="openshift-monitoring/node-exporter-h52kg"
Sep 19 04:15:17.888405 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:17.888261    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/71273c3e-c393-4009-945f-f7f42c668d75-root\") pod \"node-exporter-h52kg\" (UID: \"71273c3e-c393-4009-945f-f7f42c668d75\") " pod="openshift-monitoring/node-exporter-h52kg"
Sep 19 04:15:17.888405 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:17.888246    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/71273c3e-c393-4009-945f-f7f42c668d75-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-h52kg\" (UID: \"71273c3e-c393-4009-945f-f7f42c668d75\") " pod="openshift-monitoring/node-exporter-h52kg"
Sep 19 04:15:17.888405 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:17.888324    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/71273c3e-c393-4009-945f-f7f42c668d75-node-exporter-textfile\") pod \"node-exporter-h52kg\" (UID: \"71273c3e-c393-4009-945f-f7f42c668d75\") " pod="openshift-monitoring/node-exporter-h52kg"
Sep 19 04:15:17.888405 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:17.888351    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/71273c3e-c393-4009-945f-f7f42c668d75-sys\") pod \"node-exporter-h52kg\" (UID: \"71273c3e-c393-4009-945f-f7f42c668d75\") " pod="openshift-monitoring/node-exporter-h52kg"
Sep 19 04:15:17.888774 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:17.888418    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/71273c3e-c393-4009-945f-f7f42c668d75-sys\") pod \"node-exporter-h52kg\" (UID: \"71273c3e-c393-4009-945f-f7f42c668d75\") " pod="openshift-monitoring/node-exporter-h52kg"
Sep 19 04:15:17.888774 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:17.888722    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/71273c3e-c393-4009-945f-f7f42c668d75-node-exporter-textfile\") pod \"node-exporter-h52kg\" (UID: \"71273c3e-c393-4009-945f-f7f42c668d75\") " pod="openshift-monitoring/node-exporter-h52kg"
Sep 19 04:15:17.888948 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:17.888916    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/71273c3e-c393-4009-945f-f7f42c668d75-metrics-client-ca\") pod \"node-exporter-h52kg\" (UID: \"71273c3e-c393-4009-945f-f7f42c668d75\") " pod="openshift-monitoring/node-exporter-h52kg"
Sep 19 04:15:17.888948 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:17.888925    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/71273c3e-c393-4009-945f-f7f42c668d75-node-exporter-accelerators-collector-config\") pod \"node-exporter-h52kg\" (UID: \"71273c3e-c393-4009-945f-f7f42c668d75\") " pod="openshift-monitoring/node-exporter-h52kg"
Sep 19 04:15:17.890807 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:17.890785    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/71273c3e-c393-4009-945f-f7f42c668d75-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-h52kg\" (UID: \"71273c3e-c393-4009-945f-f7f42c668d75\") " pod="openshift-monitoring/node-exporter-h52kg"
Sep 19 04:15:17.890895 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:17.890833    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/71273c3e-c393-4009-945f-f7f42c668d75-node-exporter-tls\") pod \"node-exporter-h52kg\" (UID: \"71273c3e-c393-4009-945f-f7f42c668d75\") " pod="openshift-monitoring/node-exporter-h52kg"
Sep 19 04:15:17.927486 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:17.927459    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpvtb\" (UniqueName: \"kubernetes.io/projected/71273c3e-c393-4009-945f-f7f42c668d75-kube-api-access-mpvtb\") pod \"node-exporter-h52kg\" (UID: \"71273c3e-c393-4009-945f-f7f42c668d75\") " pod="openshift-monitoring/node-exporter-h52kg"
Sep 19 04:15:17.936484 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:17.936463    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-h52kg"
Sep 19 04:15:17.947752 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:15:17.947728    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71273c3e_c393_4009_945f_f7f42c668d75.slice/crio-a754f519de6da3dea4743d3f8b927fcfdbcfaad7d7710d3eb44fa3ab09f7ee64 WatchSource:0}: Error finding container a754f519de6da3dea4743d3f8b927fcfdbcfaad7d7710d3eb44fa3ab09f7ee64: Status 404 returned error can't find the container with id a754f519de6da3dea4743d3f8b927fcfdbcfaad7d7710d3eb44fa3ab09f7ee64
Sep 19 04:15:18.747669 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:18.747630    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-h52kg" event={"ID":"71273c3e-c393-4009-945f-f7f42c668d75","Type":"ContainerStarted","Data":"a754f519de6da3dea4743d3f8b927fcfdbcfaad7d7710d3eb44fa3ab09f7ee64"}
Sep 19 04:15:19.703521 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:19.703484    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7a6cf6e3-7960-4902-9c40-375a4de502ac-metrics-tls\") pod \"dns-default-n8jb9\" (UID: \"7a6cf6e3-7960-4902-9c40-375a4de502ac\") " pod="openshift-dns/dns-default-n8jb9"
Sep 19 04:15:19.705890 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:19.705869    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7a6cf6e3-7960-4902-9c40-375a4de502ac-metrics-tls\") pod \"dns-default-n8jb9\" (UID: \"7a6cf6e3-7960-4902-9c40-375a4de502ac\") " pod="openshift-dns/dns-default-n8jb9"
Sep 19 04:15:19.753151 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:19.753115    2579 generic.go:356] "Generic (PLEG): container finished" podID="71273c3e-c393-4009-945f-f7f42c668d75" containerID="da1d767742b2b3e84b224712b2d36dbcd0a151791d99c9d9c75f8698e2e90e24" exitCode=0
Sep 19 04:15:19.753490 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:19.753182    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-h52kg" event={"ID":"71273c3e-c393-4009-945f-f7f42c668d75","Type":"ContainerDied","Data":"da1d767742b2b3e84b224712b2d36dbcd0a151791d99c9d9c75f8698e2e90e24"}
Sep 19 04:15:19.947775 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:19.947726    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5s7cn\""
Sep 19 04:15:19.952870 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:19.952853    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-n8jb9"
Sep 19 04:15:20.286159 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:20.286081    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-n8jb9"]
Sep 19 04:15:20.290136 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:15:20.290104    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a6cf6e3_7960_4902_9c40_375a4de502ac.slice/crio-6085e73861cc50e6b72fc43b9f03f389a080eeecac239c5ceba1b6a4b92bb58a WatchSource:0}: Error finding container 6085e73861cc50e6b72fc43b9f03f389a080eeecac239c5ceba1b6a4b92bb58a: Status 404 returned error can't find the container with id 6085e73861cc50e6b72fc43b9f03f389a080eeecac239c5ceba1b6a4b92bb58a
Sep 19 04:15:20.759362 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:20.759319    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-n8jb9" event={"ID":"7a6cf6e3-7960-4902-9c40-375a4de502ac","Type":"ContainerStarted","Data":"6085e73861cc50e6b72fc43b9f03f389a080eeecac239c5ceba1b6a4b92bb58a"}
Sep 19 04:15:20.762009 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:20.761973    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-h52kg" event={"ID":"71273c3e-c393-4009-945f-f7f42c668d75","Type":"ContainerStarted","Data":"f537db166a06f5fed06a33e520eced3558da2dda0af2d2117475d34884074db1"}
Sep 19 04:15:20.762145 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:20.762033    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-h52kg" event={"ID":"71273c3e-c393-4009-945f-f7f42c668d75","Type":"ContainerStarted","Data":"568276b7e2ee834eec9ddc0486b718569736afbbce4db4d4272572ee1126c773"}
Sep 19 04:15:22.771067 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:22.770985    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-n8jb9" event={"ID":"7a6cf6e3-7960-4902-9c40-375a4de502ac","Type":"ContainerStarted","Data":"437045880d546c1c726ab9566c724e8a1f4ceff9e40cd4344612d4b0c32cbb5d"}
Sep 19 04:15:22.771067 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:22.771034    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-n8jb9" event={"ID":"7a6cf6e3-7960-4902-9c40-375a4de502ac","Type":"ContainerStarted","Data":"c570ce86185632158475bf7d948279ff46a67d9809dbd42e78c4a2fcaa3db713"}
Sep 19 04:15:22.771471 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:22.771106    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-n8jb9"
Sep 19 04:15:22.829454 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:22.829401    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-h52kg" podStartSLOduration=4.80065238 podStartE2EDuration="5.829386275s" totalImagesPullingTime="960.438064ms" totalInitContainerRuntime="68.295831ms" isStatefulPod=true podCreationTimestamp="2026-09-19 04:15:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:15:20.795199658 +0000 UTC m=+99.978160806" watchObservedRunningTime="2026-09-19 04:15:22.829386275 +0000 UTC m=+102.012347445"
Sep 19 04:15:22.829826 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:22.829804    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-n8jb9" podStartSLOduration=65.88815548 podStartE2EDuration="1m7.829797128s" totalImagesPullingTime="1.941641648s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 04:14:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:15:22.829139244 +0000 UTC m=+102.012100393" watchObservedRunningTime="2026-09-19 04:15:22.829797128 +0000 UTC m=+102.012758276"
Sep 19 04:15:23.656163 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:23.656137    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-z7dgk"
Sep 19 04:15:27.118994 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:27.118967    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-fbc69c679-f86tq"
Sep 19 04:15:32.133505 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:32.133458    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-fbc69c679-f86tq" podUID="91ea3d76-ba23-42cc-80d5-c55cecf6e264" containerName="registry" containerID="cri-o://d808a368b62b84c99c99b2a1b6b4f7b4394669daf3e028ddf870afe6dc3d4b54" gracePeriod=30
Sep 19 04:15:32.367618 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:32.367594    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-fbc69c679-f86tq"
Sep 19 04:15:32.508548 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:32.508520    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/91ea3d76-ba23-42cc-80d5-c55cecf6e264-registry-certificates\") pod \"91ea3d76-ba23-42cc-80d5-c55cecf6e264\" (UID: \"91ea3d76-ba23-42cc-80d5-c55cecf6e264\") "
Sep 19 04:15:32.508778 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:32.508555    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/91ea3d76-ba23-42cc-80d5-c55cecf6e264-trusted-ca\") pod \"91ea3d76-ba23-42cc-80d5-c55cecf6e264\" (UID: \"91ea3d76-ba23-42cc-80d5-c55cecf6e264\") "
Sep 19 04:15:32.508778 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:32.508654    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/91ea3d76-ba23-42cc-80d5-c55cecf6e264-registry-tls\") pod \"91ea3d76-ba23-42cc-80d5-c55cecf6e264\" (UID: \"91ea3d76-ba23-42cc-80d5-c55cecf6e264\") "
Sep 19 04:15:32.508778 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:32.508688    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/91ea3d76-ba23-42cc-80d5-c55cecf6e264-installation-pull-secrets\") pod \"91ea3d76-ba23-42cc-80d5-c55cecf6e264\" (UID: \"91ea3d76-ba23-42cc-80d5-c55cecf6e264\") "
Sep 19 04:15:32.508778 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:32.508727    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/91ea3d76-ba23-42cc-80d5-c55cecf6e264-ca-trust-extracted\") pod \"91ea3d76-ba23-42cc-80d5-c55cecf6e264\" (UID: \"91ea3d76-ba23-42cc-80d5-c55cecf6e264\") "
Sep 19 04:15:32.508778 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:32.508759    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/91ea3d76-ba23-42cc-80d5-c55cecf6e264-bound-sa-token\") pod \"91ea3d76-ba23-42cc-80d5-c55cecf6e264\" (UID: \"91ea3d76-ba23-42cc-80d5-c55cecf6e264\") "
Sep 19 04:15:32.508778 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:32.508777    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nqlqk\" (UniqueName: \"kubernetes.io/projected/91ea3d76-ba23-42cc-80d5-c55cecf6e264-kube-api-access-nqlqk\") pod \"91ea3d76-ba23-42cc-80d5-c55cecf6e264\" (UID: \"91ea3d76-ba23-42cc-80d5-c55cecf6e264\") "
Sep 19 04:15:32.509083 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:32.508809    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/91ea3d76-ba23-42cc-80d5-c55cecf6e264-image-registry-private-configuration\") pod \"91ea3d76-ba23-42cc-80d5-c55cecf6e264\" (UID: \"91ea3d76-ba23-42cc-80d5-c55cecf6e264\") "
Sep 19 04:15:32.509083 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:32.508991    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91ea3d76-ba23-42cc-80d5-c55cecf6e264-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "91ea3d76-ba23-42cc-80d5-c55cecf6e264" (UID: "91ea3d76-ba23-42cc-80d5-c55cecf6e264"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 04:15:32.509083 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:32.509007    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91ea3d76-ba23-42cc-80d5-c55cecf6e264-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "91ea3d76-ba23-42cc-80d5-c55cecf6e264" (UID: "91ea3d76-ba23-42cc-80d5-c55cecf6e264"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 04:15:32.509536 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:32.509396    2579 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/91ea3d76-ba23-42cc-80d5-c55cecf6e264-registry-certificates\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:15:32.509536 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:32.509569    2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/91ea3d76-ba23-42cc-80d5-c55cecf6e264-trusted-ca\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:15:32.511358 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:32.511312    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91ea3d76-ba23-42cc-80d5-c55cecf6e264-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "91ea3d76-ba23-42cc-80d5-c55cecf6e264" (UID: "91ea3d76-ba23-42cc-80d5-c55cecf6e264"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 04:15:32.511358 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:32.511333    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91ea3d76-ba23-42cc-80d5-c55cecf6e264-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "91ea3d76-ba23-42cc-80d5-c55cecf6e264" (UID: "91ea3d76-ba23-42cc-80d5-c55cecf6e264"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 04:15:32.511606 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:32.511576    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91ea3d76-ba23-42cc-80d5-c55cecf6e264-kube-api-access-nqlqk" (OuterVolumeSpecName: "kube-api-access-nqlqk") pod "91ea3d76-ba23-42cc-80d5-c55cecf6e264" (UID: "91ea3d76-ba23-42cc-80d5-c55cecf6e264"). InnerVolumeSpecName "kube-api-access-nqlqk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 04:15:32.511718 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:32.511676    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91ea3d76-ba23-42cc-80d5-c55cecf6e264-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "91ea3d76-ba23-42cc-80d5-c55cecf6e264" (UID: "91ea3d76-ba23-42cc-80d5-c55cecf6e264"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 04:15:32.511774 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:32.511732    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91ea3d76-ba23-42cc-80d5-c55cecf6e264-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "91ea3d76-ba23-42cc-80d5-c55cecf6e264" (UID: "91ea3d76-ba23-42cc-80d5-c55cecf6e264"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 04:15:32.517299 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:32.517276    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91ea3d76-ba23-42cc-80d5-c55cecf6e264-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "91ea3d76-ba23-42cc-80d5-c55cecf6e264" (UID: "91ea3d76-ba23-42cc-80d5-c55cecf6e264"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:15:32.610197 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:32.610145    2579 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/91ea3d76-ba23-42cc-80d5-c55cecf6e264-registry-tls\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:15:32.610197 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:32.610189    2579 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/91ea3d76-ba23-42cc-80d5-c55cecf6e264-installation-pull-secrets\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:15:32.610197 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:32.610199    2579 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/91ea3d76-ba23-42cc-80d5-c55cecf6e264-ca-trust-extracted\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:15:32.610197 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:32.610208    2579 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/91ea3d76-ba23-42cc-80d5-c55cecf6e264-bound-sa-token\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:15:32.610197 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:32.610217    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqlqk\" (UniqueName: \"kubernetes.io/projected/91ea3d76-ba23-42cc-80d5-c55cecf6e264-kube-api-access-nqlqk\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:15:32.610470 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:32.610226    2579 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/91ea3d76-ba23-42cc-80d5-c55cecf6e264-image-registry-private-configuration\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:15:32.776983 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:32.776916    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-n8jb9"
Sep 19 04:15:32.803986 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:32.803953    2579 generic.go:356] "Generic (PLEG): container finished" podID="91ea3d76-ba23-42cc-80d5-c55cecf6e264" containerID="d808a368b62b84c99c99b2a1b6b4f7b4394669daf3e028ddf870afe6dc3d4b54" exitCode=0
Sep 19 04:15:32.804125 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:32.803994    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-fbc69c679-f86tq"
Sep 19 04:15:32.804125 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:32.804025    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-fbc69c679-f86tq" event={"ID":"91ea3d76-ba23-42cc-80d5-c55cecf6e264","Type":"ContainerDied","Data":"d808a368b62b84c99c99b2a1b6b4f7b4394669daf3e028ddf870afe6dc3d4b54"}
Sep 19 04:15:32.804125 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:32.804067    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-fbc69c679-f86tq" event={"ID":"91ea3d76-ba23-42cc-80d5-c55cecf6e264","Type":"ContainerDied","Data":"d11fd4a6dbf04d77ae4316b1499443fdc120d567637304cbdcb47f74d1978e0c"}
Sep 19 04:15:32.804125 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:32.804088    2579 scope.go:117] "RemoveContainer" containerID="d808a368b62b84c99c99b2a1b6b4f7b4394669daf3e028ddf870afe6dc3d4b54"
Sep 19 04:15:32.813395 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:32.813381    2579 scope.go:117] "RemoveContainer" containerID="d808a368b62b84c99c99b2a1b6b4f7b4394669daf3e028ddf870afe6dc3d4b54"
Sep 19 04:15:32.813642 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:15:32.813623    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d808a368b62b84c99c99b2a1b6b4f7b4394669daf3e028ddf870afe6dc3d4b54\": container with ID starting with d808a368b62b84c99c99b2a1b6b4f7b4394669daf3e028ddf870afe6dc3d4b54 not found: ID does not exist" containerID="d808a368b62b84c99c99b2a1b6b4f7b4394669daf3e028ddf870afe6dc3d4b54"
Sep 19 04:15:32.813687 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:32.813651    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d808a368b62b84c99c99b2a1b6b4f7b4394669daf3e028ddf870afe6dc3d4b54"} err="failed to get container status \"d808a368b62b84c99c99b2a1b6b4f7b4394669daf3e028ddf870afe6dc3d4b54\": rpc error: code = NotFound desc = could not find container \"d808a368b62b84c99c99b2a1b6b4f7b4394669daf3e028ddf870afe6dc3d4b54\": container with ID starting with d808a368b62b84c99c99b2a1b6b4f7b4394669daf3e028ddf870afe6dc3d4b54 not found: ID does not exist"
Sep 19 04:15:32.865687 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:32.865663    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-fbc69c679-f86tq"]
Sep 19 04:15:32.871622 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:32.871599    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-fbc69c679-f86tq"]
Sep 19 04:15:33.350039 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:33.350004    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91ea3d76-ba23-42cc-80d5-c55cecf6e264" path="/var/lib/kubelet/pods/91ea3d76-ba23-42cc-80d5-c55cecf6e264/volumes"
Sep 19 04:15:34.199408 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:34.199380    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6594f4c75-rhcwm"]
Sep 19 04:15:34.199621 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:34.199609    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91ea3d76-ba23-42cc-80d5-c55cecf6e264" containerName="registry"
Sep 19 04:15:34.199670 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:34.199624    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="91ea3d76-ba23-42cc-80d5-c55cecf6e264" containerName="registry"
Sep 19 04:15:34.199723 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:34.199670    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="91ea3d76-ba23-42cc-80d5-c55cecf6e264" containerName="registry"
Sep 19 04:15:34.206141 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:34.206124    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-rhcwm"
Sep 19 04:15:34.210171 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:34.210147    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-6k8tn\""
Sep 19 04:15:34.210295 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:34.210147    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 19 04:15:34.210685 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:34.210667    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 19 04:15:34.235543 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:34.235522    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-rhcwm"]
Sep 19 04:15:34.322867 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:34.322838    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4kn6\" (UniqueName: \"kubernetes.io/projected/c0cd206c-8a93-45d6-80d6-edfc51d23f45-kube-api-access-v4kn6\") pod \"downloads-6594f4c75-rhcwm\" (UID: \"c0cd206c-8a93-45d6-80d6-edfc51d23f45\") " pod="openshift-console/downloads-6594f4c75-rhcwm"
Sep 19 04:15:34.423902 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:34.423870    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4kn6\" (UniqueName: \"kubernetes.io/projected/c0cd206c-8a93-45d6-80d6-edfc51d23f45-kube-api-access-v4kn6\") pod \"downloads-6594f4c75-rhcwm\" (UID: \"c0cd206c-8a93-45d6-80d6-edfc51d23f45\") " pod="openshift-console/downloads-6594f4c75-rhcwm"
Sep 19 04:15:34.435089 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:34.435068    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4kn6\" (UniqueName: \"kubernetes.io/projected/c0cd206c-8a93-45d6-80d6-edfc51d23f45-kube-api-access-v4kn6\") pod \"downloads-6594f4c75-rhcwm\" (UID: \"c0cd206c-8a93-45d6-80d6-edfc51d23f45\") " pod="openshift-console/downloads-6594f4c75-rhcwm"
Sep 19 04:15:34.514658 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:34.514557    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-rhcwm"
Sep 19 04:15:34.794733 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:34.794706    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-rhcwm"]
Sep 19 04:15:34.795620 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:15:34.795590    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0cd206c_8a93_45d6_80d6_edfc51d23f45.slice/crio-a3f7b17aec11ee0664ea67c6bfe50a62baeb3e80a6cde4e34def9abbcb16df0c WatchSource:0}: Error finding container a3f7b17aec11ee0664ea67c6bfe50a62baeb3e80a6cde4e34def9abbcb16df0c: Status 404 returned error can't find the container with id a3f7b17aec11ee0664ea67c6bfe50a62baeb3e80a6cde4e34def9abbcb16df0c
Sep 19 04:15:34.811545 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:34.811523    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-rhcwm" event={"ID":"c0cd206c-8a93-45d6-80d6-edfc51d23f45","Type":"ContainerStarted","Data":"a3f7b17aec11ee0664ea67c6bfe50a62baeb3e80a6cde4e34def9abbcb16df0c"}
Sep 19 04:15:40.761918 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:40.761884    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5b49689ccb-n92tv"]
Sep 19 04:15:40.770531 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:40.770506    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b49689ccb-n92tv"
Sep 19 04:15:40.774575 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:40.774542    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 19 04:15:40.774575 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:40.774540    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 19 04:15:40.775309 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:40.775289    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 19 04:15:40.775566 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:40.775516    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-qqtdv\""
Sep 19 04:15:40.775566 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:40.775563    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 19 04:15:40.775734 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:40.775715    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 19 04:15:40.783933 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:40.783910    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b49689ccb-n92tv"]
Sep 19 04:15:40.870983 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:40.870946    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/86c58995-ea08-4c7a-99a2-70b4dc820115-console-oauth-config\") pod \"console-5b49689ccb-n92tv\" (UID: \"86c58995-ea08-4c7a-99a2-70b4dc820115\") " pod="openshift-console/console-5b49689ccb-n92tv"
Sep 19 04:15:40.871181 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:40.870992    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgp6z\" (UniqueName: \"kubernetes.io/projected/86c58995-ea08-4c7a-99a2-70b4dc820115-kube-api-access-qgp6z\") pod \"console-5b49689ccb-n92tv\" (UID: \"86c58995-ea08-4c7a-99a2-70b4dc820115\") " pod="openshift-console/console-5b49689ccb-n92tv"
Sep 19 04:15:40.871181 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:40.871030    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/86c58995-ea08-4c7a-99a2-70b4dc820115-console-config\") pod \"console-5b49689ccb-n92tv\" (UID: \"86c58995-ea08-4c7a-99a2-70b4dc820115\") " pod="openshift-console/console-5b49689ccb-n92tv"
Sep 19 04:15:40.871181 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:40.871054    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/86c58995-ea08-4c7a-99a2-70b4dc820115-oauth-serving-cert\") pod \"console-5b49689ccb-n92tv\" (UID: \"86c58995-ea08-4c7a-99a2-70b4dc820115\") " pod="openshift-console/console-5b49689ccb-n92tv"
Sep 19 04:15:40.871181 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:40.871169    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/86c58995-ea08-4c7a-99a2-70b4dc820115-service-ca\") pod \"console-5b49689ccb-n92tv\" (UID: \"86c58995-ea08-4c7a-99a2-70b4dc820115\") " pod="openshift-console/console-5b49689ccb-n92tv"
Sep 19 04:15:40.871408 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:40.871205    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/86c58995-ea08-4c7a-99a2-70b4dc820115-console-serving-cert\") pod \"console-5b49689ccb-n92tv\" (UID: \"86c58995-ea08-4c7a-99a2-70b4dc820115\") " pod="openshift-console/console-5b49689ccb-n92tv"
Sep 19 04:15:40.971972 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:40.971937    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/86c58995-ea08-4c7a-99a2-70b4dc820115-service-ca\") pod \"console-5b49689ccb-n92tv\" (UID: \"86c58995-ea08-4c7a-99a2-70b4dc820115\") " pod="openshift-console/console-5b49689ccb-n92tv"
Sep 19 04:15:40.971972 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:40.971974    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/86c58995-ea08-4c7a-99a2-70b4dc820115-console-serving-cert\") pod \"console-5b49689ccb-n92tv\" (UID: \"86c58995-ea08-4c7a-99a2-70b4dc820115\") " pod="openshift-console/console-5b49689ccb-n92tv"
Sep 19 04:15:40.972221 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:40.972038    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/86c58995-ea08-4c7a-99a2-70b4dc820115-console-oauth-config\") pod \"console-5b49689ccb-n92tv\" (UID: \"86c58995-ea08-4c7a-99a2-70b4dc820115\") " pod="openshift-console/console-5b49689ccb-n92tv"
Sep 19 04:15:40.972221 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:40.972059    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgp6z\" (UniqueName: \"kubernetes.io/projected/86c58995-ea08-4c7a-99a2-70b4dc820115-kube-api-access-qgp6z\") pod \"console-5b49689ccb-n92tv\" (UID: \"86c58995-ea08-4c7a-99a2-70b4dc820115\") " pod="openshift-console/console-5b49689ccb-n92tv"
Sep 19 04:15:40.972221 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:40.972094    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/86c58995-ea08-4c7a-99a2-70b4dc820115-console-config\") pod \"console-5b49689ccb-n92tv\" (UID: \"86c58995-ea08-4c7a-99a2-70b4dc820115\") " pod="openshift-console/console-5b49689ccb-n92tv"
Sep 19 04:15:40.972221 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:40.972119    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/86c58995-ea08-4c7a-99a2-70b4dc820115-oauth-serving-cert\") pod \"console-5b49689ccb-n92tv\" (UID: \"86c58995-ea08-4c7a-99a2-70b4dc820115\") " pod="openshift-console/console-5b49689ccb-n92tv"
Sep 19 04:15:40.972891 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:40.972825    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/86c58995-ea08-4c7a-99a2-70b4dc820115-service-ca\") pod \"console-5b49689ccb-n92tv\" (UID: \"86c58995-ea08-4c7a-99a2-70b4dc820115\") " pod="openshift-console/console-5b49689ccb-n92tv"
Sep 19 04:15:40.973032 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:40.973009    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/86c58995-ea08-4c7a-99a2-70b4dc820115-console-config\") pod \"console-5b49689ccb-n92tv\" (UID: \"86c58995-ea08-4c7a-99a2-70b4dc820115\") " pod="openshift-console/console-5b49689ccb-n92tv"
Sep 19 04:15:40.973095 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:40.973037    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/86c58995-ea08-4c7a-99a2-70b4dc820115-oauth-serving-cert\") pod \"console-5b49689ccb-n92tv\" (UID: \"86c58995-ea08-4c7a-99a2-70b4dc820115\") " pod="openshift-console/console-5b49689ccb-n92tv"
Sep 19 04:15:40.975257 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:40.975235    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/86c58995-ea08-4c7a-99a2-70b4dc820115-console-oauth-config\") pod \"console-5b49689ccb-n92tv\" (UID: \"86c58995-ea08-4c7a-99a2-70b4dc820115\") " pod="openshift-console/console-5b49689ccb-n92tv"
Sep 19 04:15:40.975257 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:40.975254    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/86c58995-ea08-4c7a-99a2-70b4dc820115-console-serving-cert\") pod \"console-5b49689ccb-n92tv\" (UID: \"86c58995-ea08-4c7a-99a2-70b4dc820115\") " pod="openshift-console/console-5b49689ccb-n92tv"
Sep 19 04:15:40.986652 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:40.986623    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgp6z\" (UniqueName: \"kubernetes.io/projected/86c58995-ea08-4c7a-99a2-70b4dc820115-kube-api-access-qgp6z\") pod \"console-5b49689ccb-n92tv\" (UID: \"86c58995-ea08-4c7a-99a2-70b4dc820115\") " pod="openshift-console/console-5b49689ccb-n92tv"
Sep 19 04:15:41.082120 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:41.082039    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b49689ccb-n92tv"
Sep 19 04:15:41.373092 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:41.373008    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b49689ccb-n92tv"]
Sep 19 04:15:41.376408 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:15:41.376382    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86c58995_ea08_4c7a_99a2_70b4dc820115.slice/crio-e562fdc818cfafda7de9c4da612252038b1279475636e5f9b0212ea2335a6ccb WatchSource:0}: Error finding container e562fdc818cfafda7de9c4da612252038b1279475636e5f9b0212ea2335a6ccb: Status 404 returned error can't find the container with id e562fdc818cfafda7de9c4da612252038b1279475636e5f9b0212ea2335a6ccb
Sep 19 04:15:41.839620 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:41.839579    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b49689ccb-n92tv" event={"ID":"86c58995-ea08-4c7a-99a2-70b4dc820115","Type":"ContainerStarted","Data":"e562fdc818cfafda7de9c4da612252038b1279475636e5f9b0212ea2335a6ccb"}
Sep 19 04:15:43.081247 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:43.080118    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6b6bfffd4-z7rp5"]
Sep 19 04:15:43.086995 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:43.086961    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b6bfffd4-z7rp5"
Sep 19 04:15:43.100115 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:43.100083    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 19 04:15:43.103391 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:43.103362    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b6bfffd4-z7rp5"]
Sep 19 04:15:43.191951 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:43.191723    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a3f2cede-329e-4601-9e0a-65228d46bd47-oauth-serving-cert\") pod \"console-6b6bfffd4-z7rp5\" (UID: \"a3f2cede-329e-4601-9e0a-65228d46bd47\") " pod="openshift-console/console-6b6bfffd4-z7rp5"
Sep 19 04:15:43.191951 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:43.191784    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a3f2cede-329e-4601-9e0a-65228d46bd47-console-serving-cert\") pod \"console-6b6bfffd4-z7rp5\" (UID: \"a3f2cede-329e-4601-9e0a-65228d46bd47\") " pod="openshift-console/console-6b6bfffd4-z7rp5"
Sep 19 04:15:43.191951 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:43.191821    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdbjg\" (UniqueName: \"kubernetes.io/projected/a3f2cede-329e-4601-9e0a-65228d46bd47-kube-api-access-tdbjg\") pod \"console-6b6bfffd4-z7rp5\" (UID: \"a3f2cede-329e-4601-9e0a-65228d46bd47\") " pod="openshift-console/console-6b6bfffd4-z7rp5"
Sep 19 04:15:43.191951 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:43.191891    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a3f2cede-329e-4601-9e0a-65228d46bd47-console-config\") pod \"console-6b6bfffd4-z7rp5\" (UID: \"a3f2cede-329e-4601-9e0a-65228d46bd47\") " pod="openshift-console/console-6b6bfffd4-z7rp5"
Sep 19 04:15:43.191951 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:43.191913    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a3f2cede-329e-4601-9e0a-65228d46bd47-service-ca\") pod \"console-6b6bfffd4-z7rp5\" (UID: \"a3f2cede-329e-4601-9e0a-65228d46bd47\") " pod="openshift-console/console-6b6bfffd4-z7rp5"
Sep 19 04:15:43.192295 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:43.191983    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3f2cede-329e-4601-9e0a-65228d46bd47-trusted-ca-bundle\") pod \"console-6b6bfffd4-z7rp5\" (UID: \"a3f2cede-329e-4601-9e0a-65228d46bd47\") " pod="openshift-console/console-6b6bfffd4-z7rp5"
Sep 19 04:15:43.192295 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:43.192008    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a3f2cede-329e-4601-9e0a-65228d46bd47-console-oauth-config\") pod \"console-6b6bfffd4-z7rp5\" (UID: \"a3f2cede-329e-4601-9e0a-65228d46bd47\") " pod="openshift-console/console-6b6bfffd4-z7rp5"
Sep 19 04:15:43.293151 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:43.293115    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3f2cede-329e-4601-9e0a-65228d46bd47-trusted-ca-bundle\") pod \"console-6b6bfffd4-z7rp5\" (UID: \"a3f2cede-329e-4601-9e0a-65228d46bd47\") " pod="openshift-console/console-6b6bfffd4-z7rp5"
Sep 19 04:15:43.293337 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:43.293162    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a3f2cede-329e-4601-9e0a-65228d46bd47-console-oauth-config\") pod \"console-6b6bfffd4-z7rp5\" (UID: \"a3f2cede-329e-4601-9e0a-65228d46bd47\") " pod="openshift-console/console-6b6bfffd4-z7rp5"
Sep 19 04:15:43.293337 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:43.293194    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a3f2cede-329e-4601-9e0a-65228d46bd47-oauth-serving-cert\") pod \"console-6b6bfffd4-z7rp5\" (UID: \"a3f2cede-329e-4601-9e0a-65228d46bd47\") " pod="openshift-console/console-6b6bfffd4-z7rp5"
Sep 19 04:15:43.293337 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:43.293217    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a3f2cede-329e-4601-9e0a-65228d46bd47-console-serving-cert\") pod \"console-6b6bfffd4-z7rp5\" (UID: \"a3f2cede-329e-4601-9e0a-65228d46bd47\") " pod="openshift-console/console-6b6bfffd4-z7rp5"
Sep 19 04:15:43.293337 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:43.293240    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdbjg\" (UniqueName: \"kubernetes.io/projected/a3f2cede-329e-4601-9e0a-65228d46bd47-kube-api-access-tdbjg\") pod \"console-6b6bfffd4-z7rp5\" (UID: \"a3f2cede-329e-4601-9e0a-65228d46bd47\") " pod="openshift-console/console-6b6bfffd4-z7rp5"
Sep 19 04:15:43.293337 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:43.293291    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a3f2cede-329e-4601-9e0a-65228d46bd47-console-config\") pod \"console-6b6bfffd4-z7rp5\" (UID: \"a3f2cede-329e-4601-9e0a-65228d46bd47\") " pod="openshift-console/console-6b6bfffd4-z7rp5"
Sep 19 04:15:43.293337 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:43.293319    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a3f2cede-329e-4601-9e0a-65228d46bd47-service-ca\") pod \"console-6b6bfffd4-z7rp5\" (UID: \"a3f2cede-329e-4601-9e0a-65228d46bd47\") " pod="openshift-console/console-6b6bfffd4-z7rp5"
Sep 19 04:15:43.294469 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:43.294060    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a3f2cede-329e-4601-9e0a-65228d46bd47-service-ca\") pod \"console-6b6bfffd4-z7rp5\" (UID: \"a3f2cede-329e-4601-9e0a-65228d46bd47\") " pod="openshift-console/console-6b6bfffd4-z7rp5"
Sep 19 04:15:43.294469 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:43.294186    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a3f2cede-329e-4601-9e0a-65228d46bd47-oauth-serving-cert\") pod \"console-6b6bfffd4-z7rp5\" (UID: \"a3f2cede-329e-4601-9e0a-65228d46bd47\") " pod="openshift-console/console-6b6bfffd4-z7rp5"
Sep 19 04:15:43.294905 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:43.294863    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3f2cede-329e-4601-9e0a-65228d46bd47-trusted-ca-bundle\") pod \"console-6b6bfffd4-z7rp5\" (UID: \"a3f2cede-329e-4601-9e0a-65228d46bd47\") " pod="openshift-console/console-6b6bfffd4-z7rp5"
Sep 19 04:15:43.295350 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:43.295324    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a3f2cede-329e-4601-9e0a-65228d46bd47-console-config\") pod \"console-6b6bfffd4-z7rp5\" (UID: \"a3f2cede-329e-4601-9e0a-65228d46bd47\") " pod="openshift-console/console-6b6bfffd4-z7rp5"
Sep 19 04:15:43.297794 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:43.297759    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a3f2cede-329e-4601-9e0a-65228d46bd47-console-serving-cert\") pod \"console-6b6bfffd4-z7rp5\" (UID: \"a3f2cede-329e-4601-9e0a-65228d46bd47\") " pod="openshift-console/console-6b6bfffd4-z7rp5"
Sep 19 04:15:43.299874 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:43.299849    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a3f2cede-329e-4601-9e0a-65228d46bd47-console-oauth-config\") pod \"console-6b6bfffd4-z7rp5\" (UID: \"a3f2cede-329e-4601-9e0a-65228d46bd47\") " pod="openshift-console/console-6b6bfffd4-z7rp5"
Sep 19 04:15:43.306655 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:43.306608    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdbjg\" (UniqueName: \"kubernetes.io/projected/a3f2cede-329e-4601-9e0a-65228d46bd47-kube-api-access-tdbjg\") pod \"console-6b6bfffd4-z7rp5\" (UID: \"a3f2cede-329e-4601-9e0a-65228d46bd47\") " pod="openshift-console/console-6b6bfffd4-z7rp5"
Sep 19 04:15:43.400734 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:43.400630    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b6bfffd4-z7rp5"
Sep 19 04:15:46.222373 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:46.222298    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b6bfffd4-z7rp5"]
Sep 19 04:15:51.059260 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:51.059217    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3738f1c7-67af-4201-8e8d-1d0406e0edb9-metrics-certs\") pod \"network-metrics-daemon-wmk74\" (UID: \"3738f1c7-67af-4201-8e8d-1d0406e0edb9\") " pod="openshift-multus/network-metrics-daemon-wmk74"
Sep 19 04:15:51.061905 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:51.061879    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3738f1c7-67af-4201-8e8d-1d0406e0edb9-metrics-certs\") pod \"network-metrics-daemon-wmk74\" (UID: \"3738f1c7-67af-4201-8e8d-1d0406e0edb9\") " pod="openshift-multus/network-metrics-daemon-wmk74"
Sep 19 04:15:51.260288 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:51.260254    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zg2ns\""
Sep 19 04:15:51.266973 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:51.266945    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wmk74"
Sep 19 04:15:53.263415 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:15:53.263382    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3f2cede_329e_4601_9e0a_65228d46bd47.slice/crio-1f4e830239952f46427b13dc45bf90612297741d9f667b06ffaedd228c749e30 WatchSource:0}: Error finding container 1f4e830239952f46427b13dc45bf90612297741d9f667b06ffaedd228c749e30: Status 404 returned error can't find the container with id 1f4e830239952f46427b13dc45bf90612297741d9f667b06ffaedd228c749e30
Sep 19 04:15:53.553024 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:53.552996    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wmk74"]
Sep 19 04:15:53.555306 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:15:53.555274    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3738f1c7_67af_4201_8e8d_1d0406e0edb9.slice/crio-e44ea5eae6cf98026d16b66d381263d5aa2127e70b1d2afc856d7c66ed6fbacd WatchSource:0}: Error finding container e44ea5eae6cf98026d16b66d381263d5aa2127e70b1d2afc856d7c66ed6fbacd: Status 404 returned error can't find the container with id e44ea5eae6cf98026d16b66d381263d5aa2127e70b1d2afc856d7c66ed6fbacd
Sep 19 04:15:53.887235 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:53.887145    2579 generic.go:356] "Generic (PLEG): container finished" podID="1633a682-5c86-4a59-b25a-2d049b6f3577" containerID="42d579a27d981fba5895f8773777bfc8f2cbe9859c82bb4efc6f76597c995660" exitCode=0
Sep 19 04:15:53.887400 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:53.887228    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-prdpb" event={"ID":"1633a682-5c86-4a59-b25a-2d049b6f3577","Type":"ContainerDied","Data":"42d579a27d981fba5895f8773777bfc8f2cbe9859c82bb4efc6f76597c995660"}
Sep 19 04:15:53.887722 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:53.887674    2579 scope.go:117] "RemoveContainer" containerID="42d579a27d981fba5895f8773777bfc8f2cbe9859c82bb4efc6f76597c995660"
Sep 19 04:15:53.889556 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:53.889253    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-rhcwm" event={"ID":"c0cd206c-8a93-45d6-80d6-edfc51d23f45","Type":"ContainerStarted","Data":"64d206a2f4ab207b5b3ad2707011853b046f67e9d61937b716481f2ae706a9ac"}
Sep 19 04:15:53.889556 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:53.889507    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6594f4c75-rhcwm"
Sep 19 04:15:53.890764 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:53.890707    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wmk74" event={"ID":"3738f1c7-67af-4201-8e8d-1d0406e0edb9","Type":"ContainerStarted","Data":"e44ea5eae6cf98026d16b66d381263d5aa2127e70b1d2afc856d7c66ed6fbacd"}
Sep 19 04:15:53.892558 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:53.892527    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b49689ccb-n92tv" event={"ID":"86c58995-ea08-4c7a-99a2-70b4dc820115","Type":"ContainerStarted","Data":"14a96d08c95dffba3903ac91f46b86753964edb28773e4c586c95e2cd25518ff"}
Sep 19 04:15:53.894224 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:53.894202    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b6bfffd4-z7rp5" event={"ID":"a3f2cede-329e-4601-9e0a-65228d46bd47","Type":"ContainerStarted","Data":"935c49f8c099d1edf353819b892292f3d8eb32eb8cbdf7bb9203bda9bade335f"}
Sep 19 04:15:53.894310 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:53.894242    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b6bfffd4-z7rp5" event={"ID":"a3f2cede-329e-4601-9e0a-65228d46bd47","Type":"ContainerStarted","Data":"1f4e830239952f46427b13dc45bf90612297741d9f667b06ffaedd228c749e30"}
Sep 19 04:15:53.904567 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:53.904543    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6594f4c75-rhcwm"
Sep 19 04:15:54.042989 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:54.042930    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-6594f4c75-rhcwm" podStartSLOduration=1.4662564439999999 podStartE2EDuration="20.042905668s" totalImagesPullingTime="18.576649224s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 04:15:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:15:54.040143463 +0000 UTC m=+133.223104624" watchObservedRunningTime="2026-09-19 04:15:54.042905668 +0000 UTC m=+133.225866819"
Sep 19 04:15:54.043170 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:54.043018    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6b6bfffd4-z7rp5" podStartSLOduration=11.043012508 podStartE2EDuration="11.043012508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 04:15:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:15:54.002099992 +0000 UTC m=+133.185061152" watchObservedRunningTime="2026-09-19 04:15:54.043012508 +0000 UTC m=+133.225973656"
Sep 19 04:15:54.105802 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:54.105517    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5b49689ccb-n92tv" podStartSLOduration=2.199622907 podStartE2EDuration="14.105489621s" totalImagesPullingTime="11.905866714s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 04:15:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:15:54.098590421 +0000 UTC m=+133.281551580" watchObservedRunningTime="2026-09-19 04:15:54.105489621 +0000 UTC m=+133.288450770"
Sep 19 04:15:54.903683 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:54.902724    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-prdpb" event={"ID":"1633a682-5c86-4a59-b25a-2d049b6f3577","Type":"ContainerStarted","Data":"2b64634b8192113eaef2f80d51747b8cc390285db2ae08e4e2169900bd36731e"}
Sep 19 04:15:55.913529 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:55.913479    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wmk74" event={"ID":"3738f1c7-67af-4201-8e8d-1d0406e0edb9","Type":"ContainerStarted","Data":"c81e45eac168d07982726ce4cece0835cebcf736abae523d87d66dfc9c6519ce"}
Sep 19 04:15:55.914013 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:55.913578    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wmk74" event={"ID":"3738f1c7-67af-4201-8e8d-1d0406e0edb9","Type":"ContainerStarted","Data":"81d7395dad60628a1210289810444118b63b9e5c957c7f0bf6c104abc4fccbcf"}
Sep 19 04:15:55.935257 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:55.935204    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-wmk74" podStartSLOduration=133.699121389 podStartE2EDuration="2m14.935187575s" totalImagesPullingTime="1.236066186s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 04:13:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:15:55.934586735 +0000 UTC m=+135.117547883" watchObservedRunningTime="2026-09-19 04:15:55.935187575 +0000 UTC m=+135.118148722"
Sep 19 04:15:57.837220 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:57.837191    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-h52kg_71273c3e-c393-4009-945f-f7f42c668d75/init-textfile/0.log"
Sep 19 04:15:58.039772 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:58.039741    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-h52kg_71273c3e-c393-4009-945f-f7f42c668d75/node-exporter/0.log"
Sep 19 04:15:58.235676 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:15:58.235640    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-h52kg_71273c3e-c393-4009-945f-f7f42c668d75/kube-rbac-proxy/0.log"
Sep 19 04:16:01.083087 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:16:01.083051    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5b49689ccb-n92tv"
Sep 19 04:16:01.083543 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:16:01.083215    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5b49689ccb-n92tv"
Sep 19 04:16:01.088742 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:16:01.088717    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5b49689ccb-n92tv"
Sep 19 04:16:01.954556 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:16:01.954522    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5b49689ccb-n92tv"
Sep 19 04:16:03.401604 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:16:03.401570    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6b6bfffd4-z7rp5"
Sep 19 04:16:03.402204 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:16:03.401639    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6b6bfffd4-z7rp5"
Sep 19 04:16:03.406316 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:16:03.406292    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6b6bfffd4-z7rp5"
Sep 19 04:16:03.953823 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:16:03.953792    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6b6bfffd4-z7rp5"
Sep 19 04:16:04.012027 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:16:04.011985    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5b49689ccb-n92tv"]
Sep 19 04:16:04.039119 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:16:04.039089    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5b49689ccb-n92tv_86c58995-ea08-4c7a-99a2-70b4dc820115/console/0.log"
Sep 19 04:16:04.236158 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:16:04.236067    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6b6bfffd4-z7rp5_a3f2cede-329e-4601-9e0a-65228d46bd47/console/0.log"
Sep 19 04:16:04.437808 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:16:04.437772    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6594f4c75-rhcwm_c0cd206c-8a93-45d6-80d6-edfc51d23f45/download-server/0.log"
Sep 19 04:16:29.971997 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:16:29.971947    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5b49689ccb-n92tv" podUID="86c58995-ea08-4c7a-99a2-70b4dc820115" containerName="console" containerID="cri-o://14a96d08c95dffba3903ac91f46b86753964edb28773e4c586c95e2cd25518ff" gracePeriod=15
Sep 19 04:16:30.241224 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:16:30.241158    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b49689ccb-n92tv"
Sep 19 04:16:30.380616 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:16:30.380583    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/86c58995-ea08-4c7a-99a2-70b4dc820115-oauth-serving-cert\") pod \"86c58995-ea08-4c7a-99a2-70b4dc820115\" (UID: \"86c58995-ea08-4c7a-99a2-70b4dc820115\") "
Sep 19 04:16:30.380840 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:16:30.380640    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qgp6z\" (UniqueName: \"kubernetes.io/projected/86c58995-ea08-4c7a-99a2-70b4dc820115-kube-api-access-qgp6z\") pod \"86c58995-ea08-4c7a-99a2-70b4dc820115\" (UID: \"86c58995-ea08-4c7a-99a2-70b4dc820115\") "
Sep 19 04:16:30.380840 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:16:30.380674    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/86c58995-ea08-4c7a-99a2-70b4dc820115-console-serving-cert\") pod \"86c58995-ea08-4c7a-99a2-70b4dc820115\" (UID: \"86c58995-ea08-4c7a-99a2-70b4dc820115\") "
Sep 19 04:16:30.380840 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:16:30.380714    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/86c58995-ea08-4c7a-99a2-70b4dc820115-console-oauth-config\") pod \"86c58995-ea08-4c7a-99a2-70b4dc820115\" (UID: \"86c58995-ea08-4c7a-99a2-70b4dc820115\") "
Sep 19 04:16:30.380840 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:16:30.380756    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/86c58995-ea08-4c7a-99a2-70b4dc820115-service-ca\") pod \"86c58995-ea08-4c7a-99a2-70b4dc820115\" (UID: \"86c58995-ea08-4c7a-99a2-70b4dc820115\") "
Sep 19 04:16:30.380840 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:16:30.380800    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/86c58995-ea08-4c7a-99a2-70b4dc820115-console-config\") pod \"86c58995-ea08-4c7a-99a2-70b4dc820115\" (UID: \"86c58995-ea08-4c7a-99a2-70b4dc820115\") "
Sep 19 04:16:30.381178 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:16:30.381142    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86c58995-ea08-4c7a-99a2-70b4dc820115-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "86c58995-ea08-4c7a-99a2-70b4dc820115" (UID: "86c58995-ea08-4c7a-99a2-70b4dc820115"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 04:16:30.381263 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:16:30.381225    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86c58995-ea08-4c7a-99a2-70b4dc820115-console-config" (OuterVolumeSpecName: "console-config") pod "86c58995-ea08-4c7a-99a2-70b4dc820115" (UID: "86c58995-ea08-4c7a-99a2-70b4dc820115"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 04:16:30.381263 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:16:30.381223    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86c58995-ea08-4c7a-99a2-70b4dc820115-service-ca" (OuterVolumeSpecName: "service-ca") pod "86c58995-ea08-4c7a-99a2-70b4dc820115" (UID: "86c58995-ea08-4c7a-99a2-70b4dc820115"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 04:16:30.383268 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:16:30.383237    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86c58995-ea08-4c7a-99a2-70b4dc820115-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "86c58995-ea08-4c7a-99a2-70b4dc820115" (UID: "86c58995-ea08-4c7a-99a2-70b4dc820115"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 04:16:30.383379 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:16:30.383273    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86c58995-ea08-4c7a-99a2-70b4dc820115-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "86c58995-ea08-4c7a-99a2-70b4dc820115" (UID: "86c58995-ea08-4c7a-99a2-70b4dc820115"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 04:16:30.383379 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:16:30.383286    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86c58995-ea08-4c7a-99a2-70b4dc820115-kube-api-access-qgp6z" (OuterVolumeSpecName: "kube-api-access-qgp6z") pod "86c58995-ea08-4c7a-99a2-70b4dc820115" (UID: "86c58995-ea08-4c7a-99a2-70b4dc820115"). InnerVolumeSpecName "kube-api-access-qgp6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 04:16:30.481541 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:16:30.481503    2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/86c58995-ea08-4c7a-99a2-70b4dc820115-console-serving-cert\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:16:30.481541 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:16:30.481530    2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/86c58995-ea08-4c7a-99a2-70b4dc820115-console-oauth-config\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:16:30.481541 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:16:30.481541    2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/86c58995-ea08-4c7a-99a2-70b4dc820115-service-ca\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:16:30.481541 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:16:30.481549    2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/86c58995-ea08-4c7a-99a2-70b4dc820115-console-config\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:16:30.481821 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:16:30.481557    2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/86c58995-ea08-4c7a-99a2-70b4dc820115-oauth-serving-cert\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:16:30.481821 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:16:30.481565    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgp6z\" (UniqueName: \"kubernetes.io/projected/86c58995-ea08-4c7a-99a2-70b4dc820115-kube-api-access-qgp6z\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:16:31.053116 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:16:31.053080    2579 generic.go:356] "Generic (PLEG): container finished" podID="86c58995-ea08-4c7a-99a2-70b4dc820115" containerID="14a96d08c95dffba3903ac91f46b86753964edb28773e4c586c95e2cd25518ff" exitCode=0
Sep 19 04:16:31.053116 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:16:31.053121    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b49689ccb-n92tv" event={"ID":"86c58995-ea08-4c7a-99a2-70b4dc820115","Type":"ContainerDied","Data":"14a96d08c95dffba3903ac91f46b86753964edb28773e4c586c95e2cd25518ff"}
Sep 19 04:16:31.053543 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:16:31.053143    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b49689ccb-n92tv"
Sep 19 04:16:31.053543 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:16:31.053182    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b49689ccb-n92tv" event={"ID":"86c58995-ea08-4c7a-99a2-70b4dc820115","Type":"ContainerDied","Data":"e562fdc818cfafda7de9c4da612252038b1279475636e5f9b0212ea2335a6ccb"}
Sep 19 04:16:31.053543 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:16:31.053201    2579 scope.go:117] "RemoveContainer" containerID="14a96d08c95dffba3903ac91f46b86753964edb28773e4c586c95e2cd25518ff"
Sep 19 04:16:31.061519 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:16:31.061497    2579 scope.go:117] "RemoveContainer" containerID="14a96d08c95dffba3903ac91f46b86753964edb28773e4c586c95e2cd25518ff"
Sep 19 04:16:31.061855 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:16:31.061827    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14a96d08c95dffba3903ac91f46b86753964edb28773e4c586c95e2cd25518ff\": container with ID starting with 14a96d08c95dffba3903ac91f46b86753964edb28773e4c586c95e2cd25518ff not found: ID does not exist" containerID="14a96d08c95dffba3903ac91f46b86753964edb28773e4c586c95e2cd25518ff"
Sep 19 04:16:31.061927 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:16:31.061867    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14a96d08c95dffba3903ac91f46b86753964edb28773e4c586c95e2cd25518ff"} err="failed to get container status \"14a96d08c95dffba3903ac91f46b86753964edb28773e4c586c95e2cd25518ff\": rpc error: code = NotFound desc = could not find container \"14a96d08c95dffba3903ac91f46b86753964edb28773e4c586c95e2cd25518ff\": container with ID starting with 14a96d08c95dffba3903ac91f46b86753964edb28773e4c586c95e2cd25518ff not found: ID does not exist"
Sep 19 04:16:31.084339 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:16:31.084314    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5b49689ccb-n92tv"]
Sep 19 04:16:31.094767 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:16:31.094745    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5b49689ccb-n92tv"]
Sep 19 04:16:31.349659 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:16:31.349581    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86c58995-ea08-4c7a-99a2-70b4dc820115" path="/var/lib/kubelet/pods/86c58995-ea08-4c7a-99a2-70b4dc820115/volumes"
Sep 19 04:16:34.255522 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:16:34.255434    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-b6f457954-wwpfn"]
Sep 19 04:16:34.255985 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:16:34.255797    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86c58995-ea08-4c7a-99a2-70b4dc820115" containerName="console"
Sep 19 04:16:34.255985 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:16:34.255816    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="86c58995-ea08-4c7a-99a2-70b4dc820115" containerName="console"
Sep 19 04:16:34.255985 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:16:34.255884    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="86c58995-ea08-4c7a-99a2-70b4dc820115" containerName="console"
Sep 19 04:16:34.280524 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:16:34.280489    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b6f457954-wwpfn"
Sep 19 04:16:34.316385 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:16:34.316347    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b6f457954-wwpfn"]
Sep 19 04:16:34.411151 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:16:34.411088    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cfb3087d-22a4-4962-8199-2b989fe224fe-service-ca\") pod \"console-b6f457954-wwpfn\" (UID: \"cfb3087d-22a4-4962-8199-2b989fe224fe\") " pod="openshift-console/console-b6f457954-wwpfn"
Sep 19 04:16:34.411151 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:16:34.411149    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cfb3087d-22a4-4962-8199-2b989fe224fe-oauth-serving-cert\") pod \"console-b6f457954-wwpfn\" (UID: \"cfb3087d-22a4-4962-8199-2b989fe224fe\") " pod="openshift-console/console-b6f457954-wwpfn"
Sep 19 04:16:34.411343 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:16:34.411179    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbg7t\" (UniqueName: \"kubernetes.io/projected/cfb3087d-22a4-4962-8199-2b989fe224fe-kube-api-access-hbg7t\") pod \"console-b6f457954-wwpfn\" (UID: \"cfb3087d-22a4-4962-8199-2b989fe224fe\") " pod="openshift-console/console-b6f457954-wwpfn"
Sep 19 04:16:34.411343 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:16:34.411200    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cfb3087d-22a4-4962-8199-2b989fe224fe-trusted-ca-bundle\") pod \"console-b6f457954-wwpfn\" (UID: \"cfb3087d-22a4-4962-8199-2b989fe224fe\") " pod="openshift-console/console-b6f457954-wwpfn"
Sep 19 04:16:34.411343 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:16:34.411220    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cfb3087d-22a4-4962-8199-2b989fe224fe-console-serving-cert\") pod \"console-b6f457954-wwpfn\" (UID: \"cfb3087d-22a4-4962-8199-2b989fe224fe\") " pod="openshift-console/console-b6f457954-wwpfn"
Sep 19 04:16:34.411343 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:16:34.411240    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cfb3087d-22a4-4962-8199-2b989fe224fe-console-config\") pod \"console-b6f457954-wwpfn\" (UID: \"cfb3087d-22a4-4962-8199-2b989fe224fe\") " pod="openshift-console/console-b6f457954-wwpfn"
Sep 19 04:16:34.411465 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:16:34.411350    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cfb3087d-22a4-4962-8199-2b989fe224fe-console-oauth-config\") pod \"console-b6f457954-wwpfn\" (UID: \"cfb3087d-22a4-4962-8199-2b989fe224fe\") " pod="openshift-console/console-b6f457954-wwpfn"
Sep 19 04:16:34.512669 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:16:34.512577    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cfb3087d-22a4-4962-8199-2b989fe224fe-trusted-ca-bundle\") pod \"console-b6f457954-wwpfn\" (UID: \"cfb3087d-22a4-4962-8199-2b989fe224fe\") " pod="openshift-console/console-b6f457954-wwpfn"
Sep 19 04:16:34.512669 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:16:34.512615    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cfb3087d-22a4-4962-8199-2b989fe224fe-console-serving-cert\") pod \"console-b6f457954-wwpfn\" (UID: \"cfb3087d-22a4-4962-8199-2b989fe224fe\") " pod="openshift-console/console-b6f457954-wwpfn"
Sep 19 04:16:34.512669 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:16:34.512634    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cfb3087d-22a4-4962-8199-2b989fe224fe-console-config\") pod \"console-b6f457954-wwpfn\" (UID: \"cfb3087d-22a4-4962-8199-2b989fe224fe\") " pod="openshift-console/console-b6f457954-wwpfn"
Sep 19 04:16:34.512980 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:16:34.512819    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cfb3087d-22a4-4962-8199-2b989fe224fe-console-oauth-config\") pod \"console-b6f457954-wwpfn\" (UID: \"cfb3087d-22a4-4962-8199-2b989fe224fe\") " pod="openshift-console/console-b6f457954-wwpfn"
Sep 19 04:16:34.512980 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:16:34.512864    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cfb3087d-22a4-4962-8199-2b989fe224fe-service-ca\") pod \"console-b6f457954-wwpfn\" (UID: \"cfb3087d-22a4-4962-8199-2b989fe224fe\") " pod="openshift-console/console-b6f457954-wwpfn"
Sep 19 04:16:34.512980 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:16:34.512883    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cfb3087d-22a4-4962-8199-2b989fe224fe-oauth-serving-cert\") pod \"console-b6f457954-wwpfn\" (UID: \"cfb3087d-22a4-4962-8199-2b989fe224fe\") " pod="openshift-console/console-b6f457954-wwpfn"
Sep 19 04:16:34.512980 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:16:34.512903    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbg7t\" (UniqueName: \"kubernetes.io/projected/cfb3087d-22a4-4962-8199-2b989fe224fe-kube-api-access-hbg7t\") pod \"console-b6f457954-wwpfn\" (UID: \"cfb3087d-22a4-4962-8199-2b989fe224fe\") " pod="openshift-console/console-b6f457954-wwpfn"
Sep 19 04:16:34.513400 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:16:34.513369    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cfb3087d-22a4-4962-8199-2b989fe224fe-console-config\") pod \"console-b6f457954-wwpfn\" (UID: \"cfb3087d-22a4-4962-8199-2b989fe224fe\") " pod="openshift-console/console-b6f457954-wwpfn"
Sep 19 04:16:34.513555 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:16:34.513532    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cfb3087d-22a4-4962-8199-2b989fe224fe-oauth-serving-cert\") pod \"console-b6f457954-wwpfn\" (UID: \"cfb3087d-22a4-4962-8199-2b989fe224fe\") " pod="openshift-console/console-b6f457954-wwpfn"
Sep 19 04:16:34.513604 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:16:34.513548    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cfb3087d-22a4-4962-8199-2b989fe224fe-service-ca\") pod \"console-b6f457954-wwpfn\" (UID: \"cfb3087d-22a4-4962-8199-2b989fe224fe\") " pod="openshift-console/console-b6f457954-wwpfn"
Sep 19 04:16:34.513643 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:16:34.513625    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cfb3087d-22a4-4962-8199-2b989fe224fe-trusted-ca-bundle\") pod \"console-b6f457954-wwpfn\" (UID: \"cfb3087d-22a4-4962-8199-2b989fe224fe\") " pod="openshift-console/console-b6f457954-wwpfn"
Sep 19 04:16:34.515862 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:16:34.515842    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cfb3087d-22a4-4962-8199-2b989fe224fe-console-oauth-config\") pod \"console-b6f457954-wwpfn\" (UID: \"cfb3087d-22a4-4962-8199-2b989fe224fe\") " pod="openshift-console/console-b6f457954-wwpfn"
Sep 19 04:16:34.515971 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:16:34.515959    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cfb3087d-22a4-4962-8199-2b989fe224fe-console-serving-cert\") pod \"console-b6f457954-wwpfn\" (UID: \"cfb3087d-22a4-4962-8199-2b989fe224fe\") " pod="openshift-console/console-b6f457954-wwpfn"
Sep 19 04:16:34.527507 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:16:34.527481    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbg7t\" (UniqueName: \"kubernetes.io/projected/cfb3087d-22a4-4962-8199-2b989fe224fe-kube-api-access-hbg7t\") pod \"console-b6f457954-wwpfn\" (UID: \"cfb3087d-22a4-4962-8199-2b989fe224fe\") " pod="openshift-console/console-b6f457954-wwpfn"
Sep 19 04:16:34.589481 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:16:34.589436    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b6f457954-wwpfn"
Sep 19 04:16:34.902816 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:16:34.902790    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b6f457954-wwpfn"]
Sep 19 04:16:34.905749 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:16:34.905719    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfb3087d_22a4_4962_8199_2b989fe224fe.slice/crio-64a87174a170166ad008d40db98e2191900b24549e911deec6b9e68ff4749e2c WatchSource:0}: Error finding container 64a87174a170166ad008d40db98e2191900b24549e911deec6b9e68ff4749e2c: Status 404 returned error can't find the container with id 64a87174a170166ad008d40db98e2191900b24549e911deec6b9e68ff4749e2c
Sep 19 04:16:35.074340 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:16:35.074302    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b6f457954-wwpfn" event={"ID":"cfb3087d-22a4-4962-8199-2b989fe224fe","Type":"ContainerStarted","Data":"546e07faed51701e0d9f49af7fc88c37194fafca8e7efcf06bb4f6405d337e18"}
Sep 19 04:16:35.074512 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:16:35.074394    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b6f457954-wwpfn" event={"ID":"cfb3087d-22a4-4962-8199-2b989fe224fe","Type":"ContainerStarted","Data":"64a87174a170166ad008d40db98e2191900b24549e911deec6b9e68ff4749e2c"}
Sep 19 04:16:35.110478 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:16:35.110415    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-b6f457954-wwpfn" podStartSLOduration=1.110398485 podStartE2EDuration="1.110398485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 04:16:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:16:35.109850572 +0000 UTC m=+174.292811720" watchObservedRunningTime="2026-09-19 04:16:35.110398485 +0000 UTC m=+174.293359633"
Sep 19 04:16:44.590032 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:16:44.589976    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-b6f457954-wwpfn"
Sep 19 04:16:44.590461 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:16:44.590058    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-b6f457954-wwpfn"
Sep 19 04:16:44.594745 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:16:44.594725    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-b6f457954-wwpfn"
Sep 19 04:16:45.112000 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:16:45.111973    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-b6f457954-wwpfn"
Sep 19 04:16:45.215725 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:16:45.215676    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b6bfffd4-z7rp5"]
Sep 19 04:17:10.237743 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:17:10.237660    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6b6bfffd4-z7rp5" podUID="a3f2cede-329e-4601-9e0a-65228d46bd47" containerName="console" containerID="cri-o://935c49f8c099d1edf353819b892292f3d8eb32eb8cbdf7bb9203bda9bade335f" gracePeriod=15
Sep 19 04:17:10.475908 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:17:10.475884    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b6bfffd4-z7rp5"
Sep 19 04:17:10.582494 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:17:10.582398    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a3f2cede-329e-4601-9e0a-65228d46bd47-console-config\") pod \"a3f2cede-329e-4601-9e0a-65228d46bd47\" (UID: \"a3f2cede-329e-4601-9e0a-65228d46bd47\") "
Sep 19 04:17:10.582494 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:17:10.582433    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a3f2cede-329e-4601-9e0a-65228d46bd47-service-ca\") pod \"a3f2cede-329e-4601-9e0a-65228d46bd47\" (UID: \"a3f2cede-329e-4601-9e0a-65228d46bd47\") "
Sep 19 04:17:10.582494 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:17:10.582471    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a3f2cede-329e-4601-9e0a-65228d46bd47-oauth-serving-cert\") pod \"a3f2cede-329e-4601-9e0a-65228d46bd47\" (UID: \"a3f2cede-329e-4601-9e0a-65228d46bd47\") "
Sep 19 04:17:10.582494 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:17:10.582489    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a3f2cede-329e-4601-9e0a-65228d46bd47-console-oauth-config\") pod \"a3f2cede-329e-4601-9e0a-65228d46bd47\" (UID: \"a3f2cede-329e-4601-9e0a-65228d46bd47\") "
Sep 19 04:17:10.582825 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:17:10.582518    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a3f2cede-329e-4601-9e0a-65228d46bd47-console-serving-cert\") pod \"a3f2cede-329e-4601-9e0a-65228d46bd47\" (UID: \"a3f2cede-329e-4601-9e0a-65228d46bd47\") "
Sep 19 04:17:10.582825 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:17:10.582546    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tdbjg\" (UniqueName: \"kubernetes.io/projected/a3f2cede-329e-4601-9e0a-65228d46bd47-kube-api-access-tdbjg\") pod \"a3f2cede-329e-4601-9e0a-65228d46bd47\" (UID: \"a3f2cede-329e-4601-9e0a-65228d46bd47\") "
Sep 19 04:17:10.582825 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:17:10.582604    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3f2cede-329e-4601-9e0a-65228d46bd47-trusted-ca-bundle\") pod \"a3f2cede-329e-4601-9e0a-65228d46bd47\" (UID: \"a3f2cede-329e-4601-9e0a-65228d46bd47\") "
Sep 19 04:17:10.582942 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:17:10.582909    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3f2cede-329e-4601-9e0a-65228d46bd47-service-ca" (OuterVolumeSpecName: "service-ca") pod "a3f2cede-329e-4601-9e0a-65228d46bd47" (UID: "a3f2cede-329e-4601-9e0a-65228d46bd47"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 04:17:10.582942 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:17:10.582918    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3f2cede-329e-4601-9e0a-65228d46bd47-console-config" (OuterVolumeSpecName: "console-config") pod "a3f2cede-329e-4601-9e0a-65228d46bd47" (UID: "a3f2cede-329e-4601-9e0a-65228d46bd47"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 04:17:10.583216 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:17:10.583181    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3f2cede-329e-4601-9e0a-65228d46bd47-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a3f2cede-329e-4601-9e0a-65228d46bd47" (UID: "a3f2cede-329e-4601-9e0a-65228d46bd47"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 04:17:10.583216 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:17:10.583205    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3f2cede-329e-4601-9e0a-65228d46bd47-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "a3f2cede-329e-4601-9e0a-65228d46bd47" (UID: "a3f2cede-329e-4601-9e0a-65228d46bd47"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 04:17:10.584930 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:17:10.584897    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3f2cede-329e-4601-9e0a-65228d46bd47-kube-api-access-tdbjg" (OuterVolumeSpecName: "kube-api-access-tdbjg") pod "a3f2cede-329e-4601-9e0a-65228d46bd47" (UID: "a3f2cede-329e-4601-9e0a-65228d46bd47"). InnerVolumeSpecName "kube-api-access-tdbjg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 04:17:10.585033 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:17:10.584945    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3f2cede-329e-4601-9e0a-65228d46bd47-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a3f2cede-329e-4601-9e0a-65228d46bd47" (UID: "a3f2cede-329e-4601-9e0a-65228d46bd47"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 04:17:10.585033 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:17:10.584972    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3f2cede-329e-4601-9e0a-65228d46bd47-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a3f2cede-329e-4601-9e0a-65228d46bd47" (UID: "a3f2cede-329e-4601-9e0a-65228d46bd47"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 04:17:10.683267 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:17:10.683228    2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a3f2cede-329e-4601-9e0a-65228d46bd47-oauth-serving-cert\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:17:10.683267 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:17:10.683262    2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a3f2cede-329e-4601-9e0a-65228d46bd47-console-oauth-config\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:17:10.683267 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:17:10.683275    2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a3f2cede-329e-4601-9e0a-65228d46bd47-console-serving-cert\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:17:10.683508 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:17:10.683288    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdbjg\" (UniqueName: \"kubernetes.io/projected/a3f2cede-329e-4601-9e0a-65228d46bd47-kube-api-access-tdbjg\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:17:10.683508 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:17:10.683299    2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3f2cede-329e-4601-9e0a-65228d46bd47-trusted-ca-bundle\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:17:10.683508 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:17:10.683310    2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a3f2cede-329e-4601-9e0a-65228d46bd47-console-config\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:17:10.683508 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:17:10.683321    2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a3f2cede-329e-4601-9e0a-65228d46bd47-service-ca\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:17:11.196992 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:17:11.196951    2579 generic.go:356] "Generic (PLEG): container finished" podID="a3f2cede-329e-4601-9e0a-65228d46bd47" containerID="935c49f8c099d1edf353819b892292f3d8eb32eb8cbdf7bb9203bda9bade335f" exitCode=0
Sep 19 04:17:11.197163 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:17:11.197001    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b6bfffd4-z7rp5"
Sep 19 04:17:11.197163 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:17:11.197029    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b6bfffd4-z7rp5" event={"ID":"a3f2cede-329e-4601-9e0a-65228d46bd47","Type":"ContainerDied","Data":"935c49f8c099d1edf353819b892292f3d8eb32eb8cbdf7bb9203bda9bade335f"}
Sep 19 04:17:11.197163 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:17:11.197073    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b6bfffd4-z7rp5" event={"ID":"a3f2cede-329e-4601-9e0a-65228d46bd47","Type":"ContainerDied","Data":"1f4e830239952f46427b13dc45bf90612297741d9f667b06ffaedd228c749e30"}
Sep 19 04:17:11.197163 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:17:11.197096    2579 scope.go:117] "RemoveContainer" containerID="935c49f8c099d1edf353819b892292f3d8eb32eb8cbdf7bb9203bda9bade335f"
Sep 19 04:17:11.205350 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:17:11.205331    2579 scope.go:117] "RemoveContainer" containerID="935c49f8c099d1edf353819b892292f3d8eb32eb8cbdf7bb9203bda9bade335f"
Sep 19 04:17:11.205598 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:17:11.205582    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"935c49f8c099d1edf353819b892292f3d8eb32eb8cbdf7bb9203bda9bade335f\": container with ID starting with 935c49f8c099d1edf353819b892292f3d8eb32eb8cbdf7bb9203bda9bade335f not found: ID does not exist" containerID="935c49f8c099d1edf353819b892292f3d8eb32eb8cbdf7bb9203bda9bade335f"
Sep 19 04:17:11.205651 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:17:11.205606    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"935c49f8c099d1edf353819b892292f3d8eb32eb8cbdf7bb9203bda9bade335f"} err="failed to get container status \"935c49f8c099d1edf353819b892292f3d8eb32eb8cbdf7bb9203bda9bade335f\": rpc error: code = NotFound desc = could not find container \"935c49f8c099d1edf353819b892292f3d8eb32eb8cbdf7bb9203bda9bade335f\": container with ID starting with 935c49f8c099d1edf353819b892292f3d8eb32eb8cbdf7bb9203bda9bade335f not found: ID does not exist"
Sep 19 04:17:11.231356 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:17:11.231321    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b6bfffd4-z7rp5"]
Sep 19 04:17:11.236685 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:17:11.236662    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6b6bfffd4-z7rp5"]
Sep 19 04:17:11.350457 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:17:11.350422    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3f2cede-329e-4601-9e0a-65228d46bd47" path="/var/lib/kubelet/pods/a3f2cede-329e-4601-9e0a-65228d46bd47/volumes"
Sep 19 04:17:57.089608 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:17:57.089573    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rwbh2"]
Sep 19 04:17:57.090082 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:17:57.089821    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3f2cede-329e-4601-9e0a-65228d46bd47" containerName="console"
Sep 19 04:17:57.090082 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:17:57.089834    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3f2cede-329e-4601-9e0a-65228d46bd47" containerName="console"
Sep 19 04:17:57.090082 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:17:57.089901    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3f2cede-329e-4601-9e0a-65228d46bd47" containerName="console"
Sep 19 04:17:57.094096 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:17:57.094078    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rwbh2"
Sep 19 04:17:57.096879 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:17:57.096859    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-ldcjk\""
Sep 19 04:17:57.096966 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:17:57.096945    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 19 04:17:57.097821 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:17:57.097804    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 19 04:17:57.103976 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:17:57.103954    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rwbh2"]
Sep 19 04:17:57.224682 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:17:57.224641    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l5s5\" (UniqueName: \"kubernetes.io/projected/1975d0b2-cda8-41b1-bf10-9b5f3ba155ee-kube-api-access-7l5s5\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rwbh2\" (UID: \"1975d0b2-cda8-41b1-bf10-9b5f3ba155ee\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rwbh2"
Sep 19 04:17:57.224888 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:17:57.224733    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1975d0b2-cda8-41b1-bf10-9b5f3ba155ee-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rwbh2\" (UID: \"1975d0b2-cda8-41b1-bf10-9b5f3ba155ee\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rwbh2"
Sep 19 04:17:57.224888 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:17:57.224754    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1975d0b2-cda8-41b1-bf10-9b5f3ba155ee-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rwbh2\" (UID: \"1975d0b2-cda8-41b1-bf10-9b5f3ba155ee\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rwbh2"
Sep 19 04:17:57.325747 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:17:57.325688    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1975d0b2-cda8-41b1-bf10-9b5f3ba155ee-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rwbh2\" (UID: \"1975d0b2-cda8-41b1-bf10-9b5f3ba155ee\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rwbh2"
Sep 19 04:17:57.325747 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:17:57.325745    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1975d0b2-cda8-41b1-bf10-9b5f3ba155ee-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rwbh2\" (UID: \"1975d0b2-cda8-41b1-bf10-9b5f3ba155ee\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rwbh2"
Sep 19 04:17:57.325944 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:17:57.325771    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l5s5\" (UniqueName: \"kubernetes.io/projected/1975d0b2-cda8-41b1-bf10-9b5f3ba155ee-kube-api-access-7l5s5\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rwbh2\" (UID: \"1975d0b2-cda8-41b1-bf10-9b5f3ba155ee\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rwbh2"
Sep 19 04:17:57.326156 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:17:57.326135    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1975d0b2-cda8-41b1-bf10-9b5f3ba155ee-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rwbh2\" (UID: \"1975d0b2-cda8-41b1-bf10-9b5f3ba155ee\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rwbh2"
Sep 19 04:17:57.326156 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:17:57.326149    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1975d0b2-cda8-41b1-bf10-9b5f3ba155ee-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rwbh2\" (UID: \"1975d0b2-cda8-41b1-bf10-9b5f3ba155ee\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rwbh2"
Sep 19 04:17:57.339176 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:17:57.339139    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l5s5\" (UniqueName: \"kubernetes.io/projected/1975d0b2-cda8-41b1-bf10-9b5f3ba155ee-kube-api-access-7l5s5\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rwbh2\" (UID: \"1975d0b2-cda8-41b1-bf10-9b5f3ba155ee\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rwbh2"
Sep 19 04:17:57.403877 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:17:57.403847    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rwbh2"
Sep 19 04:17:57.681963 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:17:57.681886    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rwbh2"]
Sep 19 04:17:57.685336 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:17:57.685309    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1975d0b2_cda8_41b1_bf10_9b5f3ba155ee.slice/crio-74e77b02da4f0c767009bd67ebc9a9783114771b13a3046c6ce173c1f8b905b6 WatchSource:0}: Error finding container 74e77b02da4f0c767009bd67ebc9a9783114771b13a3046c6ce173c1f8b905b6: Status 404 returned error can't find the container with id 74e77b02da4f0c767009bd67ebc9a9783114771b13a3046c6ce173c1f8b905b6
Sep 19 04:17:58.356161 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:17:58.356119    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rwbh2" event={"ID":"1975d0b2-cda8-41b1-bf10-9b5f3ba155ee","Type":"ContainerStarted","Data":"74e77b02da4f0c767009bd67ebc9a9783114771b13a3046c6ce173c1f8b905b6"}
Sep 19 04:18:01.329070 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:01.329036    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jkrrv"]
Sep 19 04:18:01.333646 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:01.333622    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jkrrv"
Sep 19 04:18:01.338817 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:01.338791    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 19 04:18:01.338817 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:01.338809    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-mjhqw\""
Sep 19 04:18:01.357187 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:01.357154    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2130d608-c9dd-4f97-8348-bb1a10b7fec5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-jkrrv\" (UID: \"2130d608-c9dd-4f97-8348-bb1a10b7fec5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jkrrv"
Sep 19 04:18:01.357358 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:01.357203    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2130d608-c9dd-4f97-8348-bb1a10b7fec5-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-jkrrv\" (UID: \"2130d608-c9dd-4f97-8348-bb1a10b7fec5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jkrrv"
Sep 19 04:18:01.357358 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:01.357267    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2130d608-c9dd-4f97-8348-bb1a10b7fec5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-jkrrv\" (UID: \"2130d608-c9dd-4f97-8348-bb1a10b7fec5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jkrrv"
Sep 19 04:18:01.357358 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:01.357324    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2130d608-c9dd-4f97-8348-bb1a10b7fec5-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-jkrrv\" (UID: \"2130d608-c9dd-4f97-8348-bb1a10b7fec5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jkrrv"
Sep 19 04:18:01.357527 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:01.357368    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2130d608-c9dd-4f97-8348-bb1a10b7fec5-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-jkrrv\" (UID: \"2130d608-c9dd-4f97-8348-bb1a10b7fec5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jkrrv"
Sep 19 04:18:01.357527 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:01.357466    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2130d608-c9dd-4f97-8348-bb1a10b7fec5-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-jkrrv\" (UID: \"2130d608-c9dd-4f97-8348-bb1a10b7fec5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jkrrv"
Sep 19 04:18:01.357632 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:01.357532    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgq9z\" (UniqueName: \"kubernetes.io/projected/2130d608-c9dd-4f97-8348-bb1a10b7fec5-kube-api-access-lgq9z\") pod \"maas-default-gateway-openshift-default-5ccb968c66-jkrrv\" (UID: \"2130d608-c9dd-4f97-8348-bb1a10b7fec5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jkrrv"
Sep 19 04:18:01.357632 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:01.357569    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2130d608-c9dd-4f97-8348-bb1a10b7fec5-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-jkrrv\" (UID: \"2130d608-c9dd-4f97-8348-bb1a10b7fec5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jkrrv"
Sep 19 04:18:01.357632 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:01.357591    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2130d608-c9dd-4f97-8348-bb1a10b7fec5-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-jkrrv\" (UID: \"2130d608-c9dd-4f97-8348-bb1a10b7fec5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jkrrv"
Sep 19 04:18:01.365351 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:01.365284    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jkrrv"]
Sep 19 04:18:01.458069 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:01.458029    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2130d608-c9dd-4f97-8348-bb1a10b7fec5-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-jkrrv\" (UID: \"2130d608-c9dd-4f97-8348-bb1a10b7fec5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jkrrv"
Sep 19 04:18:01.458069 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:01.458080    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2130d608-c9dd-4f97-8348-bb1a10b7fec5-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-jkrrv\" (UID: \"2130d608-c9dd-4f97-8348-bb1a10b7fec5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jkrrv"
Sep 19 04:18:01.458320 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:01.458111    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2130d608-c9dd-4f97-8348-bb1a10b7fec5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-jkrrv\" (UID: \"2130d608-c9dd-4f97-8348-bb1a10b7fec5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jkrrv"
Sep 19 04:18:01.458320 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:01.458142    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2130d608-c9dd-4f97-8348-bb1a10b7fec5-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-jkrrv\" (UID: \"2130d608-c9dd-4f97-8348-bb1a10b7fec5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jkrrv"
Sep 19 04:18:01.458320 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:01.458173    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2130d608-c9dd-4f97-8348-bb1a10b7fec5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-jkrrv\" (UID: \"2130d608-c9dd-4f97-8348-bb1a10b7fec5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jkrrv"
Sep 19 04:18:01.458320 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:01.458198    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2130d608-c9dd-4f97-8348-bb1a10b7fec5-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-jkrrv\" (UID: \"2130d608-c9dd-4f97-8348-bb1a10b7fec5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jkrrv"
Sep 19 04:18:01.458320 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:01.458227    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2130d608-c9dd-4f97-8348-bb1a10b7fec5-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-jkrrv\" (UID: \"2130d608-c9dd-4f97-8348-bb1a10b7fec5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jkrrv"
Sep 19 04:18:01.458320 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:01.458280    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2130d608-c9dd-4f97-8348-bb1a10b7fec5-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-jkrrv\" (UID: \"2130d608-c9dd-4f97-8348-bb1a10b7fec5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jkrrv"
Sep 19 04:18:01.458609 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:01.458326    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgq9z\" (UniqueName: \"kubernetes.io/projected/2130d608-c9dd-4f97-8348-bb1a10b7fec5-kube-api-access-lgq9z\") pod \"maas-default-gateway-openshift-default-5ccb968c66-jkrrv\" (UID: \"2130d608-c9dd-4f97-8348-bb1a10b7fec5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jkrrv"
Sep 19 04:18:01.458609 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:01.458458    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2130d608-c9dd-4f97-8348-bb1a10b7fec5-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-jkrrv\" (UID: \"2130d608-c9dd-4f97-8348-bb1a10b7fec5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jkrrv"
Sep 19 04:18:01.459131 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:01.459084    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2130d608-c9dd-4f97-8348-bb1a10b7fec5-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-jkrrv\" (UID: \"2130d608-c9dd-4f97-8348-bb1a10b7fec5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jkrrv"
Sep 19 04:18:01.459131 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:01.459118    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2130d608-c9dd-4f97-8348-bb1a10b7fec5-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-jkrrv\" (UID: \"2130d608-c9dd-4f97-8348-bb1a10b7fec5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jkrrv"
Sep 19 04:18:01.459305 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:01.459122    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2130d608-c9dd-4f97-8348-bb1a10b7fec5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-jkrrv\" (UID: \"2130d608-c9dd-4f97-8348-bb1a10b7fec5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jkrrv"
Sep 19 04:18:01.459305 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:01.459195    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2130d608-c9dd-4f97-8348-bb1a10b7fec5-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-jkrrv\" (UID: \"2130d608-c9dd-4f97-8348-bb1a10b7fec5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jkrrv"
Sep 19 04:18:01.461212 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:01.461188    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2130d608-c9dd-4f97-8348-bb1a10b7fec5-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-jkrrv\" (UID: \"2130d608-c9dd-4f97-8348-bb1a10b7fec5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jkrrv"
Sep 19 04:18:01.461546 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:01.461519    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2130d608-c9dd-4f97-8348-bb1a10b7fec5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-jkrrv\" (UID: \"2130d608-c9dd-4f97-8348-bb1a10b7fec5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jkrrv"
Sep 19 04:18:01.477242 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:01.477205    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2130d608-c9dd-4f97-8348-bb1a10b7fec5-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-jkrrv\" (UID: \"2130d608-c9dd-4f97-8348-bb1a10b7fec5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jkrrv"
Sep 19 04:18:01.487249 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:01.487216    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgq9z\" (UniqueName: \"kubernetes.io/projected/2130d608-c9dd-4f97-8348-bb1a10b7fec5-kube-api-access-lgq9z\") pod \"maas-default-gateway-openshift-default-5ccb968c66-jkrrv\" (UID: \"2130d608-c9dd-4f97-8348-bb1a10b7fec5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jkrrv"
Sep 19 04:18:01.512815 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:01.512758    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-6987f5b5b9-jb9kg"]
Sep 19 04:18:01.521011 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:01.520983    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6987f5b5b9-jb9kg"
Sep 19 04:18:01.527991 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:01.527965    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-c8mbv\""
Sep 19 04:18:01.558898 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:01.558860    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk8mj\" (UniqueName: \"kubernetes.io/projected/0651ba9f-b7df-4e1e-88ba-d8cae0519cd7-kube-api-access-nk8mj\") pod \"openshift-ai-inference-openshift-ai-inference-6987f5b5b9-jb9kg\" (UID: \"0651ba9f-b7df-4e1e-88ba-d8cae0519cd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6987f5b5b9-jb9kg"
Sep 19 04:18:01.559103 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:01.558909    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0651ba9f-b7df-4e1e-88ba-d8cae0519cd7-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6987f5b5b9-jb9kg\" (UID: \"0651ba9f-b7df-4e1e-88ba-d8cae0519cd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6987f5b5b9-jb9kg"
Sep 19 04:18:01.559103 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:01.558935    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0651ba9f-b7df-4e1e-88ba-d8cae0519cd7-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-6987f5b5b9-jb9kg\" (UID: \"0651ba9f-b7df-4e1e-88ba-d8cae0519cd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6987f5b5b9-jb9kg"
Sep 19 04:18:01.559103 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:01.558996    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0651ba9f-b7df-4e1e-88ba-d8cae0519cd7-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-6987f5b5b9-jb9kg\" (UID: \"0651ba9f-b7df-4e1e-88ba-d8cae0519cd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6987f5b5b9-jb9kg"
Sep 19 04:18:01.559103 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:01.559029    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0651ba9f-b7df-4e1e-88ba-d8cae0519cd7-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6987f5b5b9-jb9kg\" (UID: \"0651ba9f-b7df-4e1e-88ba-d8cae0519cd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6987f5b5b9-jb9kg"
Sep 19 04:18:01.559103 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:01.559076    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0651ba9f-b7df-4e1e-88ba-d8cae0519cd7-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-6987f5b5b9-jb9kg\" (UID: \"0651ba9f-b7df-4e1e-88ba-d8cae0519cd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6987f5b5b9-jb9kg"
Sep 19 04:18:01.559369 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:01.559158    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0651ba9f-b7df-4e1e-88ba-d8cae0519cd7-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-6987f5b5b9-jb9kg\" (UID: \"0651ba9f-b7df-4e1e-88ba-d8cae0519cd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6987f5b5b9-jb9kg"
Sep 19 04:18:01.559369 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:01.559199    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0651ba9f-b7df-4e1e-88ba-d8cae0519cd7-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-6987f5b5b9-jb9kg\" (UID: \"0651ba9f-b7df-4e1e-88ba-d8cae0519cd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6987f5b5b9-jb9kg"
Sep 19 04:18:01.559369 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:01.559230    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0651ba9f-b7df-4e1e-88ba-d8cae0519cd7-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-6987f5b5b9-jb9kg\" (UID: \"0651ba9f-b7df-4e1e-88ba-d8cae0519cd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6987f5b5b9-jb9kg"
Sep 19 04:18:01.589114 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:01.589023    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-6987f5b5b9-jb9kg"]
Sep 19 04:18:01.646047 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:01.646010    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jkrrv"
Sep 19 04:18:01.660229 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:01.660197    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk8mj\" (UniqueName: \"kubernetes.io/projected/0651ba9f-b7df-4e1e-88ba-d8cae0519cd7-kube-api-access-nk8mj\") pod \"openshift-ai-inference-openshift-ai-inference-6987f5b5b9-jb9kg\" (UID: \"0651ba9f-b7df-4e1e-88ba-d8cae0519cd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6987f5b5b9-jb9kg"
Sep 19 04:18:01.660395 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:01.660235    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0651ba9f-b7df-4e1e-88ba-d8cae0519cd7-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6987f5b5b9-jb9kg\" (UID: \"0651ba9f-b7df-4e1e-88ba-d8cae0519cd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6987f5b5b9-jb9kg"
Sep 19 04:18:01.660395 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:01.660255    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0651ba9f-b7df-4e1e-88ba-d8cae0519cd7-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-6987f5b5b9-jb9kg\" (UID: \"0651ba9f-b7df-4e1e-88ba-d8cae0519cd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6987f5b5b9-jb9kg"
Sep 19 04:18:01.660395 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:01.660287    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0651ba9f-b7df-4e1e-88ba-d8cae0519cd7-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-6987f5b5b9-jb9kg\" (UID: \"0651ba9f-b7df-4e1e-88ba-d8cae0519cd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6987f5b5b9-jb9kg"
Sep 19 04:18:01.660395 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:01.660320    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0651ba9f-b7df-4e1e-88ba-d8cae0519cd7-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6987f5b5b9-jb9kg\" (UID: \"0651ba9f-b7df-4e1e-88ba-d8cae0519cd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6987f5b5b9-jb9kg"
Sep 19 04:18:01.660395 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:01.660380    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0651ba9f-b7df-4e1e-88ba-d8cae0519cd7-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-6987f5b5b9-jb9kg\" (UID: \"0651ba9f-b7df-4e1e-88ba-d8cae0519cd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6987f5b5b9-jb9kg"
Sep 19 04:18:01.660636 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:01.660410    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0651ba9f-b7df-4e1e-88ba-d8cae0519cd7-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-6987f5b5b9-jb9kg\" (UID: \"0651ba9f-b7df-4e1e-88ba-d8cae0519cd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6987f5b5b9-jb9kg"
Sep 19 04:18:01.660636 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:01.660443    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0651ba9f-b7df-4e1e-88ba-d8cae0519cd7-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-6987f5b5b9-jb9kg\" (UID: \"0651ba9f-b7df-4e1e-88ba-d8cae0519cd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6987f5b5b9-jb9kg"
Sep 19 04:18:01.660636 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:01.660474    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0651ba9f-b7df-4e1e-88ba-d8cae0519cd7-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-6987f5b5b9-jb9kg\" (UID: \"0651ba9f-b7df-4e1e-88ba-d8cae0519cd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6987f5b5b9-jb9kg"
Sep 19 04:18:01.660810 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:01.660656    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0651ba9f-b7df-4e1e-88ba-d8cae0519cd7-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6987f5b5b9-jb9kg\" (UID: \"0651ba9f-b7df-4e1e-88ba-d8cae0519cd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6987f5b5b9-jb9kg"
Sep 19 04:18:01.660965 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:01.660942    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0651ba9f-b7df-4e1e-88ba-d8cae0519cd7-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-6987f5b5b9-jb9kg\" (UID: \"0651ba9f-b7df-4e1e-88ba-d8cae0519cd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6987f5b5b9-jb9kg"
Sep 19 04:18:01.661039 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:01.661003    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0651ba9f-b7df-4e1e-88ba-d8cae0519cd7-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6987f5b5b9-jb9kg\" (UID: \"0651ba9f-b7df-4e1e-88ba-d8cae0519cd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6987f5b5b9-jb9kg"
Sep 19 04:18:01.661328 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:01.661261    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0651ba9f-b7df-4e1e-88ba-d8cae0519cd7-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-6987f5b5b9-jb9kg\" (UID: \"0651ba9f-b7df-4e1e-88ba-d8cae0519cd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6987f5b5b9-jb9kg"
Sep 19 04:18:01.661497 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:01.661469    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0651ba9f-b7df-4e1e-88ba-d8cae0519cd7-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-6987f5b5b9-jb9kg\" (UID: \"0651ba9f-b7df-4e1e-88ba-d8cae0519cd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6987f5b5b9-jb9kg"
Sep 19 04:18:01.663132 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:01.663111    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0651ba9f-b7df-4e1e-88ba-d8cae0519cd7-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-6987f5b5b9-jb9kg\" (UID: \"0651ba9f-b7df-4e1e-88ba-d8cae0519cd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6987f5b5b9-jb9kg"
Sep 19 04:18:01.663319 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:01.663297    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0651ba9f-b7df-4e1e-88ba-d8cae0519cd7-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-6987f5b5b9-jb9kg\" (UID: \"0651ba9f-b7df-4e1e-88ba-d8cae0519cd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6987f5b5b9-jb9kg"
Sep 19 04:18:01.679090 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:01.679054    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk8mj\" (UniqueName: \"kubernetes.io/projected/0651ba9f-b7df-4e1e-88ba-d8cae0519cd7-kube-api-access-nk8mj\") pod \"openshift-ai-inference-openshift-ai-inference-6987f5b5b9-jb9kg\" (UID: \"0651ba9f-b7df-4e1e-88ba-d8cae0519cd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6987f5b5b9-jb9kg"
Sep 19 04:18:01.718893 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:01.718861    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0651ba9f-b7df-4e1e-88ba-d8cae0519cd7-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-6987f5b5b9-jb9kg\" (UID: \"0651ba9f-b7df-4e1e-88ba-d8cae0519cd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6987f5b5b9-jb9kg"
Sep 19 04:18:01.834849 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:01.834812    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6987f5b5b9-jb9kg"
Sep 19 04:18:01.961871 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:01.961833    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crh5q5"]
Sep 19 04:18:01.965143 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:01.965116    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crh5q5"
Sep 19 04:18:02.008679 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:02.008640    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crh5q5"]
Sep 19 04:18:02.063922 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:02.063862    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/37ab62f0-2d8d-41ce-a442-e5aff9b5232f-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crh5q5\" (UID: \"37ab62f0-2d8d-41ce-a442-e5aff9b5232f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crh5q5"
Sep 19 04:18:02.064120 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:02.063927    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9vcp\" (UniqueName: \"kubernetes.io/projected/37ab62f0-2d8d-41ce-a442-e5aff9b5232f-kube-api-access-c9vcp\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crh5q5\" (UID: \"37ab62f0-2d8d-41ce-a442-e5aff9b5232f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crh5q5"
Sep 19 04:18:02.064120 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:02.063981    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/37ab62f0-2d8d-41ce-a442-e5aff9b5232f-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crh5q5\" (UID: \"37ab62f0-2d8d-41ce-a442-e5aff9b5232f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crh5q5"
Sep 19 04:18:02.114348 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:02.114261    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c698cvr"]
Sep 19 04:18:02.118069 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:02.118033    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c698cvr"
Sep 19 04:18:02.145876 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:02.145843    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c698cvr"]
Sep 19 04:18:02.165445 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:02.165358    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/37ab62f0-2d8d-41ce-a442-e5aff9b5232f-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crh5q5\" (UID: \"37ab62f0-2d8d-41ce-a442-e5aff9b5232f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crh5q5"
Sep 19 04:18:02.165445 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:02.165396    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9vcp\" (UniqueName: \"kubernetes.io/projected/37ab62f0-2d8d-41ce-a442-e5aff9b5232f-kube-api-access-c9vcp\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crh5q5\" (UID: \"37ab62f0-2d8d-41ce-a442-e5aff9b5232f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crh5q5"
Sep 19 04:18:02.165656 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:02.165601    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/37ab62f0-2d8d-41ce-a442-e5aff9b5232f-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crh5q5\" (UID: \"37ab62f0-2d8d-41ce-a442-e5aff9b5232f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crh5q5"
Sep 19 04:18:02.165750 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:02.165712    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0dc961b1-3dc6-4efa-aa8c-6c988ca1f46a-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c698cvr\" (UID: \"0dc961b1-3dc6-4efa-aa8c-6c988ca1f46a\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c698cvr"
Sep 19 04:18:02.165813 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:02.165800    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/37ab62f0-2d8d-41ce-a442-e5aff9b5232f-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crh5q5\" (UID: \"37ab62f0-2d8d-41ce-a442-e5aff9b5232f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crh5q5"
Sep 19 04:18:02.165868 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:02.165801    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbfsg\" (UniqueName: \"kubernetes.io/projected/0dc961b1-3dc6-4efa-aa8c-6c988ca1f46a-kube-api-access-bbfsg\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c698cvr\" (UID: \"0dc961b1-3dc6-4efa-aa8c-6c988ca1f46a\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c698cvr"
Sep 19 04:18:02.165930 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:02.165878    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0dc961b1-3dc6-4efa-aa8c-6c988ca1f46a-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c698cvr\" (UID: \"0dc961b1-3dc6-4efa-aa8c-6c988ca1f46a\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c698cvr"
Sep 19 04:18:02.165999 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:02.165978    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/37ab62f0-2d8d-41ce-a442-e5aff9b5232f-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crh5q5\" (UID: \"37ab62f0-2d8d-41ce-a442-e5aff9b5232f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crh5q5"
Sep 19 04:18:02.199216 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:02.199183    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9vcp\" (UniqueName: \"kubernetes.io/projected/37ab62f0-2d8d-41ce-a442-e5aff9b5232f-kube-api-access-c9vcp\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crh5q5\" (UID: \"37ab62f0-2d8d-41ce-a442-e5aff9b5232f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crh5q5"
Sep 19 04:18:02.266466 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:02.266429    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0dc961b1-3dc6-4efa-aa8c-6c988ca1f46a-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c698cvr\" (UID: \"0dc961b1-3dc6-4efa-aa8c-6c988ca1f46a\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c698cvr"
Sep 19 04:18:02.266648 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:02.266482    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbfsg\" (UniqueName: \"kubernetes.io/projected/0dc961b1-3dc6-4efa-aa8c-6c988ca1f46a-kube-api-access-bbfsg\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c698cvr\" (UID: \"0dc961b1-3dc6-4efa-aa8c-6c988ca1f46a\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c698cvr"
Sep 19 04:18:02.266648 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:02.266524    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0dc961b1-3dc6-4efa-aa8c-6c988ca1f46a-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c698cvr\" (UID: \"0dc961b1-3dc6-4efa-aa8c-6c988ca1f46a\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c698cvr"
Sep 19 04:18:02.266966 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:02.266936    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0dc961b1-3dc6-4efa-aa8c-6c988ca1f46a-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c698cvr\" (UID: \"0dc961b1-3dc6-4efa-aa8c-6c988ca1f46a\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c698cvr"
Sep 19 04:18:02.267082 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:02.266963    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0dc961b1-3dc6-4efa-aa8c-6c988ca1f46a-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c698cvr\" (UID: \"0dc961b1-3dc6-4efa-aa8c-6c988ca1f46a\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c698cvr"
Sep 19 04:18:02.275205 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:02.275155    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crh5q5"
Sep 19 04:18:02.277344 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:02.277313    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axczdr"]
Sep 19 04:18:02.280479 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:02.280457    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axczdr"
Sep 19 04:18:02.292222 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:02.292199    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbfsg\" (UniqueName: \"kubernetes.io/projected/0dc961b1-3dc6-4efa-aa8c-6c988ca1f46a-kube-api-access-bbfsg\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c698cvr\" (UID: \"0dc961b1-3dc6-4efa-aa8c-6c988ca1f46a\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c698cvr"
Sep 19 04:18:02.300589 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:02.300539    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axczdr"]
Sep 19 04:18:02.367971 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:02.367881    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k52xx\" (UniqueName: \"kubernetes.io/projected/56969e3d-69c7-44ab-b4b1-21945db80160-kube-api-access-k52xx\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axczdr\" (UID: \"56969e3d-69c7-44ab-b4b1-21945db80160\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axczdr"
Sep 19 04:18:02.367971 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:02.367937    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/56969e3d-69c7-44ab-b4b1-21945db80160-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axczdr\" (UID: \"56969e3d-69c7-44ab-b4b1-21945db80160\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axczdr"
Sep 19 04:18:02.368477 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:02.368009    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/56969e3d-69c7-44ab-b4b1-21945db80160-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axczdr\" (UID: \"56969e3d-69c7-44ab-b4b1-21945db80160\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axczdr"
Sep 19 04:18:02.429810 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:02.429768    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c698cvr"
Sep 19 04:18:02.469317 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:02.469274    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/56969e3d-69c7-44ab-b4b1-21945db80160-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axczdr\" (UID: \"56969e3d-69c7-44ab-b4b1-21945db80160\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axczdr"
Sep 19 04:18:02.469489 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:02.469463    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k52xx\" (UniqueName: \"kubernetes.io/projected/56969e3d-69c7-44ab-b4b1-21945db80160-kube-api-access-k52xx\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axczdr\" (UID: \"56969e3d-69c7-44ab-b4b1-21945db80160\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axczdr"
Sep 19 04:18:02.469558 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:02.469534    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/56969e3d-69c7-44ab-b4b1-21945db80160-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axczdr\" (UID: \"56969e3d-69c7-44ab-b4b1-21945db80160\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axczdr"
Sep 19 04:18:02.469732 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:02.469688    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/56969e3d-69c7-44ab-b4b1-21945db80160-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axczdr\" (UID: \"56969e3d-69c7-44ab-b4b1-21945db80160\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axczdr"
Sep 19 04:18:02.469882 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:02.469862    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/56969e3d-69c7-44ab-b4b1-21945db80160-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axczdr\" (UID: \"56969e3d-69c7-44ab-b4b1-21945db80160\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axczdr"
Sep 19 04:18:02.487069 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:02.487040    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k52xx\" (UniqueName: \"kubernetes.io/projected/56969e3d-69c7-44ab-b4b1-21945db80160-kube-api-access-k52xx\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axczdr\" (UID: \"56969e3d-69c7-44ab-b4b1-21945db80160\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axczdr"
Sep 19 04:18:02.497053 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:02.497020    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tlbx8"]
Sep 19 04:18:02.501133 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:02.501108    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tlbx8"
Sep 19 04:18:02.518815 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:02.518782    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tlbx8"]
Sep 19 04:18:02.570010 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:02.569972    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6523eebc-f036-4db9-ab17-b6e0eeb5cb32-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tlbx8\" (UID: \"6523eebc-f036-4db9-ab17-b6e0eeb5cb32\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tlbx8"
Sep 19 04:18:02.570302 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:02.570034    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45mq8\" (UniqueName: \"kubernetes.io/projected/6523eebc-f036-4db9-ab17-b6e0eeb5cb32-kube-api-access-45mq8\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tlbx8\" (UID: \"6523eebc-f036-4db9-ab17-b6e0eeb5cb32\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tlbx8"
Sep 19 04:18:02.570302 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:02.570064    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6523eebc-f036-4db9-ab17-b6e0eeb5cb32-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tlbx8\" (UID: \"6523eebc-f036-4db9-ab17-b6e0eeb5cb32\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tlbx8"
Sep 19 04:18:02.590368 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:02.590323    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axczdr"
Sep 19 04:18:02.670879 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:02.670844    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6523eebc-f036-4db9-ab17-b6e0eeb5cb32-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tlbx8\" (UID: \"6523eebc-f036-4db9-ab17-b6e0eeb5cb32\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tlbx8"
Sep 19 04:18:02.671078 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:02.670893    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45mq8\" (UniqueName: \"kubernetes.io/projected/6523eebc-f036-4db9-ab17-b6e0eeb5cb32-kube-api-access-45mq8\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tlbx8\" (UID: \"6523eebc-f036-4db9-ab17-b6e0eeb5cb32\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tlbx8"
Sep 19 04:18:02.671078 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:02.670922    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6523eebc-f036-4db9-ab17-b6e0eeb5cb32-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tlbx8\" (UID: \"6523eebc-f036-4db9-ab17-b6e0eeb5cb32\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tlbx8"
Sep 19 04:18:02.671338 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:02.671315    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6523eebc-f036-4db9-ab17-b6e0eeb5cb32-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tlbx8\" (UID: \"6523eebc-f036-4db9-ab17-b6e0eeb5cb32\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tlbx8"
Sep 19 04:18:02.671404 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:02.671322    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6523eebc-f036-4db9-ab17-b6e0eeb5cb32-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tlbx8\" (UID: \"6523eebc-f036-4db9-ab17-b6e0eeb5cb32\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tlbx8"
Sep 19 04:18:02.681842 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:02.681806    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45mq8\" (UniqueName: \"kubernetes.io/projected/6523eebc-f036-4db9-ab17-b6e0eeb5cb32-kube-api-access-45mq8\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tlbx8\" (UID: \"6523eebc-f036-4db9-ab17-b6e0eeb5cb32\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tlbx8"
Sep 19 04:18:02.812403 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:02.812345    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tlbx8"
Sep 19 04:18:03.078811 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:03.078012    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crh5q5"]
Sep 19 04:18:03.080525 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:03.080506    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-6987f5b5b9-jb9kg"]
Sep 19 04:18:03.087784 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:18:03.087758    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0651ba9f_b7df_4e1e_88ba_d8cae0519cd7.slice/crio-3f859429cc66b5ff7f09c72aa066311d566ddd953ebe47337c0e891e69ad0ed7 WatchSource:0}: Error finding container 3f859429cc66b5ff7f09c72aa066311d566ddd953ebe47337c0e891e69ad0ed7: Status 404 returned error can't find the container with id 3f859429cc66b5ff7f09c72aa066311d566ddd953ebe47337c0e891e69ad0ed7
Sep 19 04:18:03.088007 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:18:03.087982    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37ab62f0_2d8d_41ce_a442_e5aff9b5232f.slice/crio-8a2d3356969c155eb760bdd0f1896a2af4726d9f441c38a66cef78b90974d95e WatchSource:0}: Error finding container 8a2d3356969c155eb760bdd0f1896a2af4726d9f441c38a66cef78b90974d95e: Status 404 returned error can't find the container with id 8a2d3356969c155eb760bdd0f1896a2af4726d9f441c38a66cef78b90974d95e
Sep 19 04:18:03.121644 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:03.121586    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c698cvr"]
Sep 19 04:18:03.123425 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:03.123399    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axczdr"]
Sep 19 04:18:03.125028 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:18:03.123868    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2130d608_c9dd_4f97_8348_bb1a10b7fec5.slice/crio-ab85c1a7a1dcce493784fd10adc878f0e5c87da7c8a11282bef008266a472acb WatchSource:0}: Error finding container ab85c1a7a1dcce493784fd10adc878f0e5c87da7c8a11282bef008266a472acb: Status 404 returned error can't find the container with id ab85c1a7a1dcce493784fd10adc878f0e5c87da7c8a11282bef008266a472acb
Sep 19 04:18:03.125215 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:18:03.125055    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56969e3d_69c7_44ab_b4b1_21945db80160.slice/crio-3edc9a66eaf289ccc01e4d5e3e874d5b007e219c2d327c47279201c5ab688cc7 WatchSource:0}: Error finding container 3edc9a66eaf289ccc01e4d5e3e874d5b007e219c2d327c47279201c5ab688cc7: Status 404 returned error can't find the container with id 3edc9a66eaf289ccc01e4d5e3e874d5b007e219c2d327c47279201c5ab688cc7
Sep 19 04:18:03.125215 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:03.125187    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jkrrv"]
Sep 19 04:18:03.126075 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:18:03.125986    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dc961b1_3dc6_4efa_aa8c_6c988ca1f46a.slice/crio-7641c94ff550ce54510ce27652aebcb60032280522bf8e5d743140b8bb48c015 WatchSource:0}: Error finding container 7641c94ff550ce54510ce27652aebcb60032280522bf8e5d743140b8bb48c015: Status 404 returned error can't find the container with id 7641c94ff550ce54510ce27652aebcb60032280522bf8e5d743140b8bb48c015
Sep 19 04:18:03.143956 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:03.143927    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tlbx8"]
Sep 19 04:18:03.148445 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:18:03.148416    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6523eebc_f036_4db9_ab17_b6e0eeb5cb32.slice/crio-a8b623cda07fd99271a0a8b4ec11f78a317c4f69791518a3e0e0cc1bbd91e3b3 WatchSource:0}: Error finding container a8b623cda07fd99271a0a8b4ec11f78a317c4f69791518a3e0e0cc1bbd91e3b3: Status 404 returned error can't find the container with id a8b623cda07fd99271a0a8b4ec11f78a317c4f69791518a3e0e0cc1bbd91e3b3
Sep 19 04:18:03.377197 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:03.377165    2579 generic.go:356] "Generic (PLEG): container finished" podID="56969e3d-69c7-44ab-b4b1-21945db80160" containerID="a4566da9e8b9b04bca4f1e161d28a34e1732aafcd40f1cf852a6c4a09c6306e5" exitCode=0
Sep 19 04:18:03.377599 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:03.377252    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axczdr" event={"ID":"56969e3d-69c7-44ab-b4b1-21945db80160","Type":"ContainerDied","Data":"a4566da9e8b9b04bca4f1e161d28a34e1732aafcd40f1cf852a6c4a09c6306e5"}
Sep 19 04:18:03.377599 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:03.377304    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axczdr" event={"ID":"56969e3d-69c7-44ab-b4b1-21945db80160","Type":"ContainerStarted","Data":"3edc9a66eaf289ccc01e4d5e3e874d5b007e219c2d327c47279201c5ab688cc7"}
Sep 19 04:18:03.378834 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:03.378815    2579 generic.go:356] "Generic (PLEG): container finished" podID="0dc961b1-3dc6-4efa-aa8c-6c988ca1f46a" containerID="0f8c269ee8e58f647dee806b2c1d6451ab09ca101959d4c02aa68a00771f9a8e" exitCode=0
Sep 19 04:18:03.378918 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:03.378896    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c698cvr" event={"ID":"0dc961b1-3dc6-4efa-aa8c-6c988ca1f46a","Type":"ContainerDied","Data":"0f8c269ee8e58f647dee806b2c1d6451ab09ca101959d4c02aa68a00771f9a8e"}
Sep 19 04:18:03.379031 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:03.378941    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c698cvr" event={"ID":"0dc961b1-3dc6-4efa-aa8c-6c988ca1f46a","Type":"ContainerStarted","Data":"7641c94ff550ce54510ce27652aebcb60032280522bf8e5d743140b8bb48c015"}
Sep 19 04:18:03.380503 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:03.380487    2579 generic.go:356] "Generic (PLEG): container finished" podID="37ab62f0-2d8d-41ce-a442-e5aff9b5232f" containerID="0b952272804b7879960f4f7c1483a262ecc40effa7820312aae518246aec31c9" exitCode=0
Sep 19 04:18:03.380616 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:03.380546    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crh5q5" event={"ID":"37ab62f0-2d8d-41ce-a442-e5aff9b5232f","Type":"ContainerDied","Data":"0b952272804b7879960f4f7c1483a262ecc40effa7820312aae518246aec31c9"}
Sep 19 04:18:03.380616 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:03.380583    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crh5q5" event={"ID":"37ab62f0-2d8d-41ce-a442-e5aff9b5232f","Type":"ContainerStarted","Data":"8a2d3356969c155eb760bdd0f1896a2af4726d9f441c38a66cef78b90974d95e"}
Sep 19 04:18:03.381966 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:03.381865    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jkrrv" event={"ID":"2130d608-c9dd-4f97-8348-bb1a10b7fec5","Type":"ContainerStarted","Data":"ab85c1a7a1dcce493784fd10adc878f0e5c87da7c8a11282bef008266a472acb"}
Sep 19 04:18:03.383203 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:03.383168    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6987f5b5b9-jb9kg" event={"ID":"0651ba9f-b7df-4e1e-88ba-d8cae0519cd7","Type":"ContainerStarted","Data":"3f859429cc66b5ff7f09c72aa066311d566ddd953ebe47337c0e891e69ad0ed7"}
Sep 19 04:18:03.384737 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:03.384717    2579 generic.go:356] "Generic (PLEG): container finished" podID="6523eebc-f036-4db9-ab17-b6e0eeb5cb32" containerID="d3019797fa777e02f44ce71f99221ef590ba5d6af81de187621c484dc8c4e90c" exitCode=0
Sep 19 04:18:03.384816 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:03.384786    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tlbx8" event={"ID":"6523eebc-f036-4db9-ab17-b6e0eeb5cb32","Type":"ContainerDied","Data":"d3019797fa777e02f44ce71f99221ef590ba5d6af81de187621c484dc8c4e90c"}
Sep 19 04:18:03.384859 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:03.384818    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tlbx8" event={"ID":"6523eebc-f036-4db9-ab17-b6e0eeb5cb32","Type":"ContainerStarted","Data":"a8b623cda07fd99271a0a8b4ec11f78a317c4f69791518a3e0e0cc1bbd91e3b3"}
Sep 19 04:18:03.386549 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:03.386528    2579 generic.go:356] "Generic (PLEG): container finished" podID="1975d0b2-cda8-41b1-bf10-9b5f3ba155ee" containerID="761d17c2a256eb0cb534ea4fd9aefe322cf26b5a9c2a7a3d7ab6e4c13f8244f5" exitCode=0
Sep 19 04:18:03.386625 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:03.386578    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rwbh2" event={"ID":"1975d0b2-cda8-41b1-bf10-9b5f3ba155ee","Type":"ContainerDied","Data":"761d17c2a256eb0cb534ea4fd9aefe322cf26b5a9c2a7a3d7ab6e4c13f8244f5"}
Sep 19 04:18:05.399245 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:05.399195    2579 generic.go:356] "Generic (PLEG): container finished" podID="56969e3d-69c7-44ab-b4b1-21945db80160" containerID="18d70c05542bf8ef361ed1ac8f8457deebeea0413523e30866b299ee0fd3c0ed" exitCode=0
Sep 19 04:18:05.399245 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:05.399222    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axczdr" event={"ID":"56969e3d-69c7-44ab-b4b1-21945db80160","Type":"ContainerDied","Data":"18d70c05542bf8ef361ed1ac8f8457deebeea0413523e30866b299ee0fd3c0ed"}
Sep 19 04:18:05.401480 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:05.401458    2579 generic.go:356] "Generic (PLEG): container finished" podID="0dc961b1-3dc6-4efa-aa8c-6c988ca1f46a" containerID="caed20643f9946f18da972d0bbe0756fd9342511d411d9c2df7ef8c919945c3c" exitCode=0
Sep 19 04:18:05.401587 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:05.401548    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c698cvr" event={"ID":"0dc961b1-3dc6-4efa-aa8c-6c988ca1f46a","Type":"ContainerDied","Data":"caed20643f9946f18da972d0bbe0756fd9342511d411d9c2df7ef8c919945c3c"}
Sep 19 04:18:05.403792 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:05.403689    2579 generic.go:356] "Generic (PLEG): container finished" podID="37ab62f0-2d8d-41ce-a442-e5aff9b5232f" containerID="e7b55282e6c81206d0016b8b2af03b019eae52f2869a9d38a02e697a0f7053ee" exitCode=0
Sep 19 04:18:05.403792 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:05.403731    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crh5q5" event={"ID":"37ab62f0-2d8d-41ce-a442-e5aff9b5232f","Type":"ContainerDied","Data":"e7b55282e6c81206d0016b8b2af03b019eae52f2869a9d38a02e697a0f7053ee"}
Sep 19 04:18:05.406265 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:05.406234    2579 generic.go:356] "Generic (PLEG): container finished" podID="6523eebc-f036-4db9-ab17-b6e0eeb5cb32" containerID="7cc1017d8bbef126dd40ee0703e0a66ab32a7d67864d39856815ec37262b76b2" exitCode=0
Sep 19 04:18:05.406359 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:05.406280    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tlbx8" event={"ID":"6523eebc-f036-4db9-ab17-b6e0eeb5cb32","Type":"ContainerDied","Data":"7cc1017d8bbef126dd40ee0703e0a66ab32a7d67864d39856815ec37262b76b2"}
Sep 19 04:18:07.568744 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:07.568670    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 04:18:07.569193 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:07.568796    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 04:18:07.569193 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:07.568839    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 04:18:07.603127 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:07.602217    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 04:18:07.603127 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:07.602325    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 04:18:07.603127 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:07.602373    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 04:18:08.427328 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:08.427221    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jkrrv" event={"ID":"2130d608-c9dd-4f97-8348-bb1a10b7fec5","Type":"ContainerStarted","Data":"a1845151cc487d39d5e061bc7b6a9130ea5a4acd6456e9ed6457f094aa539d1b"}
Sep 19 04:18:08.430129 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:08.430027    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6987f5b5b9-jb9kg" event={"ID":"0651ba9f-b7df-4e1e-88ba-d8cae0519cd7","Type":"ContainerStarted","Data":"fae59cbf9724098b6da0d4f15a0b9986b4a89c2ef3a8b253b741dcf5ca867006"}
Sep 19 04:18:08.434134 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:08.433512    2579 generic.go:356] "Generic (PLEG): container finished" podID="1975d0b2-cda8-41b1-bf10-9b5f3ba155ee" containerID="6e44e1f19375b1f58a04c9edddc31255c7e206c13da1b8acd0adfb6820b17076" exitCode=0
Sep 19 04:18:08.434134 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:08.433601    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rwbh2" event={"ID":"1975d0b2-cda8-41b1-bf10-9b5f3ba155ee","Type":"ContainerDied","Data":"6e44e1f19375b1f58a04c9edddc31255c7e206c13da1b8acd0adfb6820b17076"}
Sep 19 04:18:08.454618 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:08.453562    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jkrrv" podStartSLOduration=3.011555066 podStartE2EDuration="7.453541619s" totalImagesPullingTime="4.441986553s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 04:18:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:18:08.452264081 +0000 UTC m=+267.635225232" watchObservedRunningTime="2026-09-19 04:18:08.453541619 +0000 UTC m=+267.636502769"
Sep 19 04:18:08.502753 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:08.502324    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6987f5b5b9-jb9kg" podStartSLOduration=2.99050127 podStartE2EDuration="7.502300668s" totalImagesPullingTime="4.511799398s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 04:18:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:18:08.479104958 +0000 UTC m=+267.662066108" watchObservedRunningTime="2026-09-19 04:18:08.502300668 +0000 UTC m=+267.685261820"
Sep 19 04:18:08.646281 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:08.646246    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jkrrv"
Sep 19 04:18:08.652888 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:08.652778    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jkrrv"
Sep 19 04:18:08.835340 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:08.835105    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6987f5b5b9-jb9kg"
Sep 19 04:18:08.842619 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:08.842592    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6987f5b5b9-jb9kg"
Sep 19 04:18:09.442162 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:09.442056    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6987f5b5b9-jb9kg"
Sep 19 04:18:09.442629 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:09.442409    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jkrrv"
Sep 19 04:18:09.444005 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:09.443748    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jkrrv"
Sep 19 04:18:09.444005 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:09.443920    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6987f5b5b9-jb9kg"
Sep 19 04:18:15.478812 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:15.478779    2579 generic.go:356] "Generic (PLEG): container finished" podID="0dc961b1-3dc6-4efa-aa8c-6c988ca1f46a" containerID="ab0a974df5c6e4136203949e3eddb3863816b39b0608fe35104b93b0c246119c" exitCode=0
Sep 19 04:18:15.479276 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:15.478835    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c698cvr" event={"ID":"0dc961b1-3dc6-4efa-aa8c-6c988ca1f46a","Type":"ContainerDied","Data":"ab0a974df5c6e4136203949e3eddb3863816b39b0608fe35104b93b0c246119c"}
Sep 19 04:18:15.480978 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:15.480956    2579 generic.go:356] "Generic (PLEG): container finished" podID="37ab62f0-2d8d-41ce-a442-e5aff9b5232f" containerID="47777652fa7fbc5e97f42bd39cb320d1756aba8ced6eb6ece7d37fbadc47e563" exitCode=0
Sep 19 04:18:15.481072 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:15.481037    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crh5q5" event={"ID":"37ab62f0-2d8d-41ce-a442-e5aff9b5232f","Type":"ContainerDied","Data":"47777652fa7fbc5e97f42bd39cb320d1756aba8ced6eb6ece7d37fbadc47e563"}
Sep 19 04:18:15.482987 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:15.482968    2579 generic.go:356] "Generic (PLEG): container finished" podID="6523eebc-f036-4db9-ab17-b6e0eeb5cb32" containerID="738d9f53683b809c107e47b6b701f209c70979418b41135ac528b41442a3d4bf" exitCode=0
Sep 19 04:18:15.483100 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:15.483006    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tlbx8" event={"ID":"6523eebc-f036-4db9-ab17-b6e0eeb5cb32","Type":"ContainerDied","Data":"738d9f53683b809c107e47b6b701f209c70979418b41135ac528b41442a3d4bf"}
Sep 19 04:18:15.485068 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:15.485050    2579 generic.go:356] "Generic (PLEG): container finished" podID="1975d0b2-cda8-41b1-bf10-9b5f3ba155ee" containerID="3b8e611210efe892b899b908dd9c5581312a7e851ff3760a5bd217b90ff07b38" exitCode=0
Sep 19 04:18:15.485151 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:15.485126    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rwbh2" event={"ID":"1975d0b2-cda8-41b1-bf10-9b5f3ba155ee","Type":"ContainerDied","Data":"3b8e611210efe892b899b908dd9c5581312a7e851ff3760a5bd217b90ff07b38"}
Sep 19 04:18:15.487100 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:15.487080    2579 generic.go:356] "Generic (PLEG): container finished" podID="56969e3d-69c7-44ab-b4b1-21945db80160" containerID="96ee18edb7883edd5028a5e5c475e3b69d0bd74e1c83ac115808a4773412c953" exitCode=0
Sep 19 04:18:15.487185 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:15.487113    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axczdr" event={"ID":"56969e3d-69c7-44ab-b4b1-21945db80160","Type":"ContainerDied","Data":"96ee18edb7883edd5028a5e5c475e3b69d0bd74e1c83ac115808a4773412c953"}
Sep 19 04:18:16.667576 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:16.667248    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axczdr"
Sep 19 04:18:16.682341 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:16.682312    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tlbx8"
Sep 19 04:18:16.696110 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:16.696074    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rwbh2"
Sep 19 04:18:16.699833 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:16.699373    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6523eebc-f036-4db9-ab17-b6e0eeb5cb32-bundle\") pod \"6523eebc-f036-4db9-ab17-b6e0eeb5cb32\" (UID: \"6523eebc-f036-4db9-ab17-b6e0eeb5cb32\") "
Sep 19 04:18:16.699833 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:16.699440    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-45mq8\" (UniqueName: \"kubernetes.io/projected/6523eebc-f036-4db9-ab17-b6e0eeb5cb32-kube-api-access-45mq8\") pod \"6523eebc-f036-4db9-ab17-b6e0eeb5cb32\" (UID: \"6523eebc-f036-4db9-ab17-b6e0eeb5cb32\") "
Sep 19 04:18:16.699833 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:16.699478    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/56969e3d-69c7-44ab-b4b1-21945db80160-util\") pod \"56969e3d-69c7-44ab-b4b1-21945db80160\" (UID: \"56969e3d-69c7-44ab-b4b1-21945db80160\") "
Sep 19 04:18:16.699833 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:16.699529    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k52xx\" (UniqueName: \"kubernetes.io/projected/56969e3d-69c7-44ab-b4b1-21945db80160-kube-api-access-k52xx\") pod \"56969e3d-69c7-44ab-b4b1-21945db80160\" (UID: \"56969e3d-69c7-44ab-b4b1-21945db80160\") "
Sep 19 04:18:16.699833 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:16.699580    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6523eebc-f036-4db9-ab17-b6e0eeb5cb32-util\") pod \"6523eebc-f036-4db9-ab17-b6e0eeb5cb32\" (UID: \"6523eebc-f036-4db9-ab17-b6e0eeb5cb32\") "
Sep 19 04:18:16.699833 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:16.699613    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/56969e3d-69c7-44ab-b4b1-21945db80160-bundle\") pod \"56969e3d-69c7-44ab-b4b1-21945db80160\" (UID: \"56969e3d-69c7-44ab-b4b1-21945db80160\") "
Sep 19 04:18:16.700643 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:16.700615    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6523eebc-f036-4db9-ab17-b6e0eeb5cb32-bundle" (OuterVolumeSpecName: "bundle") pod "6523eebc-f036-4db9-ab17-b6e0eeb5cb32" (UID: "6523eebc-f036-4db9-ab17-b6e0eeb5cb32"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:18:16.700784 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:16.700737    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56969e3d-69c7-44ab-b4b1-21945db80160-bundle" (OuterVolumeSpecName: "bundle") pod "56969e3d-69c7-44ab-b4b1-21945db80160" (UID: "56969e3d-69c7-44ab-b4b1-21945db80160"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:18:16.702582 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:16.702544    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56969e3d-69c7-44ab-b4b1-21945db80160-kube-api-access-k52xx" (OuterVolumeSpecName: "kube-api-access-k52xx") pod "56969e3d-69c7-44ab-b4b1-21945db80160" (UID: "56969e3d-69c7-44ab-b4b1-21945db80160"). InnerVolumeSpecName "kube-api-access-k52xx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 04:18:16.706126 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:16.706093    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6523eebc-f036-4db9-ab17-b6e0eeb5cb32-kube-api-access-45mq8" (OuterVolumeSpecName: "kube-api-access-45mq8") pod "6523eebc-f036-4db9-ab17-b6e0eeb5cb32" (UID: "6523eebc-f036-4db9-ab17-b6e0eeb5cb32"). InnerVolumeSpecName "kube-api-access-45mq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 04:18:16.707842 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:16.707813    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56969e3d-69c7-44ab-b4b1-21945db80160-util" (OuterVolumeSpecName: "util") pod "56969e3d-69c7-44ab-b4b1-21945db80160" (UID: "56969e3d-69c7-44ab-b4b1-21945db80160"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:18:16.708488 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:16.708460    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6523eebc-f036-4db9-ab17-b6e0eeb5cb32-util" (OuterVolumeSpecName: "util") pod "6523eebc-f036-4db9-ab17-b6e0eeb5cb32" (UID: "6523eebc-f036-4db9-ab17-b6e0eeb5cb32"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:18:16.710067 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:16.710044    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crh5q5"
Sep 19 04:18:16.714054 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:16.714035    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c698cvr"
Sep 19 04:18:16.801039 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:16.801004    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1975d0b2-cda8-41b1-bf10-9b5f3ba155ee-bundle\") pod \"1975d0b2-cda8-41b1-bf10-9b5f3ba155ee\" (UID: \"1975d0b2-cda8-41b1-bf10-9b5f3ba155ee\") "
Sep 19 04:18:16.801039 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:16.801041    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0dc961b1-3dc6-4efa-aa8c-6c988ca1f46a-util\") pod \"0dc961b1-3dc6-4efa-aa8c-6c988ca1f46a\" (UID: \"0dc961b1-3dc6-4efa-aa8c-6c988ca1f46a\") "
Sep 19 04:18:16.801270 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:16.801058    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0dc961b1-3dc6-4efa-aa8c-6c988ca1f46a-bundle\") pod \"0dc961b1-3dc6-4efa-aa8c-6c988ca1f46a\" (UID: \"0dc961b1-3dc6-4efa-aa8c-6c988ca1f46a\") "
Sep 19 04:18:16.801270 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:16.801084    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bbfsg\" (UniqueName: \"kubernetes.io/projected/0dc961b1-3dc6-4efa-aa8c-6c988ca1f46a-kube-api-access-bbfsg\") pod \"0dc961b1-3dc6-4efa-aa8c-6c988ca1f46a\" (UID: \"0dc961b1-3dc6-4efa-aa8c-6c988ca1f46a\") "
Sep 19 04:18:16.801270 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:16.801119    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/37ab62f0-2d8d-41ce-a442-e5aff9b5232f-bundle\") pod \"37ab62f0-2d8d-41ce-a442-e5aff9b5232f\" (UID: \"37ab62f0-2d8d-41ce-a442-e5aff9b5232f\") "
Sep 19 04:18:16.801270 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:16.801144    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1975d0b2-cda8-41b1-bf10-9b5f3ba155ee-util\") pod \"1975d0b2-cda8-41b1-bf10-9b5f3ba155ee\" (UID: \"1975d0b2-cda8-41b1-bf10-9b5f3ba155ee\") "
Sep 19 04:18:16.801270 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:16.801192    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7l5s5\" (UniqueName: \"kubernetes.io/projected/1975d0b2-cda8-41b1-bf10-9b5f3ba155ee-kube-api-access-7l5s5\") pod \"1975d0b2-cda8-41b1-bf10-9b5f3ba155ee\" (UID: \"1975d0b2-cda8-41b1-bf10-9b5f3ba155ee\") "
Sep 19 04:18:16.801270 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:16.801260    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c9vcp\" (UniqueName: \"kubernetes.io/projected/37ab62f0-2d8d-41ce-a442-e5aff9b5232f-kube-api-access-c9vcp\") pod \"37ab62f0-2d8d-41ce-a442-e5aff9b5232f\" (UID: \"37ab62f0-2d8d-41ce-a442-e5aff9b5232f\") "
Sep 19 04:18:16.801559 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:16.801275    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/37ab62f0-2d8d-41ce-a442-e5aff9b5232f-util\") pod \"37ab62f0-2d8d-41ce-a442-e5aff9b5232f\" (UID: \"37ab62f0-2d8d-41ce-a442-e5aff9b5232f\") "
Sep 19 04:18:16.801559 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:16.801468    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k52xx\" (UniqueName: \"kubernetes.io/projected/56969e3d-69c7-44ab-b4b1-21945db80160-kube-api-access-k52xx\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:18:16.801559 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:16.801486    2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6523eebc-f036-4db9-ab17-b6e0eeb5cb32-util\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:18:16.801559 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:16.801499    2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/56969e3d-69c7-44ab-b4b1-21945db80160-bundle\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:18:16.801559 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:16.801512    2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6523eebc-f036-4db9-ab17-b6e0eeb5cb32-bundle\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:18:16.801559 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:16.801524    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45mq8\" (UniqueName: \"kubernetes.io/projected/6523eebc-f036-4db9-ab17-b6e0eeb5cb32-kube-api-access-45mq8\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:18:16.801559 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:16.801536    2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/56969e3d-69c7-44ab-b4b1-21945db80160-util\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:18:16.801906 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:16.801672    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dc961b1-3dc6-4efa-aa8c-6c988ca1f46a-bundle" (OuterVolumeSpecName: "bundle") pod "0dc961b1-3dc6-4efa-aa8c-6c988ca1f46a" (UID: "0dc961b1-3dc6-4efa-aa8c-6c988ca1f46a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:18:16.802075 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:16.802024    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37ab62f0-2d8d-41ce-a442-e5aff9b5232f-bundle" (OuterVolumeSpecName: "bundle") pod "37ab62f0-2d8d-41ce-a442-e5aff9b5232f" (UID: "37ab62f0-2d8d-41ce-a442-e5aff9b5232f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:18:16.803196 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:16.803170    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1975d0b2-cda8-41b1-bf10-9b5f3ba155ee-bundle" (OuterVolumeSpecName: "bundle") pod "1975d0b2-cda8-41b1-bf10-9b5f3ba155ee" (UID: "1975d0b2-cda8-41b1-bf10-9b5f3ba155ee"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:18:16.804149 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:16.804108    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dc961b1-3dc6-4efa-aa8c-6c988ca1f46a-kube-api-access-bbfsg" (OuterVolumeSpecName: "kube-api-access-bbfsg") pod "0dc961b1-3dc6-4efa-aa8c-6c988ca1f46a" (UID: "0dc961b1-3dc6-4efa-aa8c-6c988ca1f46a"). InnerVolumeSpecName "kube-api-access-bbfsg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 04:18:16.804247 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:16.804193    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1975d0b2-cda8-41b1-bf10-9b5f3ba155ee-kube-api-access-7l5s5" (OuterVolumeSpecName: "kube-api-access-7l5s5") pod "1975d0b2-cda8-41b1-bf10-9b5f3ba155ee" (UID: "1975d0b2-cda8-41b1-bf10-9b5f3ba155ee"). InnerVolumeSpecName "kube-api-access-7l5s5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 04:18:16.804305 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:16.804285    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37ab62f0-2d8d-41ce-a442-e5aff9b5232f-kube-api-access-c9vcp" (OuterVolumeSpecName: "kube-api-access-c9vcp") pod "37ab62f0-2d8d-41ce-a442-e5aff9b5232f" (UID: "37ab62f0-2d8d-41ce-a442-e5aff9b5232f"). InnerVolumeSpecName "kube-api-access-c9vcp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 04:18:16.806084 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:16.806062    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1975d0b2-cda8-41b1-bf10-9b5f3ba155ee-util" (OuterVolumeSpecName: "util") pod "1975d0b2-cda8-41b1-bf10-9b5f3ba155ee" (UID: "1975d0b2-cda8-41b1-bf10-9b5f3ba155ee"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:18:16.807521 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:16.807498    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dc961b1-3dc6-4efa-aa8c-6c988ca1f46a-util" (OuterVolumeSpecName: "util") pod "0dc961b1-3dc6-4efa-aa8c-6c988ca1f46a" (UID: "0dc961b1-3dc6-4efa-aa8c-6c988ca1f46a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:18:16.807652 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:16.807634    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37ab62f0-2d8d-41ce-a442-e5aff9b5232f-util" (OuterVolumeSpecName: "util") pod "37ab62f0-2d8d-41ce-a442-e5aff9b5232f" (UID: "37ab62f0-2d8d-41ce-a442-e5aff9b5232f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:18:16.902212 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:16.902175    2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/37ab62f0-2d8d-41ce-a442-e5aff9b5232f-bundle\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:18:16.902212 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:16.902204    2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1975d0b2-cda8-41b1-bf10-9b5f3ba155ee-util\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:18:16.902212 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:16.902213    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7l5s5\" (UniqueName: \"kubernetes.io/projected/1975d0b2-cda8-41b1-bf10-9b5f3ba155ee-kube-api-access-7l5s5\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:18:16.902212 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:16.902223    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9vcp\" (UniqueName: \"kubernetes.io/projected/37ab62f0-2d8d-41ce-a442-e5aff9b5232f-kube-api-access-c9vcp\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:18:16.902472 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:16.902232    2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/37ab62f0-2d8d-41ce-a442-e5aff9b5232f-util\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:18:16.902472 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:16.902242    2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1975d0b2-cda8-41b1-bf10-9b5f3ba155ee-bundle\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:18:16.902472 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:16.902250    2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0dc961b1-3dc6-4efa-aa8c-6c988ca1f46a-util\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:18:16.902472 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:16.902256    2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0dc961b1-3dc6-4efa-aa8c-6c988ca1f46a-bundle\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:18:16.902472 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:16.902264    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbfsg\" (UniqueName: \"kubernetes.io/projected/0dc961b1-3dc6-4efa-aa8c-6c988ca1f46a-kube-api-access-bbfsg\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:18:17.496780 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:17.496737    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crh5q5" event={"ID":"37ab62f0-2d8d-41ce-a442-e5aff9b5232f","Type":"ContainerDied","Data":"8a2d3356969c155eb760bdd0f1896a2af4726d9f441c38a66cef78b90974d95e"}
Sep 19 04:18:17.496780 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:17.496767    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crh5q5"
Sep 19 04:18:17.496990 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:17.496796    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a2d3356969c155eb760bdd0f1896a2af4726d9f441c38a66cef78b90974d95e"
Sep 19 04:18:17.498776 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:17.498739    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tlbx8" event={"ID":"6523eebc-f036-4db9-ab17-b6e0eeb5cb32","Type":"ContainerDied","Data":"a8b623cda07fd99271a0a8b4ec11f78a317c4f69791518a3e0e0cc1bbd91e3b3"}
Sep 19 04:18:17.498776 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:17.498765    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tlbx8"
Sep 19 04:18:17.498962 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:17.498793    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a8b623cda07fd99271a0a8b4ec11f78a317c4f69791518a3e0e0cc1bbd91e3b3"
Sep 19 04:18:17.500890 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:17.500869    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rwbh2" event={"ID":"1975d0b2-cda8-41b1-bf10-9b5f3ba155ee","Type":"ContainerDied","Data":"74e77b02da4f0c767009bd67ebc9a9783114771b13a3046c6ce173c1f8b905b6"}
Sep 19 04:18:17.500979 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:17.500908    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74e77b02da4f0c767009bd67ebc9a9783114771b13a3046c6ce173c1f8b905b6"
Sep 19 04:18:17.500979 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:17.500863    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rwbh2"
Sep 19 04:18:17.502965 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:17.502945    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axczdr" event={"ID":"56969e3d-69c7-44ab-b4b1-21945db80160","Type":"ContainerDied","Data":"3edc9a66eaf289ccc01e4d5e3e874d5b007e219c2d327c47279201c5ab688cc7"}
Sep 19 04:18:17.503040 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:17.502981    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3edc9a66eaf289ccc01e4d5e3e874d5b007e219c2d327c47279201c5ab688cc7"
Sep 19 04:18:17.503082 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:17.503057    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axczdr"
Sep 19 04:18:17.504973 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:17.504955    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c698cvr" event={"ID":"0dc961b1-3dc6-4efa-aa8c-6c988ca1f46a","Type":"ContainerDied","Data":"7641c94ff550ce54510ce27652aebcb60032280522bf8e5d743140b8bb48c015"}
Sep 19 04:18:17.505047 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:17.504987    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c698cvr"
Sep 19 04:18:17.505085 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:17.504988    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7641c94ff550ce54510ce27652aebcb60032280522bf8e5d743140b8bb48c015"
Sep 19 04:18:23.214240 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:23.214200    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-fc5f7b9db-6qm7t"]
Sep 19 04:18:23.214685 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:23.214543    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56969e3d-69c7-44ab-b4b1-21945db80160" containerName="pull"
Sep 19 04:18:23.214685 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:23.214559    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="56969e3d-69c7-44ab-b4b1-21945db80160" containerName="pull"
Sep 19 04:18:23.214685 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:23.214572    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1975d0b2-cda8-41b1-bf10-9b5f3ba155ee" containerName="util"
Sep 19 04:18:23.214685 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:23.214580    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1975d0b2-cda8-41b1-bf10-9b5f3ba155ee" containerName="util"
Sep 19 04:18:23.214685 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:23.214592    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6523eebc-f036-4db9-ab17-b6e0eeb5cb32" containerName="extract"
Sep 19 04:18:23.214685 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:23.214598    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6523eebc-f036-4db9-ab17-b6e0eeb5cb32" containerName="extract"
Sep 19 04:18:23.214685 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:23.214608    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6523eebc-f036-4db9-ab17-b6e0eeb5cb32" containerName="pull"
Sep 19 04:18:23.214685 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:23.214615    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6523eebc-f036-4db9-ab17-b6e0eeb5cb32" containerName="pull"
Sep 19 04:18:23.214685 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:23.214627    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1975d0b2-cda8-41b1-bf10-9b5f3ba155ee" containerName="pull"
Sep 19 04:18:23.214685 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:23.214633    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1975d0b2-cda8-41b1-bf10-9b5f3ba155ee" containerName="pull"
Sep 19 04:18:23.214685 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:23.214640    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56969e3d-69c7-44ab-b4b1-21945db80160" containerName="extract"
Sep 19 04:18:23.214685 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:23.214647    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="56969e3d-69c7-44ab-b4b1-21945db80160" containerName="extract"
Sep 19 04:18:23.214685 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:23.214655    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37ab62f0-2d8d-41ce-a442-e5aff9b5232f" containerName="pull"
Sep 19 04:18:23.214685 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:23.214662    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="37ab62f0-2d8d-41ce-a442-e5aff9b5232f" containerName="pull"
Sep 19 04:18:23.214685 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:23.214674    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37ab62f0-2d8d-41ce-a442-e5aff9b5232f" containerName="util"
Sep 19 04:18:23.214685 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:23.214684    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="37ab62f0-2d8d-41ce-a442-e5aff9b5232f" containerName="util"
Sep 19 04:18:23.214685 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:23.214714    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0dc961b1-3dc6-4efa-aa8c-6c988ca1f46a" containerName="util"
Sep 19 04:18:23.214685 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:23.214722    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0dc961b1-3dc6-4efa-aa8c-6c988ca1f46a" containerName="util"
Sep 19 04:18:23.214685 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:23.214731    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0dc961b1-3dc6-4efa-aa8c-6c988ca1f46a" containerName="pull"
Sep 19 04:18:23.215609 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:23.214739    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0dc961b1-3dc6-4efa-aa8c-6c988ca1f46a" containerName="pull"
Sep 19 04:18:23.215609 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:23.214747    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37ab62f0-2d8d-41ce-a442-e5aff9b5232f" containerName="extract"
Sep 19 04:18:23.215609 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:23.214754    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="37ab62f0-2d8d-41ce-a442-e5aff9b5232f" containerName="extract"
Sep 19 04:18:23.215609 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:23.214793    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56969e3d-69c7-44ab-b4b1-21945db80160" containerName="util"
Sep 19 04:18:23.215609 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:23.214802    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="56969e3d-69c7-44ab-b4b1-21945db80160" containerName="util"
Sep 19 04:18:23.215609 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:23.214810    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6523eebc-f036-4db9-ab17-b6e0eeb5cb32" containerName="util"
Sep 19 04:18:23.215609 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:23.214817    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6523eebc-f036-4db9-ab17-b6e0eeb5cb32" containerName="util"
Sep 19 04:18:23.215609 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:23.214828    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1975d0b2-cda8-41b1-bf10-9b5f3ba155ee" containerName="extract"
Sep 19 04:18:23.215609 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:23.214836    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1975d0b2-cda8-41b1-bf10-9b5f3ba155ee" containerName="extract"
Sep 19 04:18:23.215609 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:23.214846    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0dc961b1-3dc6-4efa-aa8c-6c988ca1f46a" containerName="extract"
Sep 19 04:18:23.215609 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:23.214854    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0dc961b1-3dc6-4efa-aa8c-6c988ca1f46a" containerName="extract"
Sep 19 04:18:23.215609 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:23.214931    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="37ab62f0-2d8d-41ce-a442-e5aff9b5232f" containerName="extract"
Sep 19 04:18:23.215609 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:23.214943    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1975d0b2-cda8-41b1-bf10-9b5f3ba155ee" containerName="extract"
Sep 19 04:18:23.215609 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:23.214954    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6523eebc-f036-4db9-ab17-b6e0eeb5cb32" containerName="extract"
Sep 19 04:18:23.215609 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:23.214962    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="56969e3d-69c7-44ab-b4b1-21945db80160" containerName="extract"
Sep 19 04:18:23.215609 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:23.214971    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0dc961b1-3dc6-4efa-aa8c-6c988ca1f46a" containerName="extract"
Sep 19 04:18:23.267465 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:23.267436    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-fc5f7b9db-6qm7t"]
Sep 19 04:18:23.267610 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:23.267556    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fc5f7b9db-6qm7t"
Sep 19 04:18:23.274455 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:23.274437    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 19 04:18:23.275312 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:23.275295    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 19 04:18:23.275431 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:23.275415    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Sep 19 04:18:23.275496 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:23.275481    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Sep 19 04:18:23.275545 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:23.275521    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Sep 19 04:18:23.275607 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:23.275588    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-6cqv5\""
Sep 19 04:18:23.349605 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:23.349573    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7f60da12-b971-4cd4-a8e3-de671d1f1d37-manifests\") pod \"opendatahub-operator-controller-manager-fc5f7b9db-6qm7t\" (UID: \"7f60da12-b971-4cd4-a8e3-de671d1f1d37\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fc5f7b9db-6qm7t"
Sep 19 04:18:23.349802 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:23.349628    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjnfz\" (UniqueName: \"kubernetes.io/projected/7f60da12-b971-4cd4-a8e3-de671d1f1d37-kube-api-access-pjnfz\") pod \"opendatahub-operator-controller-manager-fc5f7b9db-6qm7t\" (UID: \"7f60da12-b971-4cd4-a8e3-de671d1f1d37\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fc5f7b9db-6qm7t"
Sep 19 04:18:23.349802 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:23.349731    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7f60da12-b971-4cd4-a8e3-de671d1f1d37-apiservice-cert\") pod \"opendatahub-operator-controller-manager-fc5f7b9db-6qm7t\" (UID: \"7f60da12-b971-4cd4-a8e3-de671d1f1d37\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fc5f7b9db-6qm7t"
Sep 19 04:18:23.349802 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:23.349753    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7f60da12-b971-4cd4-a8e3-de671d1f1d37-webhook-cert\") pod \"opendatahub-operator-controller-manager-fc5f7b9db-6qm7t\" (UID: \"7f60da12-b971-4cd4-a8e3-de671d1f1d37\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fc5f7b9db-6qm7t"
Sep 19 04:18:23.349802 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:23.349778    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7f60da12-b971-4cd4-a8e3-de671d1f1d37-tmp\") pod \"opendatahub-operator-controller-manager-fc5f7b9db-6qm7t\" (UID: \"7f60da12-b971-4cd4-a8e3-de671d1f1d37\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fc5f7b9db-6qm7t"
Sep 19 04:18:23.349933 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:23.349814    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7f60da12-b971-4cd4-a8e3-de671d1f1d37-metrics-cert\") pod \"opendatahub-operator-controller-manager-fc5f7b9db-6qm7t\" (UID: \"7f60da12-b971-4cd4-a8e3-de671d1f1d37\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fc5f7b9db-6qm7t"
Sep 19 04:18:23.450411 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:23.450364    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7f60da12-b971-4cd4-a8e3-de671d1f1d37-metrics-cert\") pod \"opendatahub-operator-controller-manager-fc5f7b9db-6qm7t\" (UID: \"7f60da12-b971-4cd4-a8e3-de671d1f1d37\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fc5f7b9db-6qm7t"
Sep 19 04:18:23.450592 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:23.450444    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7f60da12-b971-4cd4-a8e3-de671d1f1d37-manifests\") pod \"opendatahub-operator-controller-manager-fc5f7b9db-6qm7t\" (UID: \"7f60da12-b971-4cd4-a8e3-de671d1f1d37\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fc5f7b9db-6qm7t"
Sep 19 04:18:23.450592 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:23.450464    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjnfz\" (UniqueName: \"kubernetes.io/projected/7f60da12-b971-4cd4-a8e3-de671d1f1d37-kube-api-access-pjnfz\") pod \"opendatahub-operator-controller-manager-fc5f7b9db-6qm7t\" (UID: \"7f60da12-b971-4cd4-a8e3-de671d1f1d37\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fc5f7b9db-6qm7t"
Sep 19 04:18:23.450592 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:23.450488    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7f60da12-b971-4cd4-a8e3-de671d1f1d37-apiservice-cert\") pod \"opendatahub-operator-controller-manager-fc5f7b9db-6qm7t\" (UID: \"7f60da12-b971-4cd4-a8e3-de671d1f1d37\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fc5f7b9db-6qm7t"
Sep 19 04:18:23.450592 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:23.450514    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7f60da12-b971-4cd4-a8e3-de671d1f1d37-webhook-cert\") pod \"opendatahub-operator-controller-manager-fc5f7b9db-6qm7t\" (UID: \"7f60da12-b971-4cd4-a8e3-de671d1f1d37\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fc5f7b9db-6qm7t"
Sep 19 04:18:23.450592 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:23.450543    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7f60da12-b971-4cd4-a8e3-de671d1f1d37-tmp\") pod \"opendatahub-operator-controller-manager-fc5f7b9db-6qm7t\" (UID: \"7f60da12-b971-4cd4-a8e3-de671d1f1d37\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fc5f7b9db-6qm7t"
Sep 19 04:18:23.450979 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:23.450948    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7f60da12-b971-4cd4-a8e3-de671d1f1d37-manifests\") pod \"opendatahub-operator-controller-manager-fc5f7b9db-6qm7t\" (UID: \"7f60da12-b971-4cd4-a8e3-de671d1f1d37\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fc5f7b9db-6qm7t"
Sep 19 04:18:23.453126 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:23.453094    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7f60da12-b971-4cd4-a8e3-de671d1f1d37-tmp\") pod \"opendatahub-operator-controller-manager-fc5f7b9db-6qm7t\" (UID: \"7f60da12-b971-4cd4-a8e3-de671d1f1d37\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fc5f7b9db-6qm7t"
Sep 19 04:18:23.453288 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:23.453264    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7f60da12-b971-4cd4-a8e3-de671d1f1d37-apiservice-cert\") pod \"opendatahub-operator-controller-manager-fc5f7b9db-6qm7t\" (UID: \"7f60da12-b971-4cd4-a8e3-de671d1f1d37\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fc5f7b9db-6qm7t"
Sep 19 04:18:23.453451 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:23.453432    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7f60da12-b971-4cd4-a8e3-de671d1f1d37-metrics-cert\") pod \"opendatahub-operator-controller-manager-fc5f7b9db-6qm7t\" (UID: \"7f60da12-b971-4cd4-a8e3-de671d1f1d37\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fc5f7b9db-6qm7t"
Sep 19 04:18:23.453531 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:23.453512    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7f60da12-b971-4cd4-a8e3-de671d1f1d37-webhook-cert\") pod \"opendatahub-operator-controller-manager-fc5f7b9db-6qm7t\" (UID: \"7f60da12-b971-4cd4-a8e3-de671d1f1d37\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fc5f7b9db-6qm7t"
Sep 19 04:18:23.469799 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:23.469735    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjnfz\" (UniqueName: \"kubernetes.io/projected/7f60da12-b971-4cd4-a8e3-de671d1f1d37-kube-api-access-pjnfz\") pod \"opendatahub-operator-controller-manager-fc5f7b9db-6qm7t\" (UID: \"7f60da12-b971-4cd4-a8e3-de671d1f1d37\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fc5f7b9db-6qm7t"
Sep 19 04:18:23.577457 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:23.577418    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fc5f7b9db-6qm7t"
Sep 19 04:18:23.879725 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:23.879684    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-fc5f7b9db-6qm7t"]
Sep 19 04:18:23.880805 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:18:23.880773    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f60da12_b971_4cd4_a8e3_de671d1f1d37.slice/crio-49e0467ff3a598bbb6c48ead83083582d478e3803a11c53ffc2d40a371a163ed WatchSource:0}: Error finding container 49e0467ff3a598bbb6c48ead83083582d478e3803a11c53ffc2d40a371a163ed: Status 404 returned error can't find the container with id 49e0467ff3a598bbb6c48ead83083582d478e3803a11c53ffc2d40a371a163ed
Sep 19 04:18:24.536822 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:24.536777    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fc5f7b9db-6qm7t" event={"ID":"7f60da12-b971-4cd4-a8e3-de671d1f1d37","Type":"ContainerStarted","Data":"49e0467ff3a598bbb6c48ead83083582d478e3803a11c53ffc2d40a371a163ed"}
Sep 19 04:18:27.553779 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:27.553655    2579 generic.go:356] "Generic (PLEG): container finished" podID="7f60da12-b971-4cd4-a8e3-de671d1f1d37" containerID="760a2a20ca9bf0f0c42bbfc0177b07676452e471ec4f8a27dd87f1f9e6573664" exitCode=0
Sep 19 04:18:27.554258 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:27.553802    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fc5f7b9db-6qm7t" event={"ID":"7f60da12-b971-4cd4-a8e3-de671d1f1d37","Type":"ContainerDied","Data":"760a2a20ca9bf0f0c42bbfc0177b07676452e471ec4f8a27dd87f1f9e6573664"}
Sep 19 04:18:28.560139 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:28.560103    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fc5f7b9db-6qm7t" event={"ID":"7f60da12-b971-4cd4-a8e3-de671d1f1d37","Type":"ContainerStarted","Data":"e6ba410d2a2a6bc338ca1aed9dce5686a62991913364defdf1f16077790195cc"}
Sep 19 04:18:28.560557 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:28.560224    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fc5f7b9db-6qm7t"
Sep 19 04:18:28.602400 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:28.602352    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fc5f7b9db-6qm7t" podStartSLOduration=1.974524671 podStartE2EDuration="5.602338782s" totalImagesPullingTime="3.223644635s" totalInitContainerRuntime="404.169476ms" isStatefulPod=false podCreationTimestamp="2026-09-19 04:18:23 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:18:28.601131785 +0000 UTC m=+287.784092934" watchObservedRunningTime="2026-09-19 04:18:28.602338782 +0000 UTC m=+287.785299929"
Sep 19 04:18:28.644174 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:28.644146    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6d9b9f85cb-627xv"]
Sep 19 04:18:28.647362 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:28.647345    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6d9b9f85cb-627xv"
Sep 19 04:18:28.652115 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:28.652076    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 19 04:18:28.652244 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:28.652220    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-fdkr2\""
Sep 19 04:18:28.655986 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:28.655965    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 19 04:18:28.695612 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:28.695584    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnrh7\" (UniqueName: \"kubernetes.io/projected/a64e50da-98d3-4659-ab01-486ba441519a-kube-api-access-mnrh7\") pod \"limitador-operator-controller-manager-6d9b9f85cb-627xv\" (UID: \"a64e50da-98d3-4659-ab01-486ba441519a\") " pod="kuadrant-system/limitador-operator-controller-manager-6d9b9f85cb-627xv"
Sep 19 04:18:28.706427 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:28.706403    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6d9b9f85cb-627xv"]
Sep 19 04:18:28.796649 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:28.796612    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnrh7\" (UniqueName: \"kubernetes.io/projected/a64e50da-98d3-4659-ab01-486ba441519a-kube-api-access-mnrh7\") pod \"limitador-operator-controller-manager-6d9b9f85cb-627xv\" (UID: \"a64e50da-98d3-4659-ab01-486ba441519a\") " pod="kuadrant-system/limitador-operator-controller-manager-6d9b9f85cb-627xv"
Sep 19 04:18:28.819125 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:28.819065    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnrh7\" (UniqueName: \"kubernetes.io/projected/a64e50da-98d3-4659-ab01-486ba441519a-kube-api-access-mnrh7\") pod \"limitador-operator-controller-manager-6d9b9f85cb-627xv\" (UID: \"a64e50da-98d3-4659-ab01-486ba441519a\") " pod="kuadrant-system/limitador-operator-controller-manager-6d9b9f85cb-627xv"
Sep 19 04:18:28.956996 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:28.956962    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6d9b9f85cb-627xv"
Sep 19 04:18:29.264055 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:29.264029    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6d9b9f85cb-627xv"]
Sep 19 04:18:29.268049 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:18:29.268021    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda64e50da_98d3_4659_ab01_486ba441519a.slice/crio-5d261c768bd133fcee6cd180302889c4fbcfebf0a9445b996442163d6bdd39ff WatchSource:0}: Error finding container 5d261c768bd133fcee6cd180302889c4fbcfebf0a9445b996442163d6bdd39ff: Status 404 returned error can't find the container with id 5d261c768bd133fcee6cd180302889c4fbcfebf0a9445b996442163d6bdd39ff
Sep 19 04:18:29.565021 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:29.564938    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6d9b9f85cb-627xv" event={"ID":"a64e50da-98d3-4659-ab01-486ba441519a","Type":"ContainerStarted","Data":"5d261c768bd133fcee6cd180302889c4fbcfebf0a9445b996442163d6bdd39ff"}
Sep 19 04:18:31.575960 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:31.575924    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6d9b9f85cb-627xv" event={"ID":"a64e50da-98d3-4659-ab01-486ba441519a","Type":"ContainerStarted","Data":"f68e6d5ea4cbdde5c1150a5cf9abca515f58ae38bddd21e079b222afc802fa58"}
Sep 19 04:18:31.576401 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:31.576064    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-6d9b9f85cb-627xv"
Sep 19 04:18:31.643509 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:31.643463    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-6d9b9f85cb-627xv" podStartSLOduration=2.193104295 podStartE2EDuration="3.643449288s" totalImagesPullingTime="1.450344993s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 04:18:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:18:31.640751961 +0000 UTC m=+290.823713103" watchObservedRunningTime="2026-09-19 04:18:31.643449288 +0000 UTC m=+290.826410436"
Sep 19 04:18:33.461042 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:33.461008    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b449b7679-jxlq7"]
Sep 19 04:18:33.466006 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:33.465988    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b449b7679-jxlq7"
Sep 19 04:18:33.470126 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:33.470106    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-q2ng2\""
Sep 19 04:18:33.470854 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:33.470838    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 19 04:18:33.487798 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:33.487770    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b449b7679-jxlq7"]
Sep 19 04:18:33.534182 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:33.534152    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4bd6\" (UniqueName: \"kubernetes.io/projected/557f44c8-ad27-4fda-aa1f-2c04cda00c33-kube-api-access-v4bd6\") pod \"dns-operator-controller-manager-7b449b7679-jxlq7\" (UID: \"557f44c8-ad27-4fda-aa1f-2c04cda00c33\") " pod="kuadrant-system/dns-operator-controller-manager-7b449b7679-jxlq7"
Sep 19 04:18:33.634809 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:33.634768    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4bd6\" (UniqueName: \"kubernetes.io/projected/557f44c8-ad27-4fda-aa1f-2c04cda00c33-kube-api-access-v4bd6\") pod \"dns-operator-controller-manager-7b449b7679-jxlq7\" (UID: \"557f44c8-ad27-4fda-aa1f-2c04cda00c33\") " pod="kuadrant-system/dns-operator-controller-manager-7b449b7679-jxlq7"
Sep 19 04:18:33.649467 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:33.649441    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4bd6\" (UniqueName: \"kubernetes.io/projected/557f44c8-ad27-4fda-aa1f-2c04cda00c33-kube-api-access-v4bd6\") pod \"dns-operator-controller-manager-7b449b7679-jxlq7\" (UID: \"557f44c8-ad27-4fda-aa1f-2c04cda00c33\") " pod="kuadrant-system/dns-operator-controller-manager-7b449b7679-jxlq7"
Sep 19 04:18:33.775965 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:33.775883    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b449b7679-jxlq7"
Sep 19 04:18:34.069175 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:34.069151    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b449b7679-jxlq7"]
Sep 19 04:18:34.070330 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:18:34.070303    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod557f44c8_ad27_4fda_aa1f_2c04cda00c33.slice/crio-fa63dcbfc2a8539321623f059e1dd0921e335e509adbd978755941729b5a8060 WatchSource:0}: Error finding container fa63dcbfc2a8539321623f059e1dd0921e335e509adbd978755941729b5a8060: Status 404 returned error can't find the container with id fa63dcbfc2a8539321623f059e1dd0921e335e509adbd978755941729b5a8060
Sep 19 04:18:34.590257 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:34.590216    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b449b7679-jxlq7" event={"ID":"557f44c8-ad27-4fda-aa1f-2c04cda00c33","Type":"ContainerStarted","Data":"fa63dcbfc2a8539321623f059e1dd0921e335e509adbd978755941729b5a8060"}
Sep 19 04:18:36.601981 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:36.601945    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b449b7679-jxlq7" event={"ID":"557f44c8-ad27-4fda-aa1f-2c04cda00c33","Type":"ContainerStarted","Data":"9547fcd66460fccd133850a440d06c56b637bb1dc8db02e995e66443915cff9f"}
Sep 19 04:18:36.602446 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:36.602088    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b449b7679-jxlq7"
Sep 19 04:18:36.638147 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:36.638096    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b449b7679-jxlq7" podStartSLOduration=1.6742962559999999 podStartE2EDuration="3.638079687s" totalImagesPullingTime="1.963783431s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 04:18:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:18:36.636577297 +0000 UTC m=+295.819538446" watchObservedRunningTime="2026-09-19 04:18:36.638079687 +0000 UTC m=+295.821040836"
Sep 19 04:18:38.205342 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:38.205309    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-s5cv9"]
Sep 19 04:18:38.208595 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:38.208578    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-s5cv9"
Sep 19 04:18:38.212465 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:38.212444    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 19 04:18:38.212593 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:38.212574    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-65mdw\""
Sep 19 04:18:38.214471 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:38.214449    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 19 04:18:38.271961 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:38.271929    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a57b2a71-2e67-49cd-9a11-95445fe25300-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-s5cv9\" (UID: \"a57b2a71-2e67-49cd-9a11-95445fe25300\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-s5cv9"
Sep 19 04:18:38.272122 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:38.271980    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtwhs\" (UniqueName: \"kubernetes.io/projected/a57b2a71-2e67-49cd-9a11-95445fe25300-kube-api-access-rtwhs\") pod \"cert-manager-operator-controller-manager-684f79c9b9-s5cv9\" (UID: \"a57b2a71-2e67-49cd-9a11-95445fe25300\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-s5cv9"
Sep 19 04:18:38.318626 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:38.318595    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-s5cv9"]
Sep 19 04:18:38.372554 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:38.372518    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a57b2a71-2e67-49cd-9a11-95445fe25300-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-s5cv9\" (UID: \"a57b2a71-2e67-49cd-9a11-95445fe25300\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-s5cv9"
Sep 19 04:18:38.372749 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:38.372570    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtwhs\" (UniqueName: \"kubernetes.io/projected/a57b2a71-2e67-49cd-9a11-95445fe25300-kube-api-access-rtwhs\") pod \"cert-manager-operator-controller-manager-684f79c9b9-s5cv9\" (UID: \"a57b2a71-2e67-49cd-9a11-95445fe25300\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-s5cv9"
Sep 19 04:18:38.372984 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:38.372967    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a57b2a71-2e67-49cd-9a11-95445fe25300-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-s5cv9\" (UID: \"a57b2a71-2e67-49cd-9a11-95445fe25300\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-s5cv9"
Sep 19 04:18:38.383877 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:38.383848    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtwhs\" (UniqueName: \"kubernetes.io/projected/a57b2a71-2e67-49cd-9a11-95445fe25300-kube-api-access-rtwhs\") pod \"cert-manager-operator-controller-manager-684f79c9b9-s5cv9\" (UID: \"a57b2a71-2e67-49cd-9a11-95445fe25300\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-s5cv9"
Sep 19 04:18:38.517769 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:38.517679    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-s5cv9"
Sep 19 04:18:38.812131 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:38.812109    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-s5cv9"]
Sep 19 04:18:38.813981 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:18:38.813956    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda57b2a71_2e67_49cd_9a11_95445fe25300.slice/crio-21bd62654277cc4f5c54c7b01be1421f6b6570df6ee8bd09d8582e413c06f326 WatchSource:0}: Error finding container 21bd62654277cc4f5c54c7b01be1421f6b6570df6ee8bd09d8582e413c06f326: Status 404 returned error can't find the container with id 21bd62654277cc4f5c54c7b01be1421f6b6570df6ee8bd09d8582e413c06f326
Sep 19 04:18:39.567173 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:39.567144    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fc5f7b9db-6qm7t"
Sep 19 04:18:39.616159 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:39.616125    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-s5cv9" event={"ID":"a57b2a71-2e67-49cd-9a11-95445fe25300","Type":"ContainerStarted","Data":"21bd62654277cc4f5c54c7b01be1421f6b6570df6ee8bd09d8582e413c06f326"}
Sep 19 04:18:41.241774 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:41.241736    2579 kubelet.go:1648] "Image garbage collection succeeded"
Sep 19 04:18:41.376796 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:41.376614    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-p62fr"]
Sep 19 04:18:41.392604 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:41.381485    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-p62fr"
Sep 19 04:18:41.392604 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:41.384380    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-sffsr\""
Sep 19 04:18:41.395808 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:41.395783    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-p62fr"]
Sep 19 04:18:41.499414 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:41.499335    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/10734032-f90d-4666-8d77-0573f1cce689-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-54b6d5c989-p62fr\" (UID: \"10734032-f90d-4666-8d77-0573f1cce689\") " pod="kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-p62fr"
Sep 19 04:18:41.499414 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:41.499384    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tm7s\" (UniqueName: \"kubernetes.io/projected/10734032-f90d-4666-8d77-0573f1cce689-kube-api-access-5tm7s\") pod \"kuadrant-operator-controller-manager-54b6d5c989-p62fr\" (UID: \"10734032-f90d-4666-8d77-0573f1cce689\") " pod="kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-p62fr"
Sep 19 04:18:41.600523 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:41.600490    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/10734032-f90d-4666-8d77-0573f1cce689-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-54b6d5c989-p62fr\" (UID: \"10734032-f90d-4666-8d77-0573f1cce689\") " pod="kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-p62fr"
Sep 19 04:18:41.600731 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:41.600539    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tm7s\" (UniqueName: \"kubernetes.io/projected/10734032-f90d-4666-8d77-0573f1cce689-kube-api-access-5tm7s\") pod \"kuadrant-operator-controller-manager-54b6d5c989-p62fr\" (UID: \"10734032-f90d-4666-8d77-0573f1cce689\") " pod="kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-p62fr"
Sep 19 04:18:41.601036 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:41.601002    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/10734032-f90d-4666-8d77-0573f1cce689-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-54b6d5c989-p62fr\" (UID: \"10734032-f90d-4666-8d77-0573f1cce689\") " pod="kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-p62fr"
Sep 19 04:18:41.627297 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:41.627261    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-s5cv9" event={"ID":"a57b2a71-2e67-49cd-9a11-95445fe25300","Type":"ContainerStarted","Data":"66bd874a282a2a4a6868331889018b3f496016c1d4b507e80bbf4688ce92b392"}
Sep 19 04:18:41.628802 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:41.628779    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tm7s\" (UniqueName: \"kubernetes.io/projected/10734032-f90d-4666-8d77-0573f1cce689-kube-api-access-5tm7s\") pod \"kuadrant-operator-controller-manager-54b6d5c989-p62fr\" (UID: \"10734032-f90d-4666-8d77-0573f1cce689\") " pod="kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-p62fr"
Sep 19 04:18:41.692626 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:41.692592    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-p62fr"
Sep 19 04:18:41.703407 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:41.703351    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-s5cv9" podStartSLOduration=1.900009878 podStartE2EDuration="3.703330565s" totalImagesPullingTime="1.803320687s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 04:18:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:18:41.701509166 +0000 UTC m=+300.884470326" watchObservedRunningTime="2026-09-19 04:18:41.703330565 +0000 UTC m=+300.886291714"
Sep 19 04:18:42.027978 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:18:42.027953    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10734032_f90d_4666_8d77_0573f1cce689.slice/crio-ca668ab271e921b8a59f6363592e8b446872bf9b962065ec87a5bb16e4bd791a WatchSource:0}: Error finding container ca668ab271e921b8a59f6363592e8b446872bf9b962065ec87a5bb16e4bd791a: Status 404 returned error can't find the container with id ca668ab271e921b8a59f6363592e8b446872bf9b962065ec87a5bb16e4bd791a
Sep 19 04:18:42.033625 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:42.033598    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-p62fr"]
Sep 19 04:18:42.582443 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:42.582403    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-6d9b9f85cb-627xv"
Sep 19 04:18:42.635873 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:42.635834    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-p62fr" event={"ID":"10734032-f90d-4666-8d77-0573f1cce689","Type":"ContainerStarted","Data":"ca668ab271e921b8a59f6363592e8b446872bf9b962065ec87a5bb16e4bd791a"}
Sep 19 04:18:46.661011 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:46.660972    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-p62fr" event={"ID":"10734032-f90d-4666-8d77-0573f1cce689","Type":"ContainerStarted","Data":"c57daf08ef7c72e9a974a9f2c46ed5c45cb7f04dd41db2bcd7f33627216b51e5"}
Sep 19 04:18:46.661399 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:46.661098    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-p62fr"
Sep 19 04:18:46.725042 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:46.724981    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-p62fr" podStartSLOduration=2.118193823 podStartE2EDuration="5.724963991s" totalImagesPullingTime="3.606770168s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 04:18:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:18:46.724858755 +0000 UTC m=+305.907819902" watchObservedRunningTime="2026-09-19 04:18:46.724963991 +0000 UTC m=+305.907925139"
Sep 19 04:18:47.284229 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:47.284194    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-6c7cb5c9cb-g98cq"]
Sep 19 04:18:47.306375 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:47.306352    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-6c7cb5c9cb-g98cq"
Sep 19 04:18:47.318367 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:47.318341    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-xddr8\""
Sep 19 04:18:47.319208 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:47.319185    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-6c7cb5c9cb-g98cq"]
Sep 19 04:18:47.448905 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:47.448870    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqc2z\" (UniqueName: \"kubernetes.io/projected/79e96918-0b4e-4940-bb71-ccdfb8b118e2-kube-api-access-vqc2z\") pod \"authorino-operator-6c7cb5c9cb-g98cq\" (UID: \"79e96918-0b4e-4940-bb71-ccdfb8b118e2\") " pod="kuadrant-system/authorino-operator-6c7cb5c9cb-g98cq"
Sep 19 04:18:47.549883 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:47.549803    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqc2z\" (UniqueName: \"kubernetes.io/projected/79e96918-0b4e-4940-bb71-ccdfb8b118e2-kube-api-access-vqc2z\") pod \"authorino-operator-6c7cb5c9cb-g98cq\" (UID: \"79e96918-0b4e-4940-bb71-ccdfb8b118e2\") " pod="kuadrant-system/authorino-operator-6c7cb5c9cb-g98cq"
Sep 19 04:18:47.570362 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:47.570335    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqc2z\" (UniqueName: \"kubernetes.io/projected/79e96918-0b4e-4940-bb71-ccdfb8b118e2-kube-api-access-vqc2z\") pod \"authorino-operator-6c7cb5c9cb-g98cq\" (UID: \"79e96918-0b4e-4940-bb71-ccdfb8b118e2\") " pod="kuadrant-system/authorino-operator-6c7cb5c9cb-g98cq"
Sep 19 04:18:47.607898 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:47.607868    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b449b7679-jxlq7"
Sep 19 04:18:47.615378 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:47.615354    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-6c7cb5c9cb-g98cq"
Sep 19 04:18:47.908010 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:47.907977    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-6c7cb5c9cb-g98cq"]
Sep 19 04:18:47.911620 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:18:47.911585    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79e96918_0b4e_4940_bb71_ccdfb8b118e2.slice/crio-5a264f810d4a146329f7d6dd05e26366ff5623d9dd8e65b67bc35e287c2f0a07 WatchSource:0}: Error finding container 5a264f810d4a146329f7d6dd05e26366ff5623d9dd8e65b67bc35e287c2f0a07: Status 404 returned error can't find the container with id 5a264f810d4a146329f7d6dd05e26366ff5623d9dd8e65b67bc35e287c2f0a07
Sep 19 04:18:47.914025 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:47.914009    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 19 04:18:48.677654 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:48.677614    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-6c7cb5c9cb-g98cq" event={"ID":"79e96918-0b4e-4940-bb71-ccdfb8b118e2","Type":"ContainerStarted","Data":"5a264f810d4a146329f7d6dd05e26366ff5623d9dd8e65b67bc35e287c2f0a07"}
Sep 19 04:18:54.166913 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:54.166880    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cgg7jw"]
Sep 19 04:18:54.171122 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:54.171101    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cgg7jw"
Sep 19 04:18:54.175369 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:54.175350    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-5r5k9\""
Sep 19 04:18:54.206975 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:54.206948    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cgg7jw"]
Sep 19 04:18:54.303798 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:54.303765    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/382f9c83-8f9d-4e1d-a6d8-a2de02a9fc3c-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cgg7jw\" (UID: \"382f9c83-8f9d-4e1d-a6d8-a2de02a9fc3c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cgg7jw"
Sep 19 04:18:54.303798 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:54.303801    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/382f9c83-8f9d-4e1d-a6d8-a2de02a9fc3c-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cgg7jw\" (UID: \"382f9c83-8f9d-4e1d-a6d8-a2de02a9fc3c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cgg7jw"
Sep 19 04:18:54.304013 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:54.303825    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/382f9c83-8f9d-4e1d-a6d8-a2de02a9fc3c-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cgg7jw\" (UID: \"382f9c83-8f9d-4e1d-a6d8-a2de02a9fc3c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cgg7jw"
Sep 19 04:18:54.304013 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:54.303842    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/382f9c83-8f9d-4e1d-a6d8-a2de02a9fc3c-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cgg7jw\" (UID: \"382f9c83-8f9d-4e1d-a6d8-a2de02a9fc3c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cgg7jw"
Sep 19 04:18:54.304013 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:54.303906    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/382f9c83-8f9d-4e1d-a6d8-a2de02a9fc3c-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cgg7jw\" (UID: \"382f9c83-8f9d-4e1d-a6d8-a2de02a9fc3c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cgg7jw"
Sep 19 04:18:54.304013 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:54.303942    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/382f9c83-8f9d-4e1d-a6d8-a2de02a9fc3c-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cgg7jw\" (UID: \"382f9c83-8f9d-4e1d-a6d8-a2de02a9fc3c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cgg7jw"
Sep 19 04:18:54.304013 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:54.304007    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/382f9c83-8f9d-4e1d-a6d8-a2de02a9fc3c-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cgg7jw\" (UID: \"382f9c83-8f9d-4e1d-a6d8-a2de02a9fc3c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cgg7jw"
Sep 19 04:18:54.304171 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:54.304028    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/382f9c83-8f9d-4e1d-a6d8-a2de02a9fc3c-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cgg7jw\" (UID: \"382f9c83-8f9d-4e1d-a6d8-a2de02a9fc3c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cgg7jw"
Sep 19 04:18:54.304171 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:54.304048    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xqt5\" (UniqueName: \"kubernetes.io/projected/382f9c83-8f9d-4e1d-a6d8-a2de02a9fc3c-kube-api-access-2xqt5\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cgg7jw\" (UID: \"382f9c83-8f9d-4e1d-a6d8-a2de02a9fc3c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cgg7jw"
Sep 19 04:18:54.405243 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:54.405210    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/382f9c83-8f9d-4e1d-a6d8-a2de02a9fc3c-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cgg7jw\" (UID: \"382f9c83-8f9d-4e1d-a6d8-a2de02a9fc3c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cgg7jw"
Sep 19 04:18:54.405243 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:54.405246    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/382f9c83-8f9d-4e1d-a6d8-a2de02a9fc3c-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cgg7jw\" (UID: \"382f9c83-8f9d-4e1d-a6d8-a2de02a9fc3c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cgg7jw"
Sep 19 04:18:54.405443 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:54.405373    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/382f9c83-8f9d-4e1d-a6d8-a2de02a9fc3c-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cgg7jw\" (UID: \"382f9c83-8f9d-4e1d-a6d8-a2de02a9fc3c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cgg7jw"
Sep 19 04:18:54.405443 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:54.405412    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/382f9c83-8f9d-4e1d-a6d8-a2de02a9fc3c-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cgg7jw\" (UID: \"382f9c83-8f9d-4e1d-a6d8-a2de02a9fc3c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cgg7jw"
Sep 19 04:18:54.405540 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:54.405483    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/382f9c83-8f9d-4e1d-a6d8-a2de02a9fc3c-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cgg7jw\" (UID: \"382f9c83-8f9d-4e1d-a6d8-a2de02a9fc3c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cgg7jw"
Sep 19 04:18:54.405540 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:54.405511    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/382f9c83-8f9d-4e1d-a6d8-a2de02a9fc3c-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cgg7jw\" (UID: \"382f9c83-8f9d-4e1d-a6d8-a2de02a9fc3c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cgg7jw"
Sep 19 04:18:54.405644 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:54.405538    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xqt5\" (UniqueName: \"kubernetes.io/projected/382f9c83-8f9d-4e1d-a6d8-a2de02a9fc3c-kube-api-access-2xqt5\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cgg7jw\" (UID: \"382f9c83-8f9d-4e1d-a6d8-a2de02a9fc3c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cgg7jw"
Sep 19 04:18:54.405644 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:54.405577    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/382f9c83-8f9d-4e1d-a6d8-a2de02a9fc3c-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cgg7jw\" (UID: \"382f9c83-8f9d-4e1d-a6d8-a2de02a9fc3c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cgg7jw"
Sep 19 04:18:54.405644 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:54.405602    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/382f9c83-8f9d-4e1d-a6d8-a2de02a9fc3c-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cgg7jw\" (UID: \"382f9c83-8f9d-4e1d-a6d8-a2de02a9fc3c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cgg7jw"
Sep 19 04:18:54.405644 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:54.405633    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/382f9c83-8f9d-4e1d-a6d8-a2de02a9fc3c-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cgg7jw\" (UID: \"382f9c83-8f9d-4e1d-a6d8-a2de02a9fc3c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cgg7jw"
Sep 19 04:18:54.405874 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:54.405763    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/382f9c83-8f9d-4e1d-a6d8-a2de02a9fc3c-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cgg7jw\" (UID: \"382f9c83-8f9d-4e1d-a6d8-a2de02a9fc3c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cgg7jw"
Sep 19 04:18:54.405939 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:54.405906    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/382f9c83-8f9d-4e1d-a6d8-a2de02a9fc3c-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cgg7jw\" (UID: \"382f9c83-8f9d-4e1d-a6d8-a2de02a9fc3c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cgg7jw"
Sep 19 04:18:54.405989 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:54.405960    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/382f9c83-8f9d-4e1d-a6d8-a2de02a9fc3c-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cgg7jw\" (UID: \"382f9c83-8f9d-4e1d-a6d8-a2de02a9fc3c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cgg7jw"
Sep 19 04:18:54.406081 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:54.406058    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/382f9c83-8f9d-4e1d-a6d8-a2de02a9fc3c-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cgg7jw\" (UID: \"382f9c83-8f9d-4e1d-a6d8-a2de02a9fc3c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cgg7jw"
Sep 19 04:18:54.407824 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:54.407805    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/382f9c83-8f9d-4e1d-a6d8-a2de02a9fc3c-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cgg7jw\" (UID: \"382f9c83-8f9d-4e1d-a6d8-a2de02a9fc3c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cgg7jw"
Sep 19 04:18:54.408201 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:54.408184    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/382f9c83-8f9d-4e1d-a6d8-a2de02a9fc3c-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cgg7jw\" (UID: \"382f9c83-8f9d-4e1d-a6d8-a2de02a9fc3c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cgg7jw"
Sep 19 04:18:54.424676 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:54.424615    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xqt5\" (UniqueName: \"kubernetes.io/projected/382f9c83-8f9d-4e1d-a6d8-a2de02a9fc3c-kube-api-access-2xqt5\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cgg7jw\" (UID: \"382f9c83-8f9d-4e1d-a6d8-a2de02a9fc3c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cgg7jw"
Sep 19 04:18:54.426330 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:54.426303    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/382f9c83-8f9d-4e1d-a6d8-a2de02a9fc3c-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cgg7jw\" (UID: \"382f9c83-8f9d-4e1d-a6d8-a2de02a9fc3c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cgg7jw"
Sep 19 04:18:54.481599 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:54.481566    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cgg7jw"
Sep 19 04:18:54.773688 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:54.773662    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cgg7jw"]
Sep 19 04:18:54.775450 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:18:54.775411    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod382f9c83_8f9d_4e1d_a6d8_a2de02a9fc3c.slice/crio-72c058357780236946454dfb6bd4ad4b4998260ac8595c3e71bba65bf75816ba WatchSource:0}: Error finding container 72c058357780236946454dfb6bd4ad4b4998260ac8595c3e71bba65bf75816ba: Status 404 returned error can't find the container with id 72c058357780236946454dfb6bd4ad4b4998260ac8595c3e71bba65bf75816ba
Sep 19 04:18:54.777592 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:54.777561    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 04:18:54.777654 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:54.777626    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 04:18:54.777712 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:54.777655    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 04:18:55.290803 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:55.290772    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-54765d59fd-4ddzl"]
Sep 19 04:18:55.293815 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:55.293799    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-54765d59fd-4ddzl"
Sep 19 04:18:55.299520 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:55.299496    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 19 04:18:55.299520 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:55.299513    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 19 04:18:55.300387 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:55.300370    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-mzppr\""
Sep 19 04:18:55.309537 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:55.309518    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\""
Sep 19 04:18:55.320830 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:55.320805    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-54765d59fd-4ddzl"]
Sep 19 04:18:55.412494 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:55.412467    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/06647565-1498-4fe1-a34d-f31450f84ee4-manifests\") pod \"kserve-module-controller-manager-54765d59fd-4ddzl\" (UID: \"06647565-1498-4fe1-a34d-f31450f84ee4\") " pod="opendatahub/kserve-module-controller-manager-54765d59fd-4ddzl"
Sep 19 04:18:55.412636 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:55.412503    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxhp7\" (UniqueName: \"kubernetes.io/projected/06647565-1498-4fe1-a34d-f31450f84ee4-kube-api-access-gxhp7\") pod \"kserve-module-controller-manager-54765d59fd-4ddzl\" (UID: \"06647565-1498-4fe1-a34d-f31450f84ee4\") " pod="opendatahub/kserve-module-controller-manager-54765d59fd-4ddzl"
Sep 19 04:18:55.412636 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:55.412531    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/06647565-1498-4fe1-a34d-f31450f84ee4-tmp\") pod \"kserve-module-controller-manager-54765d59fd-4ddzl\" (UID: \"06647565-1498-4fe1-a34d-f31450f84ee4\") " pod="opendatahub/kserve-module-controller-manager-54765d59fd-4ddzl"
Sep 19 04:18:55.412746 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:55.412675    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/06647565-1498-4fe1-a34d-f31450f84ee4-trusted-ca\") pod \"kserve-module-controller-manager-54765d59fd-4ddzl\" (UID: \"06647565-1498-4fe1-a34d-f31450f84ee4\") " pod="opendatahub/kserve-module-controller-manager-54765d59fd-4ddzl"
Sep 19 04:18:55.513917 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:55.513884    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/06647565-1498-4fe1-a34d-f31450f84ee4-trusted-ca\") pod \"kserve-module-controller-manager-54765d59fd-4ddzl\" (UID: \"06647565-1498-4fe1-a34d-f31450f84ee4\") " pod="opendatahub/kserve-module-controller-manager-54765d59fd-4ddzl"
Sep 19 04:18:55.513917 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:55.513921    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/06647565-1498-4fe1-a34d-f31450f84ee4-manifests\") pod \"kserve-module-controller-manager-54765d59fd-4ddzl\" (UID: \"06647565-1498-4fe1-a34d-f31450f84ee4\") " pod="opendatahub/kserve-module-controller-manager-54765d59fd-4ddzl"
Sep 19 04:18:55.514169 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:55.513955    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxhp7\" (UniqueName: \"kubernetes.io/projected/06647565-1498-4fe1-a34d-f31450f84ee4-kube-api-access-gxhp7\") pod \"kserve-module-controller-manager-54765d59fd-4ddzl\" (UID: \"06647565-1498-4fe1-a34d-f31450f84ee4\") " pod="opendatahub/kserve-module-controller-manager-54765d59fd-4ddzl"
Sep 19 04:18:55.514169 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:55.514002    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/06647565-1498-4fe1-a34d-f31450f84ee4-tmp\") pod \"kserve-module-controller-manager-54765d59fd-4ddzl\" (UID: \"06647565-1498-4fe1-a34d-f31450f84ee4\") " pod="opendatahub/kserve-module-controller-manager-54765d59fd-4ddzl"
Sep 19 04:18:55.514543 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:55.514503    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/06647565-1498-4fe1-a34d-f31450f84ee4-tmp\") pod \"kserve-module-controller-manager-54765d59fd-4ddzl\" (UID: \"06647565-1498-4fe1-a34d-f31450f84ee4\") " pod="opendatahub/kserve-module-controller-manager-54765d59fd-4ddzl"
Sep 19 04:18:55.514652 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:55.514603    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/06647565-1498-4fe1-a34d-f31450f84ee4-manifests\") pod \"kserve-module-controller-manager-54765d59fd-4ddzl\" (UID: \"06647565-1498-4fe1-a34d-f31450f84ee4\") " pod="opendatahub/kserve-module-controller-manager-54765d59fd-4ddzl"
Sep 19 04:18:55.515060 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:55.515042    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/06647565-1498-4fe1-a34d-f31450f84ee4-trusted-ca\") pod \"kserve-module-controller-manager-54765d59fd-4ddzl\" (UID: \"06647565-1498-4fe1-a34d-f31450f84ee4\") " pod="opendatahub/kserve-module-controller-manager-54765d59fd-4ddzl"
Sep 19 04:18:55.552574 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:55.552496    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxhp7\" (UniqueName: \"kubernetes.io/projected/06647565-1498-4fe1-a34d-f31450f84ee4-kube-api-access-gxhp7\") pod \"kserve-module-controller-manager-54765d59fd-4ddzl\" (UID: \"06647565-1498-4fe1-a34d-f31450f84ee4\") " pod="opendatahub/kserve-module-controller-manager-54765d59fd-4ddzl"
Sep 19 04:18:55.603240 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:55.603200    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-54765d59fd-4ddzl"
Sep 19 04:18:55.710723 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:55.710676    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cgg7jw" event={"ID":"382f9c83-8f9d-4e1d-a6d8-a2de02a9fc3c","Type":"ContainerStarted","Data":"2436908e46ac0fa27681beb3f40d0ad675db50e421770ef862c06bb73d635043"}
Sep 19 04:18:55.710894 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:55.710741    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cgg7jw" event={"ID":"382f9c83-8f9d-4e1d-a6d8-a2de02a9fc3c","Type":"ContainerStarted","Data":"72c058357780236946454dfb6bd4ad4b4998260ac8595c3e71bba65bf75816ba"}
Sep 19 04:18:55.751462 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:55.751407    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cgg7jw" podStartSLOduration=1.7513926340000001 podStartE2EDuration="1.751392634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 04:18:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:18:55.748427144 +0000 UTC m=+314.931388292" watchObservedRunningTime="2026-09-19 04:18:55.751392634 +0000 UTC m=+314.934353782"
Sep 19 04:18:55.901660 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:55.901580    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-54765d59fd-4ddzl"]
Sep 19 04:18:55.903986 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:18:55.903961    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06647565_1498_4fe1_a34d_f31450f84ee4.slice/crio-e205c7cad8c9130b7129cbf49459baebc3625d994ecf720c019d19d9881d86fd WatchSource:0}: Error finding container e205c7cad8c9130b7129cbf49459baebc3625d994ecf720c019d19d9881d86fd: Status 404 returned error can't find the container with id e205c7cad8c9130b7129cbf49459baebc3625d994ecf720c019d19d9881d86fd
Sep 19 04:18:56.482289 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:56.482253    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cgg7jw"
Sep 19 04:18:56.488752 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:56.488717    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cgg7jw"
Sep 19 04:18:56.734571 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:56.731764    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-54765d59fd-4ddzl" event={"ID":"06647565-1498-4fe1-a34d-f31450f84ee4","Type":"ContainerStarted","Data":"e205c7cad8c9130b7129cbf49459baebc3625d994ecf720c019d19d9881d86fd"}
Sep 19 04:18:56.734571 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:56.732206    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cgg7jw"
Sep 19 04:18:56.734834 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:56.734765    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cgg7jw"
Sep 19 04:18:57.669929 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:57.669889    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-p62fr"
Sep 19 04:18:57.741387 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:57.741338    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-6c7cb5c9cb-g98cq" event={"ID":"79e96918-0b4e-4940-bb71-ccdfb8b118e2","Type":"ContainerStarted","Data":"9860fedadd69649230acca0459b8293a8ea96615609c1932574596a9d919da6e"}
Sep 19 04:18:57.741579 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:57.741466    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-6c7cb5c9cb-g98cq"
Sep 19 04:18:57.768145 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:57.768083    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-6c7cb5c9cb-g98cq" podStartSLOduration=1.8243998860000001 podStartE2EDuration="10.768060322s" totalImagesPullingTime="8.943660436s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 04:18:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:18:57.765669631 +0000 UTC m=+316.948630779" watchObservedRunningTime="2026-09-19 04:18:57.768060322 +0000 UTC m=+316.951021472"
Sep 19 04:18:58.748942 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:58.748900    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-54765d59fd-4ddzl" event={"ID":"06647565-1498-4fe1-a34d-f31450f84ee4","Type":"ContainerStarted","Data":"60650a812430cd6eb5cff4f7cd52bd7794252ee736a6706cd7b0a2244122a699"}
Sep 19 04:18:58.749517 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:58.749053    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-54765d59fd-4ddzl"
Sep 19 04:18:58.781126 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:18:58.781072    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-54765d59fd-4ddzl" podStartSLOduration=1.335389818 podStartE2EDuration="3.781057307s" totalImagesPullingTime="2.445667489s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 04:18:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:18:58.779875675 +0000 UTC m=+317.962836823" watchObservedRunningTime="2026-09-19 04:18:58.781057307 +0000 UTC m=+317.964018455"
Sep 19 04:19:08.751777 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:19:08.751741    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-6c7cb5c9cb-g98cq"
Sep 19 04:19:19.755067 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:19:19.755034    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-54765d59fd-4ddzl"
Sep 19 04:19:52.444771 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:19:52.444680    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-78f7d99c47-p42kb"]
Sep 19 04:19:52.450832 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:19:52.450811    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-77df98bf98-rzvj8"]
Sep 19 04:19:52.450948 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:19:52.450929    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-78f7d99c47-p42kb"
Sep 19 04:19:52.453820 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:19:52.453803    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-77df98bf98-rzvj8"
Sep 19 04:19:52.472094 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:19:52.472069    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-ss7d4\""
Sep 19 04:19:52.472206 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:19:52.472075    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Sep 19 04:19:52.472265 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:19:52.472234    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Sep 19 04:19:52.472480 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:19:52.472465    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-mbwd2\""
Sep 19 04:19:52.495167 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:19:52.495137    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-77df98bf98-rzvj8"]
Sep 19 04:19:52.515681 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:19:52.515610    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-78f7d99c47-p42kb"]
Sep 19 04:19:52.589410 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:19:52.589382    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7a1ac630-2440-475f-aa6f-c6a416a80c72-cert\") pod \"llmisvc-controller-manager-78f7d99c47-p42kb\" (UID: \"7a1ac630-2440-475f-aa6f-c6a416a80c72\") " pod="opendatahub/llmisvc-controller-manager-78f7d99c47-p42kb"
Sep 19 04:19:52.589582 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:19:52.589429    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb7vq\" (UniqueName: \"kubernetes.io/projected/7a1ac630-2440-475f-aa6f-c6a416a80c72-kube-api-access-sb7vq\") pod \"llmisvc-controller-manager-78f7d99c47-p42kb\" (UID: \"7a1ac630-2440-475f-aa6f-c6a416a80c72\") " pod="opendatahub/llmisvc-controller-manager-78f7d99c47-p42kb"
Sep 19 04:19:52.589582 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:19:52.589449    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2rhv\" (UniqueName: \"kubernetes.io/projected/6e9a8fd5-94c9-4432-8542-07b855ea91d6-kube-api-access-g2rhv\") pod \"model-serving-api-77df98bf98-rzvj8\" (UID: \"6e9a8fd5-94c9-4432-8542-07b855ea91d6\") " pod="opendatahub/model-serving-api-77df98bf98-rzvj8"
Sep 19 04:19:52.589582 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:19:52.589469    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e9a8fd5-94c9-4432-8542-07b855ea91d6-tls-certs\") pod \"model-serving-api-77df98bf98-rzvj8\" (UID: \"6e9a8fd5-94c9-4432-8542-07b855ea91d6\") " pod="opendatahub/model-serving-api-77df98bf98-rzvj8"
Sep 19 04:19:52.690105 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:19:52.690066    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7a1ac630-2440-475f-aa6f-c6a416a80c72-cert\") pod \"llmisvc-controller-manager-78f7d99c47-p42kb\" (UID: \"7a1ac630-2440-475f-aa6f-c6a416a80c72\") " pod="opendatahub/llmisvc-controller-manager-78f7d99c47-p42kb"
Sep 19 04:19:52.690278 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:19:52.690148    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb7vq\" (UniqueName: \"kubernetes.io/projected/7a1ac630-2440-475f-aa6f-c6a416a80c72-kube-api-access-sb7vq\") pod \"llmisvc-controller-manager-78f7d99c47-p42kb\" (UID: \"7a1ac630-2440-475f-aa6f-c6a416a80c72\") " pod="opendatahub/llmisvc-controller-manager-78f7d99c47-p42kb"
Sep 19 04:19:52.690278 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:19:52.690177    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2rhv\" (UniqueName: \"kubernetes.io/projected/6e9a8fd5-94c9-4432-8542-07b855ea91d6-kube-api-access-g2rhv\") pod \"model-serving-api-77df98bf98-rzvj8\" (UID: \"6e9a8fd5-94c9-4432-8542-07b855ea91d6\") " pod="opendatahub/model-serving-api-77df98bf98-rzvj8"
Sep 19 04:19:52.690278 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:19:52.690199    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e9a8fd5-94c9-4432-8542-07b855ea91d6-tls-certs\") pod \"model-serving-api-77df98bf98-rzvj8\" (UID: \"6e9a8fd5-94c9-4432-8542-07b855ea91d6\") " pod="opendatahub/model-serving-api-77df98bf98-rzvj8"
Sep 19 04:19:52.692582 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:19:52.692560    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e9a8fd5-94c9-4432-8542-07b855ea91d6-tls-certs\") pod \"model-serving-api-77df98bf98-rzvj8\" (UID: \"6e9a8fd5-94c9-4432-8542-07b855ea91d6\") " pod="opendatahub/model-serving-api-77df98bf98-rzvj8"
Sep 19 04:19:52.692673 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:19:52.692626    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7a1ac630-2440-475f-aa6f-c6a416a80c72-cert\") pod \"llmisvc-controller-manager-78f7d99c47-p42kb\" (UID: \"7a1ac630-2440-475f-aa6f-c6a416a80c72\") " pod="opendatahub/llmisvc-controller-manager-78f7d99c47-p42kb"
Sep 19 04:19:52.718159 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:19:52.718103    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2rhv\" (UniqueName: \"kubernetes.io/projected/6e9a8fd5-94c9-4432-8542-07b855ea91d6-kube-api-access-g2rhv\") pod \"model-serving-api-77df98bf98-rzvj8\" (UID: \"6e9a8fd5-94c9-4432-8542-07b855ea91d6\") " pod="opendatahub/model-serving-api-77df98bf98-rzvj8"
Sep 19 04:19:52.722994 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:19:52.722975    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb7vq\" (UniqueName: \"kubernetes.io/projected/7a1ac630-2440-475f-aa6f-c6a416a80c72-kube-api-access-sb7vq\") pod \"llmisvc-controller-manager-78f7d99c47-p42kb\" (UID: \"7a1ac630-2440-475f-aa6f-c6a416a80c72\") " pod="opendatahub/llmisvc-controller-manager-78f7d99c47-p42kb"
Sep 19 04:19:52.761555 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:19:52.761534    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-78f7d99c47-p42kb"
Sep 19 04:19:52.767309 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:19:52.767295    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-77df98bf98-rzvj8"
Sep 19 04:19:53.062438 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:19:53.062354    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-78f7d99c47-p42kb"]
Sep 19 04:19:53.065362 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:19:53.065333    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod7a1ac630_2440_475f_aa6f_c6a416a80c72.slice/crio-69fbed6bad4899a0758e0795df576a97d885e7e646075d25d222f17f7caa81a6 WatchSource:0}: Error finding container 69fbed6bad4899a0758e0795df576a97d885e7e646075d25d222f17f7caa81a6: Status 404 returned error can't find the container with id 69fbed6bad4899a0758e0795df576a97d885e7e646075d25d222f17f7caa81a6
Sep 19 04:19:53.071060 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:19:53.071040    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-77df98bf98-rzvj8"]
Sep 19 04:19:53.072907 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:19:53.072889    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e9a8fd5_94c9_4432_8542_07b855ea91d6.slice/crio-dcdec8d091bf8836681d51b548ccbf6e4cdc8a24eeeea8a69a40a27b5d51cace WatchSource:0}: Error finding container dcdec8d091bf8836681d51b548ccbf6e4cdc8a24eeeea8a69a40a27b5d51cace: Status 404 returned error can't find the container with id dcdec8d091bf8836681d51b548ccbf6e4cdc8a24eeeea8a69a40a27b5d51cace
Sep 19 04:19:54.025191 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:19:54.025149    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-77df98bf98-rzvj8" event={"ID":"6e9a8fd5-94c9-4432-8542-07b855ea91d6","Type":"ContainerStarted","Data":"dcdec8d091bf8836681d51b548ccbf6e4cdc8a24eeeea8a69a40a27b5d51cace"}
Sep 19 04:19:54.026757 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:19:54.026727    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-78f7d99c47-p42kb" event={"ID":"7a1ac630-2440-475f-aa6f-c6a416a80c72","Type":"ContainerStarted","Data":"69fbed6bad4899a0758e0795df576a97d885e7e646075d25d222f17f7caa81a6"}
Sep 19 04:19:56.039038 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:19:56.038997    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-78f7d99c47-p42kb" event={"ID":"7a1ac630-2440-475f-aa6f-c6a416a80c72","Type":"ContainerStarted","Data":"29041a68a27aec2cc518badbb2699a856a4e0d66e11e1da0636fea8bbae935e8"}
Sep 19 04:19:56.039470 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:19:56.039091    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-78f7d99c47-p42kb"
Sep 19 04:19:56.040645 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:19:56.040623    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-77df98bf98-rzvj8" event={"ID":"6e9a8fd5-94c9-4432-8542-07b855ea91d6","Type":"ContainerStarted","Data":"92a08122f18784e5b1de935497045d3c9fd347b376db5c08148776686f6654eb"}
Sep 19 04:19:56.040791 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:19:56.040663    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-77df98bf98-rzvj8"
Sep 19 04:19:56.070442 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:19:56.070395    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-78f7d99c47-p42kb" podStartSLOduration=1.865340342 podStartE2EDuration="4.070381971s" totalImagesPullingTime="2.205041629s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 04:19:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:19:56.069462377 +0000 UTC m=+375.252423524" watchObservedRunningTime="2026-09-19 04:19:56.070381971 +0000 UTC m=+375.253343119"
Sep 19 04:19:56.094532 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:19:56.094479    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-77df98bf98-rzvj8" podStartSLOduration=1.902248809 podStartE2EDuration="4.094463564s" totalImagesPullingTime="2.192214755s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 04:19:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:19:56.091303399 +0000 UTC m=+375.274264561" watchObservedRunningTime="2026-09-19 04:19:56.094463564 +0000 UTC m=+375.277424713"
Sep 19 04:20:07.050050 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:20:07.050009    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-77df98bf98-rzvj8"
Sep 19 04:20:27.048547 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:20:27.048517    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-78f7d99c47-p42kb"
Sep 19 04:21:22.235618 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:21:22.235588    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5cff7757cb-j49nf"]
Sep 19 04:21:22.239990 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:21:22.239972    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cff7757cb-j49nf"
Sep 19 04:21:22.290251 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:21:22.290224    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5cff7757cb-j49nf"]
Sep 19 04:21:22.346958 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:21:22.346926    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b26f88b6-5bc5-46dd-992b-3c214d4a4d04-service-ca\") pod \"console-5cff7757cb-j49nf\" (UID: \"b26f88b6-5bc5-46dd-992b-3c214d4a4d04\") " pod="openshift-console/console-5cff7757cb-j49nf"
Sep 19 04:21:22.346958 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:21:22.346959    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b26f88b6-5bc5-46dd-992b-3c214d4a4d04-trusted-ca-bundle\") pod \"console-5cff7757cb-j49nf\" (UID: \"b26f88b6-5bc5-46dd-992b-3c214d4a4d04\") " pod="openshift-console/console-5cff7757cb-j49nf"
Sep 19 04:21:22.346958 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:21:22.346975    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj6ws\" (UniqueName: \"kubernetes.io/projected/b26f88b6-5bc5-46dd-992b-3c214d4a4d04-kube-api-access-cj6ws\") pod \"console-5cff7757cb-j49nf\" (UID: \"b26f88b6-5bc5-46dd-992b-3c214d4a4d04\") " pod="openshift-console/console-5cff7757cb-j49nf"
Sep 19 04:21:22.347185 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:21:22.347102    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b26f88b6-5bc5-46dd-992b-3c214d4a4d04-console-oauth-config\") pod \"console-5cff7757cb-j49nf\" (UID: \"b26f88b6-5bc5-46dd-992b-3c214d4a4d04\") " pod="openshift-console/console-5cff7757cb-j49nf"
Sep 19 04:21:22.347185 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:21:22.347132    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b26f88b6-5bc5-46dd-992b-3c214d4a4d04-oauth-serving-cert\") pod \"console-5cff7757cb-j49nf\" (UID: \"b26f88b6-5bc5-46dd-992b-3c214d4a4d04\") " pod="openshift-console/console-5cff7757cb-j49nf"
Sep 19 04:21:22.347185 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:21:22.347162    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b26f88b6-5bc5-46dd-992b-3c214d4a4d04-console-config\") pod \"console-5cff7757cb-j49nf\" (UID: \"b26f88b6-5bc5-46dd-992b-3c214d4a4d04\") " pod="openshift-console/console-5cff7757cb-j49nf"
Sep 19 04:21:22.347281 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:21:22.347189    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b26f88b6-5bc5-46dd-992b-3c214d4a4d04-console-serving-cert\") pod \"console-5cff7757cb-j49nf\" (UID: \"b26f88b6-5bc5-46dd-992b-3c214d4a4d04\") " pod="openshift-console/console-5cff7757cb-j49nf"
Sep 19 04:21:22.447833 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:21:22.447796    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b26f88b6-5bc5-46dd-992b-3c214d4a4d04-console-serving-cert\") pod \"console-5cff7757cb-j49nf\" (UID: \"b26f88b6-5bc5-46dd-992b-3c214d4a4d04\") " pod="openshift-console/console-5cff7757cb-j49nf"
Sep 19 04:21:22.448013 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:21:22.447842    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b26f88b6-5bc5-46dd-992b-3c214d4a4d04-service-ca\") pod \"console-5cff7757cb-j49nf\" (UID: \"b26f88b6-5bc5-46dd-992b-3c214d4a4d04\") " pod="openshift-console/console-5cff7757cb-j49nf"
Sep 19 04:21:22.448013 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:21:22.447861    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b26f88b6-5bc5-46dd-992b-3c214d4a4d04-trusted-ca-bundle\") pod \"console-5cff7757cb-j49nf\" (UID: \"b26f88b6-5bc5-46dd-992b-3c214d4a4d04\") " pod="openshift-console/console-5cff7757cb-j49nf"
Sep 19 04:21:22.448013 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:21:22.447881    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj6ws\" (UniqueName: \"kubernetes.io/projected/b26f88b6-5bc5-46dd-992b-3c214d4a4d04-kube-api-access-cj6ws\") pod \"console-5cff7757cb-j49nf\" (UID: \"b26f88b6-5bc5-46dd-992b-3c214d4a4d04\") " pod="openshift-console/console-5cff7757cb-j49nf"
Sep 19 04:21:22.448013 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:21:22.447987    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b26f88b6-5bc5-46dd-992b-3c214d4a4d04-console-oauth-config\") pod \"console-5cff7757cb-j49nf\" (UID: \"b26f88b6-5bc5-46dd-992b-3c214d4a4d04\") " pod="openshift-console/console-5cff7757cb-j49nf"
Sep 19 04:21:22.448214 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:21:22.448021    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b26f88b6-5bc5-46dd-992b-3c214d4a4d04-oauth-serving-cert\") pod \"console-5cff7757cb-j49nf\" (UID: \"b26f88b6-5bc5-46dd-992b-3c214d4a4d04\") " pod="openshift-console/console-5cff7757cb-j49nf"
Sep 19 04:21:22.448214 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:21:22.448057    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b26f88b6-5bc5-46dd-992b-3c214d4a4d04-console-config\") pod \"console-5cff7757cb-j49nf\" (UID: \"b26f88b6-5bc5-46dd-992b-3c214d4a4d04\") " pod="openshift-console/console-5cff7757cb-j49nf"
Sep 19 04:21:22.448828 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:21:22.448685    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b26f88b6-5bc5-46dd-992b-3c214d4a4d04-console-config\") pod \"console-5cff7757cb-j49nf\" (UID: \"b26f88b6-5bc5-46dd-992b-3c214d4a4d04\") " pod="openshift-console/console-5cff7757cb-j49nf"
Sep 19 04:21:22.448828 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:21:22.448685    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b26f88b6-5bc5-46dd-992b-3c214d4a4d04-service-ca\") pod \"console-5cff7757cb-j49nf\" (UID: \"b26f88b6-5bc5-46dd-992b-3c214d4a4d04\") " pod="openshift-console/console-5cff7757cb-j49nf"
Sep 19 04:21:22.448828 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:21:22.448776    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b26f88b6-5bc5-46dd-992b-3c214d4a4d04-oauth-serving-cert\") pod \"console-5cff7757cb-j49nf\" (UID: \"b26f88b6-5bc5-46dd-992b-3c214d4a4d04\") " pod="openshift-console/console-5cff7757cb-j49nf"
Sep 19 04:21:22.448828 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:21:22.448802    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b26f88b6-5bc5-46dd-992b-3c214d4a4d04-trusted-ca-bundle\") pod \"console-5cff7757cb-j49nf\" (UID: \"b26f88b6-5bc5-46dd-992b-3c214d4a4d04\") " pod="openshift-console/console-5cff7757cb-j49nf"
Sep 19 04:21:22.450527 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:21:22.450507    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b26f88b6-5bc5-46dd-992b-3c214d4a4d04-console-serving-cert\") pod \"console-5cff7757cb-j49nf\" (UID: \"b26f88b6-5bc5-46dd-992b-3c214d4a4d04\") " pod="openshift-console/console-5cff7757cb-j49nf"
Sep 19 04:21:22.450618 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:21:22.450572    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b26f88b6-5bc5-46dd-992b-3c214d4a4d04-console-oauth-config\") pod \"console-5cff7757cb-j49nf\" (UID: \"b26f88b6-5bc5-46dd-992b-3c214d4a4d04\") " pod="openshift-console/console-5cff7757cb-j49nf"
Sep 19 04:21:22.459983 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:21:22.459957    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj6ws\" (UniqueName: \"kubernetes.io/projected/b26f88b6-5bc5-46dd-992b-3c214d4a4d04-kube-api-access-cj6ws\") pod \"console-5cff7757cb-j49nf\" (UID: \"b26f88b6-5bc5-46dd-992b-3c214d4a4d04\") " pod="openshift-console/console-5cff7757cb-j49nf"
Sep 19 04:21:22.549153 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:21:22.549057    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cff7757cb-j49nf"
Sep 19 04:21:22.840780 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:21:22.840641    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5cff7757cb-j49nf"]
Sep 19 04:21:22.845025 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:21:22.844998    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb26f88b6_5bc5_46dd_992b_3c214d4a4d04.slice/crio-e8293c734d255f8425f13cdbba593d5cf3beceff6333aace25afe76c0e59f76a WatchSource:0}: Error finding container e8293c734d255f8425f13cdbba593d5cf3beceff6333aace25afe76c0e59f76a: Status 404 returned error can't find the container with id e8293c734d255f8425f13cdbba593d5cf3beceff6333aace25afe76c0e59f76a
Sep 19 04:21:23.482604 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:21:23.482569    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cff7757cb-j49nf" event={"ID":"b26f88b6-5bc5-46dd-992b-3c214d4a4d04","Type":"ContainerStarted","Data":"ace100b32ac8d79459897366295429e5295da542a4e4bf330923e63781728efd"}
Sep 19 04:21:23.483090 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:21:23.482622    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cff7757cb-j49nf" event={"ID":"b26f88b6-5bc5-46dd-992b-3c214d4a4d04","Type":"ContainerStarted","Data":"e8293c734d255f8425f13cdbba593d5cf3beceff6333aace25afe76c0e59f76a"}
Sep 19 04:21:23.506361 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:21:23.506311    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5cff7757cb-j49nf" podStartSLOduration=1.506297157 podStartE2EDuration="1.506297157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 04:21:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:21:23.505835278 +0000 UTC m=+462.688796427" watchObservedRunningTime="2026-09-19 04:21:23.506297157 +0000 UTC m=+462.689258304"
Sep 19 04:21:32.550066 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:21:32.550020    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5cff7757cb-j49nf"
Sep 19 04:21:32.550468 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:21:32.550092    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5cff7757cb-j49nf"
Sep 19 04:21:32.555152 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:21:32.555130    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5cff7757cb-j49nf"
Sep 19 04:21:33.534026 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:21:33.533999    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5cff7757cb-j49nf"
Sep 19 04:21:33.598815 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:21:33.598787    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-b6f457954-wwpfn"]
Sep 19 04:21:58.619136 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:21:58.619090    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-b6f457954-wwpfn" podUID="cfb3087d-22a4-4962-8199-2b989fe224fe" containerName="console" containerID="cri-o://546e07faed51701e0d9f49af7fc88c37194fafca8e7efcf06bb4f6405d337e18" gracePeriod=15
Sep 19 04:21:58.870805 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:21:58.870746    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b6f457954-wwpfn"
Sep 19 04:21:58.977703 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:21:58.977668    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cfb3087d-22a4-4962-8199-2b989fe224fe-trusted-ca-bundle\") pod \"cfb3087d-22a4-4962-8199-2b989fe224fe\" (UID: \"cfb3087d-22a4-4962-8199-2b989fe224fe\") "
Sep 19 04:21:58.977890 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:21:58.977773    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cfb3087d-22a4-4962-8199-2b989fe224fe-console-config\") pod \"cfb3087d-22a4-4962-8199-2b989fe224fe\" (UID: \"cfb3087d-22a4-4962-8199-2b989fe224fe\") "
Sep 19 04:21:58.977890 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:21:58.977816    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cfb3087d-22a4-4962-8199-2b989fe224fe-oauth-serving-cert\") pod \"cfb3087d-22a4-4962-8199-2b989fe224fe\" (UID: \"cfb3087d-22a4-4962-8199-2b989fe224fe\") "
Sep 19 04:21:58.977890 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:21:58.977847    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hbg7t\" (UniqueName: \"kubernetes.io/projected/cfb3087d-22a4-4962-8199-2b989fe224fe-kube-api-access-hbg7t\") pod \"cfb3087d-22a4-4962-8199-2b989fe224fe\" (UID: \"cfb3087d-22a4-4962-8199-2b989fe224fe\") "
Sep 19 04:21:58.978051 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:21:58.977892    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cfb3087d-22a4-4962-8199-2b989fe224fe-console-oauth-config\") pod \"cfb3087d-22a4-4962-8199-2b989fe224fe\" (UID: \"cfb3087d-22a4-4962-8199-2b989fe224fe\") "
Sep 19 04:21:58.978051 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:21:58.977920    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cfb3087d-22a4-4962-8199-2b989fe224fe-service-ca\") pod \"cfb3087d-22a4-4962-8199-2b989fe224fe\" (UID: \"cfb3087d-22a4-4962-8199-2b989fe224fe\") "
Sep 19 04:21:58.978051 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:21:58.977951    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cfb3087d-22a4-4962-8199-2b989fe224fe-console-serving-cert\") pod \"cfb3087d-22a4-4962-8199-2b989fe224fe\" (UID: \"cfb3087d-22a4-4962-8199-2b989fe224fe\") "
Sep 19 04:21:58.978265 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:21:58.978215    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfb3087d-22a4-4962-8199-2b989fe224fe-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "cfb3087d-22a4-4962-8199-2b989fe224fe" (UID: "cfb3087d-22a4-4962-8199-2b989fe224fe"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 04:21:58.978265 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:21:58.978237    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfb3087d-22a4-4962-8199-2b989fe224fe-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "cfb3087d-22a4-4962-8199-2b989fe224fe" (UID: "cfb3087d-22a4-4962-8199-2b989fe224fe"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 04:21:58.978459 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:21:58.978271    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfb3087d-22a4-4962-8199-2b989fe224fe-console-config" (OuterVolumeSpecName: "console-config") pod "cfb3087d-22a4-4962-8199-2b989fe224fe" (UID: "cfb3087d-22a4-4962-8199-2b989fe224fe"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 04:21:58.978459 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:21:58.978340    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfb3087d-22a4-4962-8199-2b989fe224fe-service-ca" (OuterVolumeSpecName: "service-ca") pod "cfb3087d-22a4-4962-8199-2b989fe224fe" (UID: "cfb3087d-22a4-4962-8199-2b989fe224fe"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 04:21:58.980237 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:21:58.980204    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfb3087d-22a4-4962-8199-2b989fe224fe-kube-api-access-hbg7t" (OuterVolumeSpecName: "kube-api-access-hbg7t") pod "cfb3087d-22a4-4962-8199-2b989fe224fe" (UID: "cfb3087d-22a4-4962-8199-2b989fe224fe"). InnerVolumeSpecName "kube-api-access-hbg7t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 04:21:58.980339 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:21:58.980263    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfb3087d-22a4-4962-8199-2b989fe224fe-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "cfb3087d-22a4-4962-8199-2b989fe224fe" (UID: "cfb3087d-22a4-4962-8199-2b989fe224fe"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 04:21:58.980339 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:21:58.980285    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfb3087d-22a4-4962-8199-2b989fe224fe-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "cfb3087d-22a4-4962-8199-2b989fe224fe" (UID: "cfb3087d-22a4-4962-8199-2b989fe224fe"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 04:21:59.078904 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:21:59.078865    2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cfb3087d-22a4-4962-8199-2b989fe224fe-console-config\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:21:59.078904 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:21:59.078899    2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cfb3087d-22a4-4962-8199-2b989fe224fe-oauth-serving-cert\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:21:59.078904 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:21:59.078913    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbg7t\" (UniqueName: \"kubernetes.io/projected/cfb3087d-22a4-4962-8199-2b989fe224fe-kube-api-access-hbg7t\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:21:59.079149 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:21:59.078925    2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cfb3087d-22a4-4962-8199-2b989fe224fe-console-oauth-config\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:21:59.079149 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:21:59.078939    2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cfb3087d-22a4-4962-8199-2b989fe224fe-service-ca\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:21:59.079149 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:21:59.078952    2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cfb3087d-22a4-4962-8199-2b989fe224fe-console-serving-cert\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:21:59.079149 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:21:59.078962    2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cfb3087d-22a4-4962-8199-2b989fe224fe-trusted-ca-bundle\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:21:59.657100 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:21:59.657069    2579 generic.go:356] "Generic (PLEG): container finished" podID="cfb3087d-22a4-4962-8199-2b989fe224fe" containerID="546e07faed51701e0d9f49af7fc88c37194fafca8e7efcf06bb4f6405d337e18" exitCode=0
Sep 19 04:21:59.657517 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:21:59.657126    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b6f457954-wwpfn"
Sep 19 04:21:59.657517 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:21:59.657157    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b6f457954-wwpfn" event={"ID":"cfb3087d-22a4-4962-8199-2b989fe224fe","Type":"ContainerDied","Data":"546e07faed51701e0d9f49af7fc88c37194fafca8e7efcf06bb4f6405d337e18"}
Sep 19 04:21:59.657517 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:21:59.657206    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b6f457954-wwpfn" event={"ID":"cfb3087d-22a4-4962-8199-2b989fe224fe","Type":"ContainerDied","Data":"64a87174a170166ad008d40db98e2191900b24549e911deec6b9e68ff4749e2c"}
Sep 19 04:21:59.657517 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:21:59.657225    2579 scope.go:117] "RemoveContainer" containerID="546e07faed51701e0d9f49af7fc88c37194fafca8e7efcf06bb4f6405d337e18"
Sep 19 04:21:59.668416 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:21:59.668397    2579 scope.go:117] "RemoveContainer" containerID="546e07faed51701e0d9f49af7fc88c37194fafca8e7efcf06bb4f6405d337e18"
Sep 19 04:21:59.668726 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:21:59.668681    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"546e07faed51701e0d9f49af7fc88c37194fafca8e7efcf06bb4f6405d337e18\": container with ID starting with 546e07faed51701e0d9f49af7fc88c37194fafca8e7efcf06bb4f6405d337e18 not found: ID does not exist" containerID="546e07faed51701e0d9f49af7fc88c37194fafca8e7efcf06bb4f6405d337e18"
Sep 19 04:21:59.668821 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:21:59.668734    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"546e07faed51701e0d9f49af7fc88c37194fafca8e7efcf06bb4f6405d337e18"} err="failed to get container status \"546e07faed51701e0d9f49af7fc88c37194fafca8e7efcf06bb4f6405d337e18\": rpc error: code = NotFound desc = could not find container \"546e07faed51701e0d9f49af7fc88c37194fafca8e7efcf06bb4f6405d337e18\": container with ID starting with 546e07faed51701e0d9f49af7fc88c37194fafca8e7efcf06bb4f6405d337e18 not found: ID does not exist"
Sep 19 04:21:59.682457 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:21:59.682435    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-b6f457954-wwpfn"]
Sep 19 04:21:59.684170 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:21:59.684148    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-b6f457954-wwpfn"]
Sep 19 04:22:01.351239 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:22:01.351210    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cfb3087d-22a4-4962-8199-2b989fe224fe" path="/var/lib/kubelet/pods/cfb3087d-22a4-4962-8199-2b989fe224fe/volumes"
Sep 19 04:29:09.095053 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:29:09.095025    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-54b6d5c989-p62fr_10734032-f90d-4666-8d77-0573f1cce689/manager/0.log"
Sep 19 04:29:09.277119 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:29:09.277078    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-p62fr"]
Sep 19 04:29:09.277325 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:29:09.277303    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-p62fr" podUID="10734032-f90d-4666-8d77-0573f1cce689" containerName="manager" containerID="cri-o://c57daf08ef7c72e9a974a9f2c46ed5c45cb7f04dd41db2bcd7f33627216b51e5" gracePeriod=10
Sep 19 04:29:09.365431 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:29:09.365347    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-z2g72"]
Sep 19 04:29:09.365743 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:29:09.365729    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfb3087d-22a4-4962-8199-2b989fe224fe" containerName="console"
Sep 19 04:29:09.365790 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:29:09.365748    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfb3087d-22a4-4962-8199-2b989fe224fe" containerName="console"
Sep 19 04:29:09.365827 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:29:09.365815    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfb3087d-22a4-4962-8199-2b989fe224fe" containerName="console"
Sep 19 04:29:09.368737 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:29:09.368722    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-z2g72"
Sep 19 04:29:09.411240 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:29:09.411198    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-z2g72"]
Sep 19 04:29:09.419645 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:29:09.419614    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttx4j\" (UniqueName: \"kubernetes.io/projected/d2cb2792-51a4-4365-95f5-de0f536efd2c-kube-api-access-ttx4j\") pod \"kuadrant-operator-controller-manager-54b6d5c989-z2g72\" (UID: \"d2cb2792-51a4-4365-95f5-de0f536efd2c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-z2g72"
Sep 19 04:29:09.419860 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:29:09.419735    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d2cb2792-51a4-4365-95f5-de0f536efd2c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-54b6d5c989-z2g72\" (UID: \"d2cb2792-51a4-4365-95f5-de0f536efd2c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-z2g72"
Sep 19 04:29:09.520180 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:29:09.520142    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d2cb2792-51a4-4365-95f5-de0f536efd2c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-54b6d5c989-z2g72\" (UID: \"d2cb2792-51a4-4365-95f5-de0f536efd2c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-z2g72"
Sep 19 04:29:09.520328 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:29:09.520203    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttx4j\" (UniqueName: \"kubernetes.io/projected/d2cb2792-51a4-4365-95f5-de0f536efd2c-kube-api-access-ttx4j\") pod \"kuadrant-operator-controller-manager-54b6d5c989-z2g72\" (UID: \"d2cb2792-51a4-4365-95f5-de0f536efd2c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-z2g72"
Sep 19 04:29:09.520574 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:29:09.520547    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d2cb2792-51a4-4365-95f5-de0f536efd2c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-54b6d5c989-z2g72\" (UID: \"d2cb2792-51a4-4365-95f5-de0f536efd2c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-z2g72"
Sep 19 04:29:09.531334 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:29:09.531304    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-p62fr"
Sep 19 04:29:09.532319 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:29:09.532294    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttx4j\" (UniqueName: \"kubernetes.io/projected/d2cb2792-51a4-4365-95f5-de0f536efd2c-kube-api-access-ttx4j\") pod \"kuadrant-operator-controller-manager-54b6d5c989-z2g72\" (UID: \"d2cb2792-51a4-4365-95f5-de0f536efd2c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-z2g72"
Sep 19 04:29:09.620867 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:29:09.620766    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5tm7s\" (UniqueName: \"kubernetes.io/projected/10734032-f90d-4666-8d77-0573f1cce689-kube-api-access-5tm7s\") pod \"10734032-f90d-4666-8d77-0573f1cce689\" (UID: \"10734032-f90d-4666-8d77-0573f1cce689\") "
Sep 19 04:29:09.620867 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:29:09.620819    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/10734032-f90d-4666-8d77-0573f1cce689-extensions-socket-volume\") pod \"10734032-f90d-4666-8d77-0573f1cce689\" (UID: \"10734032-f90d-4666-8d77-0573f1cce689\") "
Sep 19 04:29:09.621162 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:29:09.621139    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10734032-f90d-4666-8d77-0573f1cce689-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "10734032-f90d-4666-8d77-0573f1cce689" (UID: "10734032-f90d-4666-8d77-0573f1cce689"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:29:09.623119 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:29:09.623092    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10734032-f90d-4666-8d77-0573f1cce689-kube-api-access-5tm7s" (OuterVolumeSpecName: "kube-api-access-5tm7s") pod "10734032-f90d-4666-8d77-0573f1cce689" (UID: "10734032-f90d-4666-8d77-0573f1cce689"). InnerVolumeSpecName "kube-api-access-5tm7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 04:29:09.678468 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:29:09.678436    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-z2g72"
Sep 19 04:29:09.722093 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:29:09.722053    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tm7s\" (UniqueName: \"kubernetes.io/projected/10734032-f90d-4666-8d77-0573f1cce689-kube-api-access-5tm7s\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:29:09.722093 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:29:09.722089    2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/10734032-f90d-4666-8d77-0573f1cce689-extensions-socket-volume\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:29:09.758337 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:29:09.758308    2579 generic.go:356] "Generic (PLEG): container finished" podID="10734032-f90d-4666-8d77-0573f1cce689" containerID="c57daf08ef7c72e9a974a9f2c46ed5c45cb7f04dd41db2bcd7f33627216b51e5" exitCode=0
Sep 19 04:29:09.758488 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:29:09.758357    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-p62fr"
Sep 19 04:29:09.758488 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:29:09.758348    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-p62fr" event={"ID":"10734032-f90d-4666-8d77-0573f1cce689","Type":"ContainerDied","Data":"c57daf08ef7c72e9a974a9f2c46ed5c45cb7f04dd41db2bcd7f33627216b51e5"}
Sep 19 04:29:09.758488 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:29:09.758470    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-p62fr" event={"ID":"10734032-f90d-4666-8d77-0573f1cce689","Type":"ContainerDied","Data":"ca668ab271e921b8a59f6363592e8b446872bf9b962065ec87a5bb16e4bd791a"}
Sep 19 04:29:09.758588 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:29:09.758498    2579 scope.go:117] "RemoveContainer" containerID="c57daf08ef7c72e9a974a9f2c46ed5c45cb7f04dd41db2bcd7f33627216b51e5"
Sep 19 04:29:09.767984 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:29:09.767967    2579 scope.go:117] "RemoveContainer" containerID="c57daf08ef7c72e9a974a9f2c46ed5c45cb7f04dd41db2bcd7f33627216b51e5"
Sep 19 04:29:09.768265 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:29:09.768249    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c57daf08ef7c72e9a974a9f2c46ed5c45cb7f04dd41db2bcd7f33627216b51e5\": container with ID starting with c57daf08ef7c72e9a974a9f2c46ed5c45cb7f04dd41db2bcd7f33627216b51e5 not found: ID does not exist" containerID="c57daf08ef7c72e9a974a9f2c46ed5c45cb7f04dd41db2bcd7f33627216b51e5"
Sep 19 04:29:09.768309 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:29:09.768273    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c57daf08ef7c72e9a974a9f2c46ed5c45cb7f04dd41db2bcd7f33627216b51e5"} err="failed to get container status \"c57daf08ef7c72e9a974a9f2c46ed5c45cb7f04dd41db2bcd7f33627216b51e5\": rpc error: code = NotFound desc = could not find container \"c57daf08ef7c72e9a974a9f2c46ed5c45cb7f04dd41db2bcd7f33627216b51e5\": container with ID starting with c57daf08ef7c72e9a974a9f2c46ed5c45cb7f04dd41db2bcd7f33627216b51e5 not found: ID does not exist"
Sep 19 04:29:09.788283 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:29:09.788253    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-p62fr"]
Sep 19 04:29:09.798667 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:29:09.798644    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-p62fr"]
Sep 19 04:29:09.975343 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:29:09.975318    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-z2g72"]
Sep 19 04:29:09.977606 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:29:09.977580    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2cb2792_51a4_4365_95f5_de0f536efd2c.slice/crio-dfa61e43c6b05c651acfedbaa7b5a5e03ecf1aa19c33118dbbcb211936572160 WatchSource:0}: Error finding container dfa61e43c6b05c651acfedbaa7b5a5e03ecf1aa19c33118dbbcb211936572160: Status 404 returned error can't find the container with id dfa61e43c6b05c651acfedbaa7b5a5e03ecf1aa19c33118dbbcb211936572160
Sep 19 04:29:10.765985 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:29:10.765950    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-z2g72" event={"ID":"d2cb2792-51a4-4365-95f5-de0f536efd2c","Type":"ContainerStarted","Data":"68de29a2166259ee1c510a5a49c0e986026e078e83b7b550650c3b46be061e68"}
Sep 19 04:29:10.766363 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:29:10.766036    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-z2g72" event={"ID":"d2cb2792-51a4-4365-95f5-de0f536efd2c","Type":"ContainerStarted","Data":"dfa61e43c6b05c651acfedbaa7b5a5e03ecf1aa19c33118dbbcb211936572160"}
Sep 19 04:29:10.766363 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:29:10.766082    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-z2g72"
Sep 19 04:29:10.803844 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:29:10.803787    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-z2g72" podStartSLOduration=1.803769532 podStartE2EDuration="1.803769532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 04:29:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:29:10.801441208 +0000 UTC m=+929.984402356" watchObservedRunningTime="2026-09-19 04:29:10.803769532 +0000 UTC m=+929.986730680"
Sep 19 04:29:11.350777 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:29:11.350743    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10734032-f90d-4666-8d77-0573f1cce689" path="/var/lib/kubelet/pods/10734032-f90d-4666-8d77-0573f1cce689/volumes"
Sep 19 04:29:21.773574 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:29:21.773546    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-z2g72"
Sep 19 04:29:29.535784 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:29:29.535753    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-7hchz"]
Sep 19 04:29:29.536138 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:29:29.536073    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10734032-f90d-4666-8d77-0573f1cce689" containerName="manager"
Sep 19 04:29:29.536138 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:29:29.536083    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="10734032-f90d-4666-8d77-0573f1cce689" containerName="manager"
Sep 19 04:29:29.536212 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:29:29.536142    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="10734032-f90d-4666-8d77-0573f1cce689" containerName="manager"
Sep 19 04:29:29.540369 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:29:29.540339    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-7hchz"
Sep 19 04:29:29.544285 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:29:29.544262    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-677rc\""
Sep 19 04:29:29.551887 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:29:29.551863    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-7hchz"]
Sep 19 04:29:29.698024 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:29:29.697983    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-rz9bs"]
Sep 19 04:29:29.702762 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:29:29.702736    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-rz9bs"
Sep 19 04:29:29.704164 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:29:29.704136    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9kdq\" (UniqueName: \"kubernetes.io/projected/898ce840-1854-4773-9ef1-b2b22e8aca17-kube-api-access-g9kdq\") pod \"authorino-7bcfd8f9cd-7hchz\" (UID: \"898ce840-1854-4773-9ef1-b2b22e8aca17\") " pod="kuadrant-system/authorino-7bcfd8f9cd-7hchz"
Sep 19 04:29:29.705472 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:29:29.705439    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-rz9bs"]
Sep 19 04:29:29.805195 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:29:29.805112    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4chq\" (UniqueName: \"kubernetes.io/projected/7160552c-acfc-41be-9313-fe519bf13786-kube-api-access-q4chq\") pod \"authorino-59c974bd48-rz9bs\" (UID: \"7160552c-acfc-41be-9313-fe519bf13786\") " pod="kuadrant-system/authorino-59c974bd48-rz9bs"
Sep 19 04:29:29.805355 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:29:29.805219    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9kdq\" (UniqueName: \"kubernetes.io/projected/898ce840-1854-4773-9ef1-b2b22e8aca17-kube-api-access-g9kdq\") pod \"authorino-7bcfd8f9cd-7hchz\" (UID: \"898ce840-1854-4773-9ef1-b2b22e8aca17\") " pod="kuadrant-system/authorino-7bcfd8f9cd-7hchz"
Sep 19 04:29:29.814761 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:29:29.814727    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9kdq\" (UniqueName: \"kubernetes.io/projected/898ce840-1854-4773-9ef1-b2b22e8aca17-kube-api-access-g9kdq\") pod \"authorino-7bcfd8f9cd-7hchz\" (UID: \"898ce840-1854-4773-9ef1-b2b22e8aca17\") " pod="kuadrant-system/authorino-7bcfd8f9cd-7hchz"
Sep 19 04:29:29.849973 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:29:29.849935    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-7hchz"
Sep 19 04:29:29.906279 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:29:29.906248    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4chq\" (UniqueName: \"kubernetes.io/projected/7160552c-acfc-41be-9313-fe519bf13786-kube-api-access-q4chq\") pod \"authorino-59c974bd48-rz9bs\" (UID: \"7160552c-acfc-41be-9313-fe519bf13786\") " pod="kuadrant-system/authorino-59c974bd48-rz9bs"
Sep 19 04:29:29.916661 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:29:29.916629    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4chq\" (UniqueName: \"kubernetes.io/projected/7160552c-acfc-41be-9313-fe519bf13786-kube-api-access-q4chq\") pod \"authorino-59c974bd48-rz9bs\" (UID: \"7160552c-acfc-41be-9313-fe519bf13786\") " pod="kuadrant-system/authorino-59c974bd48-rz9bs"
Sep 19 04:29:30.012530 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:29:30.012492    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-rz9bs"
Sep 19 04:29:30.120097 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:29:30.120069    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-7hchz"]
Sep 19 04:29:30.121834 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:29:30.121805    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod898ce840_1854_4773_9ef1_b2b22e8aca17.slice/crio-670d709f2fedae582b8cfd6f0d5291dbe311a3dd81146a80aa45fbd7b1cd8a8b WatchSource:0}: Error finding container 670d709f2fedae582b8cfd6f0d5291dbe311a3dd81146a80aa45fbd7b1cd8a8b: Status 404 returned error can't find the container with id 670d709f2fedae582b8cfd6f0d5291dbe311a3dd81146a80aa45fbd7b1cd8a8b
Sep 19 04:29:30.123139 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:29:30.123124    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 19 04:29:30.284752 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:29:30.284716    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-rz9bs"]
Sep 19 04:29:30.287547 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:29:30.287523    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7160552c_acfc_41be_9313_fe519bf13786.slice/crio-b2cd62f6a63b8414291c159b77d15ad92c51ffe3a0c4a02efeda3ad109dd8277 WatchSource:0}: Error finding container b2cd62f6a63b8414291c159b77d15ad92c51ffe3a0c4a02efeda3ad109dd8277: Status 404 returned error can't find the container with id b2cd62f6a63b8414291c159b77d15ad92c51ffe3a0c4a02efeda3ad109dd8277
Sep 19 04:29:30.870017 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:29:30.869978    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-rz9bs" event={"ID":"7160552c-acfc-41be-9313-fe519bf13786","Type":"ContainerStarted","Data":"b2cd62f6a63b8414291c159b77d15ad92c51ffe3a0c4a02efeda3ad109dd8277"}
Sep 19 04:29:30.871122 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:29:30.871100    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-7hchz" event={"ID":"898ce840-1854-4773-9ef1-b2b22e8aca17","Type":"ContainerStarted","Data":"670d709f2fedae582b8cfd6f0d5291dbe311a3dd81146a80aa45fbd7b1cd8a8b"}
Sep 19 04:29:32.883048 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:29:32.883013    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-rz9bs" event={"ID":"7160552c-acfc-41be-9313-fe519bf13786","Type":"ContainerStarted","Data":"a60075ed936653b044642e9fb91c37b422bd7b80a8ff7e95aaf28b706b5763b6"}
Sep 19 04:29:32.884526 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:29:32.884502    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-7hchz" event={"ID":"898ce840-1854-4773-9ef1-b2b22e8aca17","Type":"ContainerStarted","Data":"4ba3b416258b280f8549321064351406c4a4ac23b9c992ab6358b0b0442cb550"}
Sep 19 04:29:32.900806 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:29:32.900756    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-rz9bs" podStartSLOduration=2.233605543 podStartE2EDuration="3.900739825s" totalImagesPullingTime="1.667134282s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 04:29:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:29:32.900367822 +0000 UTC m=+952.083328967" watchObservedRunningTime="2026-09-19 04:29:32.900739825 +0000 UTC m=+952.083700990"
Sep 19 04:29:32.917526 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:29:32.917467    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-7hchz" podStartSLOduration=2.0885484500000002 podStartE2EDuration="3.917445387s" totalImagesPullingTime="1.828896937s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 04:29:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:29:32.916190489 +0000 UTC m=+952.099151840" watchObservedRunningTime="2026-09-19 04:29:32.917445387 +0000 UTC m=+952.100406536"
Sep 19 04:29:32.952552 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:29:32.952517    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-7hchz"]
Sep 19 04:29:34.894709 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:29:34.894642    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-7hchz" podUID="898ce840-1854-4773-9ef1-b2b22e8aca17" containerName="authorino" containerID="cri-o://4ba3b416258b280f8549321064351406c4a4ac23b9c992ab6358b0b0442cb550" gracePeriod=30
Sep 19 04:29:35.136668 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:29:35.136643    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-7hchz"
Sep 19 04:29:35.256312 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:29:35.256268    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g9kdq\" (UniqueName: \"kubernetes.io/projected/898ce840-1854-4773-9ef1-b2b22e8aca17-kube-api-access-g9kdq\") pod \"898ce840-1854-4773-9ef1-b2b22e8aca17\" (UID: \"898ce840-1854-4773-9ef1-b2b22e8aca17\") "
Sep 19 04:29:35.258502 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:29:35.258470    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/898ce840-1854-4773-9ef1-b2b22e8aca17-kube-api-access-g9kdq" (OuterVolumeSpecName: "kube-api-access-g9kdq") pod "898ce840-1854-4773-9ef1-b2b22e8aca17" (UID: "898ce840-1854-4773-9ef1-b2b22e8aca17"). InnerVolumeSpecName "kube-api-access-g9kdq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 04:29:35.357289 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:29:35.357044    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9kdq\" (UniqueName: \"kubernetes.io/projected/898ce840-1854-4773-9ef1-b2b22e8aca17-kube-api-access-g9kdq\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:29:35.900374 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:29:35.900329    2579 generic.go:356] "Generic (PLEG): container finished" podID="898ce840-1854-4773-9ef1-b2b22e8aca17" containerID="4ba3b416258b280f8549321064351406c4a4ac23b9c992ab6358b0b0442cb550" exitCode=0
Sep 19 04:29:35.900799 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:29:35.900383    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-7hchz"
Sep 19 04:29:35.900799 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:29:35.900419    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-7hchz" event={"ID":"898ce840-1854-4773-9ef1-b2b22e8aca17","Type":"ContainerDied","Data":"4ba3b416258b280f8549321064351406c4a4ac23b9c992ab6358b0b0442cb550"}
Sep 19 04:29:35.900799 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:29:35.900461    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-7hchz" event={"ID":"898ce840-1854-4773-9ef1-b2b22e8aca17","Type":"ContainerDied","Data":"670d709f2fedae582b8cfd6f0d5291dbe311a3dd81146a80aa45fbd7b1cd8a8b"}
Sep 19 04:29:35.900799 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:29:35.900477    2579 scope.go:117] "RemoveContainer" containerID="4ba3b416258b280f8549321064351406c4a4ac23b9c992ab6358b0b0442cb550"
Sep 19 04:29:35.911463 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:29:35.911435    2579 scope.go:117] "RemoveContainer" containerID="4ba3b416258b280f8549321064351406c4a4ac23b9c992ab6358b0b0442cb550"
Sep 19 04:29:35.912011 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:29:35.911987    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ba3b416258b280f8549321064351406c4a4ac23b9c992ab6358b0b0442cb550\": container with ID starting with 4ba3b416258b280f8549321064351406c4a4ac23b9c992ab6358b0b0442cb550 not found: ID does not exist" containerID="4ba3b416258b280f8549321064351406c4a4ac23b9c992ab6358b0b0442cb550"
Sep 19 04:29:35.912099 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:29:35.912019    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ba3b416258b280f8549321064351406c4a4ac23b9c992ab6358b0b0442cb550"} err="failed to get container status \"4ba3b416258b280f8549321064351406c4a4ac23b9c992ab6358b0b0442cb550\": rpc error: code = NotFound desc = could not find container \"4ba3b416258b280f8549321064351406c4a4ac23b9c992ab6358b0b0442cb550\": container with ID starting with 4ba3b416258b280f8549321064351406c4a4ac23b9c992ab6358b0b0442cb550 not found: ID does not exist"
Sep 19 04:29:35.922280 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:29:35.922246    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-7hchz"]
Sep 19 04:29:35.929617 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:29:35.929589    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-7hchz"]
Sep 19 04:29:37.351144 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:29:37.351110    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="898ce840-1854-4773-9ef1-b2b22e8aca17" path="/var/lib/kubelet/pods/898ce840-1854-4773-9ef1-b2b22e8aca17/volumes"
Sep 19 04:30:06.529809 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:06.529722    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7c6df5dfb9-gxtf7"]
Sep 19 04:30:06.530272 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:06.530037    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="898ce840-1854-4773-9ef1-b2b22e8aca17" containerName="authorino"
Sep 19 04:30:06.530272 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:06.530048    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="898ce840-1854-4773-9ef1-b2b22e8aca17" containerName="authorino"
Sep 19 04:30:06.530272 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:06.530111    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="898ce840-1854-4773-9ef1-b2b22e8aca17" containerName="authorino"
Sep 19 04:30:06.533001 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:06.532983    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7c6df5dfb9-gxtf7"
Sep 19 04:30:06.536422 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:06.536398    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 19 04:30:06.536422 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:06.536411    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-sfrkl\""
Sep 19 04:30:06.536644 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:06.536412    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 19 04:30:06.537513 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:06.537496    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 19 04:30:06.547050 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:06.547022    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7c6df5dfb9-gxtf7"]
Sep 19 04:30:06.625796 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:06.625754    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbb5m\" (UniqueName: \"kubernetes.io/projected/134b42ac-1d50-433b-8aed-23e296c6e35e-kube-api-access-kbb5m\") pod \"maas-controller-7c6df5dfb9-gxtf7\" (UID: \"134b42ac-1d50-433b-8aed-23e296c6e35e\") " pod="opendatahub/maas-controller-7c6df5dfb9-gxtf7"
Sep 19 04:30:06.625796 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:06.625800    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/134b42ac-1d50-433b-8aed-23e296c6e35e-metrics-certs\") pod \"maas-controller-7c6df5dfb9-gxtf7\" (UID: \"134b42ac-1d50-433b-8aed-23e296c6e35e\") " pod="opendatahub/maas-controller-7c6df5dfb9-gxtf7"
Sep 19 04:30:06.626006 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:06.625820    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/134b42ac-1d50-433b-8aed-23e296c6e35e-webhook-cert\") pod \"maas-controller-7c6df5dfb9-gxtf7\" (UID: \"134b42ac-1d50-433b-8aed-23e296c6e35e\") " pod="opendatahub/maas-controller-7c6df5dfb9-gxtf7"
Sep 19 04:30:06.712231 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:06.712197    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7bf95c4bb8-r9rcf"]
Sep 19 04:30:06.715618 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:06.715597    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7bf95c4bb8-r9rcf"
Sep 19 04:30:06.726790 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:06.726751    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/134b42ac-1d50-433b-8aed-23e296c6e35e-metrics-certs\") pod \"maas-controller-7c6df5dfb9-gxtf7\" (UID: \"134b42ac-1d50-433b-8aed-23e296c6e35e\") " pod="opendatahub/maas-controller-7c6df5dfb9-gxtf7"
Sep 19 04:30:06.726918 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:06.726802    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/134b42ac-1d50-433b-8aed-23e296c6e35e-webhook-cert\") pod \"maas-controller-7c6df5dfb9-gxtf7\" (UID: \"134b42ac-1d50-433b-8aed-23e296c6e35e\") " pod="opendatahub/maas-controller-7c6df5dfb9-gxtf7"
Sep 19 04:30:06.726918 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:30:06.726906    2579 secret.go:189] Couldn't get secret opendatahub/maas-controller-metrics-tls: secret "maas-controller-metrics-tls" not found
Sep 19 04:30:06.726996 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:30:06.726975    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/134b42ac-1d50-433b-8aed-23e296c6e35e-metrics-certs podName:134b42ac-1d50-433b-8aed-23e296c6e35e nodeName:}" failed. No retries permitted until 2026-09-19 04:30:07.226954161 +0000 UTC m=+986.409915303 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/134b42ac-1d50-433b-8aed-23e296c6e35e-metrics-certs") pod "maas-controller-7c6df5dfb9-gxtf7" (UID: "134b42ac-1d50-433b-8aed-23e296c6e35e") : secret "maas-controller-metrics-tls" not found
Sep 19 04:30:06.726996 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:06.726912    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbb5m\" (UniqueName: \"kubernetes.io/projected/134b42ac-1d50-433b-8aed-23e296c6e35e-kube-api-access-kbb5m\") pod \"maas-controller-7c6df5dfb9-gxtf7\" (UID: \"134b42ac-1d50-433b-8aed-23e296c6e35e\") " pod="opendatahub/maas-controller-7c6df5dfb9-gxtf7"
Sep 19 04:30:06.727169 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:30:06.727140    2579 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Sep 19 04:30:06.727246 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:30:06.727215    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/134b42ac-1d50-433b-8aed-23e296c6e35e-webhook-cert podName:134b42ac-1d50-433b-8aed-23e296c6e35e nodeName:}" failed. No retries permitted until 2026-09-19 04:30:07.227196921 +0000 UTC m=+986.410158047 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/134b42ac-1d50-433b-8aed-23e296c6e35e-webhook-cert") pod "maas-controller-7c6df5dfb9-gxtf7" (UID: "134b42ac-1d50-433b-8aed-23e296c6e35e") : secret "maas-controller-webhook-cert" not found
Sep 19 04:30:06.728149 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:06.728125    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7bf95c4bb8-r9rcf"]
Sep 19 04:30:06.739843 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:06.739808    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbb5m\" (UniqueName: \"kubernetes.io/projected/134b42ac-1d50-433b-8aed-23e296c6e35e-kube-api-access-kbb5m\") pod \"maas-controller-7c6df5dfb9-gxtf7\" (UID: \"134b42ac-1d50-433b-8aed-23e296c6e35e\") " pod="opendatahub/maas-controller-7c6df5dfb9-gxtf7"
Sep 19 04:30:06.827651 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:06.827560    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbcpt\" (UniqueName: \"kubernetes.io/projected/6d5aa4b7-5d5f-4ce9-8ac5-79b0005ea2d3-kube-api-access-bbcpt\") pod \"maas-controller-7bf95c4bb8-r9rcf\" (UID: \"6d5aa4b7-5d5f-4ce9-8ac5-79b0005ea2d3\") " pod="opendatahub/maas-controller-7bf95c4bb8-r9rcf"
Sep 19 04:30:06.827651 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:06.827609    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6d5aa4b7-5d5f-4ce9-8ac5-79b0005ea2d3-webhook-cert\") pod \"maas-controller-7bf95c4bb8-r9rcf\" (UID: \"6d5aa4b7-5d5f-4ce9-8ac5-79b0005ea2d3\") " pod="opendatahub/maas-controller-7bf95c4bb8-r9rcf"
Sep 19 04:30:06.827651 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:06.827634    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d5aa4b7-5d5f-4ce9-8ac5-79b0005ea2d3-metrics-certs\") pod \"maas-controller-7bf95c4bb8-r9rcf\" (UID: \"6d5aa4b7-5d5f-4ce9-8ac5-79b0005ea2d3\") " pod="opendatahub/maas-controller-7bf95c4bb8-r9rcf"
Sep 19 04:30:06.928328 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:06.928281    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6d5aa4b7-5d5f-4ce9-8ac5-79b0005ea2d3-webhook-cert\") pod \"maas-controller-7bf95c4bb8-r9rcf\" (UID: \"6d5aa4b7-5d5f-4ce9-8ac5-79b0005ea2d3\") " pod="opendatahub/maas-controller-7bf95c4bb8-r9rcf"
Sep 19 04:30:06.928328 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:06.928338    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d5aa4b7-5d5f-4ce9-8ac5-79b0005ea2d3-metrics-certs\") pod \"maas-controller-7bf95c4bb8-r9rcf\" (UID: \"6d5aa4b7-5d5f-4ce9-8ac5-79b0005ea2d3\") " pod="opendatahub/maas-controller-7bf95c4bb8-r9rcf"
Sep 19 04:30:06.928555 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:06.928442    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbcpt\" (UniqueName: \"kubernetes.io/projected/6d5aa4b7-5d5f-4ce9-8ac5-79b0005ea2d3-kube-api-access-bbcpt\") pod \"maas-controller-7bf95c4bb8-r9rcf\" (UID: \"6d5aa4b7-5d5f-4ce9-8ac5-79b0005ea2d3\") " pod="opendatahub/maas-controller-7bf95c4bb8-r9rcf"
Sep 19 04:30:06.930941 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:06.930917    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d5aa4b7-5d5f-4ce9-8ac5-79b0005ea2d3-metrics-certs\") pod \"maas-controller-7bf95c4bb8-r9rcf\" (UID: \"6d5aa4b7-5d5f-4ce9-8ac5-79b0005ea2d3\") " pod="opendatahub/maas-controller-7bf95c4bb8-r9rcf"
Sep 19 04:30:06.931059 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:06.930952    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6d5aa4b7-5d5f-4ce9-8ac5-79b0005ea2d3-webhook-cert\") pod \"maas-controller-7bf95c4bb8-r9rcf\" (UID: \"6d5aa4b7-5d5f-4ce9-8ac5-79b0005ea2d3\") " pod="opendatahub/maas-controller-7bf95c4bb8-r9rcf"
Sep 19 04:30:06.937916 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:06.937896    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbcpt\" (UniqueName: \"kubernetes.io/projected/6d5aa4b7-5d5f-4ce9-8ac5-79b0005ea2d3-kube-api-access-bbcpt\") pod \"maas-controller-7bf95c4bb8-r9rcf\" (UID: \"6d5aa4b7-5d5f-4ce9-8ac5-79b0005ea2d3\") " pod="opendatahub/maas-controller-7bf95c4bb8-r9rcf"
Sep 19 04:30:07.025871 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:07.025834    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7bf95c4bb8-r9rcf"
Sep 19 04:30:07.230629 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:07.230581    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/134b42ac-1d50-433b-8aed-23e296c6e35e-metrics-certs\") pod \"maas-controller-7c6df5dfb9-gxtf7\" (UID: \"134b42ac-1d50-433b-8aed-23e296c6e35e\") " pod="opendatahub/maas-controller-7c6df5dfb9-gxtf7"
Sep 19 04:30:07.230629 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:07.230644    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/134b42ac-1d50-433b-8aed-23e296c6e35e-webhook-cert\") pod \"maas-controller-7c6df5dfb9-gxtf7\" (UID: \"134b42ac-1d50-433b-8aed-23e296c6e35e\") " pod="opendatahub/maas-controller-7c6df5dfb9-gxtf7"
Sep 19 04:30:07.233322 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:07.233293    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/134b42ac-1d50-433b-8aed-23e296c6e35e-webhook-cert\") pod \"maas-controller-7c6df5dfb9-gxtf7\" (UID: \"134b42ac-1d50-433b-8aed-23e296c6e35e\") " pod="opendatahub/maas-controller-7c6df5dfb9-gxtf7"
Sep 19 04:30:07.233429 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:07.233322    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/134b42ac-1d50-433b-8aed-23e296c6e35e-metrics-certs\") pod \"maas-controller-7c6df5dfb9-gxtf7\" (UID: \"134b42ac-1d50-433b-8aed-23e296c6e35e\") " pod="opendatahub/maas-controller-7c6df5dfb9-gxtf7"
Sep 19 04:30:07.301741 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:07.301715    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7bf95c4bb8-r9rcf"]
Sep 19 04:30:07.303678 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:30:07.303653    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d5aa4b7_5d5f_4ce9_8ac5_79b0005ea2d3.slice/crio-24f1117692d968d03ecf5636334f1cc1f9577295fc57aa840bd3b994ce9b087e WatchSource:0}: Error finding container 24f1117692d968d03ecf5636334f1cc1f9577295fc57aa840bd3b994ce9b087e: Status 404 returned error can't find the container with id 24f1117692d968d03ecf5636334f1cc1f9577295fc57aa840bd3b994ce9b087e
Sep 19 04:30:07.443488 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:07.443456    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7c6df5dfb9-gxtf7"
Sep 19 04:30:07.720268 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:07.720170    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7c6df5dfb9-gxtf7"]
Sep 19 04:30:07.725899 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:30:07.725872    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod134b42ac_1d50_433b_8aed_23e296c6e35e.slice/crio-f361713178ca991fcfa4b66275e624eaa56827d47f486dac2455509e9077d69c WatchSource:0}: Error finding container f361713178ca991fcfa4b66275e624eaa56827d47f486dac2455509e9077d69c: Status 404 returned error can't find the container with id f361713178ca991fcfa4b66275e624eaa56827d47f486dac2455509e9077d69c
Sep 19 04:30:08.059768 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:08.059657    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7c6df5dfb9-gxtf7" event={"ID":"134b42ac-1d50-433b-8aed-23e296c6e35e","Type":"ContainerStarted","Data":"f361713178ca991fcfa4b66275e624eaa56827d47f486dac2455509e9077d69c"}
Sep 19 04:30:08.061052 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:08.061014    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7bf95c4bb8-r9rcf" event={"ID":"6d5aa4b7-5d5f-4ce9-8ac5-79b0005ea2d3","Type":"ContainerStarted","Data":"24f1117692d968d03ecf5636334f1cc1f9577295fc57aa840bd3b994ce9b087e"}
Sep 19 04:30:10.074927 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:10.074875    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7c6df5dfb9-gxtf7" event={"ID":"134b42ac-1d50-433b-8aed-23e296c6e35e","Type":"ContainerStarted","Data":"007fc523ce2d6badc7d41098db5a39b2d42db392ac4e742f33f3a47481d9cb84"}
Sep 19 04:30:10.075388 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:10.074964    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7c6df5dfb9-gxtf7"
Sep 19 04:30:10.076384 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:10.076359    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7bf95c4bb8-r9rcf" event={"ID":"6d5aa4b7-5d5f-4ce9-8ac5-79b0005ea2d3","Type":"ContainerStarted","Data":"b94a8accb5e0adac00948d60ace8a1b903a2ed8b08970ef9a2219e8467378699"}
Sep 19 04:30:10.076515 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:10.076476    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7bf95c4bb8-r9rcf"
Sep 19 04:30:10.096080 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:10.096016    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7c6df5dfb9-gxtf7" podStartSLOduration=2.7796464199999997 podStartE2EDuration="4.095999366s" totalImagesPullingTime="1.316352946s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 04:30:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:30:10.093953417 +0000 UTC m=+989.276914576" watchObservedRunningTime="2026-09-19 04:30:10.095999366 +0000 UTC m=+989.278960575"
Sep 19 04:30:10.115161 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:10.115100    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7bf95c4bb8-r9rcf" podStartSLOduration=2.380018869 podStartE2EDuration="4.11508224s" totalImagesPullingTime="1.735063371s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 04:30:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:30:10.113915544 +0000 UTC m=+989.296876692" watchObservedRunningTime="2026-09-19 04:30:10.11508224 +0000 UTC m=+989.298043389"
Sep 19 04:30:21.083885 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:21.083846    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7bf95c4bb8-r9rcf"
Sep 19 04:30:21.084379 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:21.083935    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7c6df5dfb9-gxtf7"
Sep 19 04:30:21.162110 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:21.162076    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7c6df5dfb9-gxtf7"]
Sep 19 04:30:21.162322 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:21.162294    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7c6df5dfb9-gxtf7" podUID="134b42ac-1d50-433b-8aed-23e296c6e35e" containerName="manager" containerID="cri-o://007fc523ce2d6badc7d41098db5a39b2d42db392ac4e742f33f3a47481d9cb84" gracePeriod=30
Sep 19 04:30:21.413380 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:21.413352    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7c6df5dfb9-gxtf7"
Sep 19 04:30:21.559979 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:21.559942    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/134b42ac-1d50-433b-8aed-23e296c6e35e-metrics-certs\") pod \"134b42ac-1d50-433b-8aed-23e296c6e35e\" (UID: \"134b42ac-1d50-433b-8aed-23e296c6e35e\") "
Sep 19 04:30:21.560175 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:21.559996    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/134b42ac-1d50-433b-8aed-23e296c6e35e-webhook-cert\") pod \"134b42ac-1d50-433b-8aed-23e296c6e35e\" (UID: \"134b42ac-1d50-433b-8aed-23e296c6e35e\") "
Sep 19 04:30:21.560246 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:21.560170    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kbb5m\" (UniqueName: \"kubernetes.io/projected/134b42ac-1d50-433b-8aed-23e296c6e35e-kube-api-access-kbb5m\") pod \"134b42ac-1d50-433b-8aed-23e296c6e35e\" (UID: \"134b42ac-1d50-433b-8aed-23e296c6e35e\") "
Sep 19 04:30:21.562685 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:21.562653    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/134b42ac-1d50-433b-8aed-23e296c6e35e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "134b42ac-1d50-433b-8aed-23e296c6e35e" (UID: "134b42ac-1d50-433b-8aed-23e296c6e35e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 04:30:21.562890 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:21.562801    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/134b42ac-1d50-433b-8aed-23e296c6e35e-kube-api-access-kbb5m" (OuterVolumeSpecName: "kube-api-access-kbb5m") pod "134b42ac-1d50-433b-8aed-23e296c6e35e" (UID: "134b42ac-1d50-433b-8aed-23e296c6e35e"). InnerVolumeSpecName "kube-api-access-kbb5m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 04:30:21.562987 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:21.562878    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/134b42ac-1d50-433b-8aed-23e296c6e35e-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "134b42ac-1d50-433b-8aed-23e296c6e35e" (UID: "134b42ac-1d50-433b-8aed-23e296c6e35e"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 04:30:21.661617 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:21.661581    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbb5m\" (UniqueName: \"kubernetes.io/projected/134b42ac-1d50-433b-8aed-23e296c6e35e-kube-api-access-kbb5m\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:30:21.661617 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:21.661611    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/134b42ac-1d50-433b-8aed-23e296c6e35e-metrics-certs\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:30:21.661617 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:21.661620    2579 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/134b42ac-1d50-433b-8aed-23e296c6e35e-webhook-cert\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:30:22.141311 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:22.141226    2579 generic.go:356] "Generic (PLEG): container finished" podID="134b42ac-1d50-433b-8aed-23e296c6e35e" containerID="007fc523ce2d6badc7d41098db5a39b2d42db392ac4e742f33f3a47481d9cb84" exitCode=0
Sep 19 04:30:22.141311 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:22.141264    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7c6df5dfb9-gxtf7"
Sep 19 04:30:22.141311 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:22.141296    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7c6df5dfb9-gxtf7" event={"ID":"134b42ac-1d50-433b-8aed-23e296c6e35e","Type":"ContainerDied","Data":"007fc523ce2d6badc7d41098db5a39b2d42db392ac4e742f33f3a47481d9cb84"}
Sep 19 04:30:22.141834 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:22.141342    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7c6df5dfb9-gxtf7" event={"ID":"134b42ac-1d50-433b-8aed-23e296c6e35e","Type":"ContainerDied","Data":"f361713178ca991fcfa4b66275e624eaa56827d47f486dac2455509e9077d69c"}
Sep 19 04:30:22.141834 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:22.141367    2579 scope.go:117] "RemoveContainer" containerID="007fc523ce2d6badc7d41098db5a39b2d42db392ac4e742f33f3a47481d9cb84"
Sep 19 04:30:22.150681 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:22.150661    2579 scope.go:117] "RemoveContainer" containerID="007fc523ce2d6badc7d41098db5a39b2d42db392ac4e742f33f3a47481d9cb84"
Sep 19 04:30:22.151016 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:30:22.150996    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"007fc523ce2d6badc7d41098db5a39b2d42db392ac4e742f33f3a47481d9cb84\": container with ID starting with 007fc523ce2d6badc7d41098db5a39b2d42db392ac4e742f33f3a47481d9cb84 not found: ID does not exist" containerID="007fc523ce2d6badc7d41098db5a39b2d42db392ac4e742f33f3a47481d9cb84"
Sep 19 04:30:22.151092 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:22.151025    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"007fc523ce2d6badc7d41098db5a39b2d42db392ac4e742f33f3a47481d9cb84"} err="failed to get container status \"007fc523ce2d6badc7d41098db5a39b2d42db392ac4e742f33f3a47481d9cb84\": rpc error: code = NotFound desc = could not find container \"007fc523ce2d6badc7d41098db5a39b2d42db392ac4e742f33f3a47481d9cb84\": container with ID starting with 007fc523ce2d6badc7d41098db5a39b2d42db392ac4e742f33f3a47481d9cb84 not found: ID does not exist"
Sep 19 04:30:22.165210 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:22.165180    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7c6df5dfb9-gxtf7"]
Sep 19 04:30:22.168390 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:22.168367    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7c6df5dfb9-gxtf7"]
Sep 19 04:30:23.351191 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:23.351157    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="134b42ac-1d50-433b-8aed-23e296c6e35e" path="/var/lib/kubelet/pods/134b42ac-1d50-433b-8aed-23e296c6e35e/volumes"
Sep 19 04:30:43.101491 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:43.101449    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-8wv77"]
Sep 19 04:30:43.102023 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:43.102002    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="134b42ac-1d50-433b-8aed-23e296c6e35e" containerName="manager"
Sep 19 04:30:43.102122 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:43.102025    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="134b42ac-1d50-433b-8aed-23e296c6e35e" containerName="manager"
Sep 19 04:30:43.102179 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:43.102127    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="134b42ac-1d50-433b-8aed-23e296c6e35e" containerName="manager"
Sep 19 04:30:43.108365 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:43.108327    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-8wv77"
Sep 19 04:30:43.111432 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:43.111408    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 19 04:30:43.111554 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:43.111415    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-ffv42\""
Sep 19 04:30:43.120984 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:43.120950    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-8wv77"]
Sep 19 04:30:43.126220 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:43.126179    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-sjhch"]
Sep 19 04:30:43.130884 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:43.130852    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-sjhch"
Sep 19 04:30:43.139959 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:43.139935    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98f8b146-35a4-45fa-9fa7-29ce6d46902e-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-8wv77\" (UID: \"98f8b146-35a4-45fa-9fa7-29ce6d46902e\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-8wv77"
Sep 19 04:30:43.140099 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:43.140004    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsqtj\" (UniqueName: \"kubernetes.io/projected/98f8b146-35a4-45fa-9fa7-29ce6d46902e-kube-api-access-zsqtj\") pod \"payload-pre-processing-59fbc47bf8-8wv77\" (UID: \"98f8b146-35a4-45fa-9fa7-29ce6d46902e\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-8wv77"
Sep 19 04:30:43.140099 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:43.140087    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-444ht\" (UniqueName: \"kubernetes.io/projected/fa5e4819-0ed6-44af-a94a-c3529a77ef4b-kube-api-access-444ht\") pod \"payload-processing-5b54b7fbb7-sjhch\" (UID: \"fa5e4819-0ed6-44af-a94a-c3529a77ef4b\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-sjhch"
Sep 19 04:30:43.140218 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:43.140173    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa5e4819-0ed6-44af-a94a-c3529a77ef4b-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-sjhch\" (UID: \"fa5e4819-0ed6-44af-a94a-c3529a77ef4b\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-sjhch"
Sep 19 04:30:43.145438 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:43.145376    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-sjhch"]
Sep 19 04:30:43.241132 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:43.241096    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa5e4819-0ed6-44af-a94a-c3529a77ef4b-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-sjhch\" (UID: \"fa5e4819-0ed6-44af-a94a-c3529a77ef4b\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-sjhch"
Sep 19 04:30:43.241311 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:43.241207    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98f8b146-35a4-45fa-9fa7-29ce6d46902e-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-8wv77\" (UID: \"98f8b146-35a4-45fa-9fa7-29ce6d46902e\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-8wv77"
Sep 19 04:30:43.241311 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:43.241234    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsqtj\" (UniqueName: \"kubernetes.io/projected/98f8b146-35a4-45fa-9fa7-29ce6d46902e-kube-api-access-zsqtj\") pod \"payload-pre-processing-59fbc47bf8-8wv77\" (UID: \"98f8b146-35a4-45fa-9fa7-29ce6d46902e\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-8wv77"
Sep 19 04:30:43.241311 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:43.241261    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-444ht\" (UniqueName: \"kubernetes.io/projected/fa5e4819-0ed6-44af-a94a-c3529a77ef4b-kube-api-access-444ht\") pod \"payload-processing-5b54b7fbb7-sjhch\" (UID: \"fa5e4819-0ed6-44af-a94a-c3529a77ef4b\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-sjhch"
Sep 19 04:30:43.241980 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:43.241959    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa5e4819-0ed6-44af-a94a-c3529a77ef4b-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-sjhch\" (UID: \"fa5e4819-0ed6-44af-a94a-c3529a77ef4b\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-sjhch"
Sep 19 04:30:43.242157 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:43.242135    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98f8b146-35a4-45fa-9fa7-29ce6d46902e-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-8wv77\" (UID: \"98f8b146-35a4-45fa-9fa7-29ce6d46902e\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-8wv77"
Sep 19 04:30:43.254375 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:43.254340    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-444ht\" (UniqueName: \"kubernetes.io/projected/fa5e4819-0ed6-44af-a94a-c3529a77ef4b-kube-api-access-444ht\") pod \"payload-processing-5b54b7fbb7-sjhch\" (UID: \"fa5e4819-0ed6-44af-a94a-c3529a77ef4b\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-sjhch"
Sep 19 04:30:43.254560 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:43.254536    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsqtj\" (UniqueName: \"kubernetes.io/projected/98f8b146-35a4-45fa-9fa7-29ce6d46902e-kube-api-access-zsqtj\") pod \"payload-pre-processing-59fbc47bf8-8wv77\" (UID: \"98f8b146-35a4-45fa-9fa7-29ce6d46902e\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-8wv77"
Sep 19 04:30:43.420449 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:43.420408    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-8wv77"
Sep 19 04:30:43.444881 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:43.444808    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-sjhch"
Sep 19 04:30:43.741977 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:43.741826    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-8wv77"]
Sep 19 04:30:43.747013 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:30:43.746964    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98f8b146_35a4_45fa_9fa7_29ce6d46902e.slice/crio-595a89d8940d9db6924dea2550ce7c21133f9fbafa1af1381a28bce0d54d7240 WatchSource:0}: Error finding container 595a89d8940d9db6924dea2550ce7c21133f9fbafa1af1381a28bce0d54d7240: Status 404 returned error can't find the container with id 595a89d8940d9db6924dea2550ce7c21133f9fbafa1af1381a28bce0d54d7240
Sep 19 04:30:43.768120 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:43.768034    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-sjhch"]
Sep 19 04:30:43.770314 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:30:43.770283    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa5e4819_0ed6_44af_a94a_c3529a77ef4b.slice/crio-87d4d9b3aa88519a323e92194b8a390392a05c5a6a5aa3f8cefb7a0a9dc680ae WatchSource:0}: Error finding container 87d4d9b3aa88519a323e92194b8a390392a05c5a6a5aa3f8cefb7a0a9dc680ae: Status 404 returned error can't find the container with id 87d4d9b3aa88519a323e92194b8a390392a05c5a6a5aa3f8cefb7a0a9dc680ae
Sep 19 04:30:44.255605 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:44.255565    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-sjhch" event={"ID":"fa5e4819-0ed6-44af-a94a-c3529a77ef4b","Type":"ContainerStarted","Data":"87d4d9b3aa88519a323e92194b8a390392a05c5a6a5aa3f8cefb7a0a9dc680ae"}
Sep 19 04:30:44.257160 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:44.257131    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-8wv77" event={"ID":"98f8b146-35a4-45fa-9fa7-29ce6d46902e","Type":"ContainerStarted","Data":"595a89d8940d9db6924dea2550ce7c21133f9fbafa1af1381a28bce0d54d7240"}
Sep 19 04:30:46.270020 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:46.269983    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-8wv77" event={"ID":"98f8b146-35a4-45fa-9fa7-29ce6d46902e","Type":"ContainerStarted","Data":"63e69fd30f8c505e0c9afc2c5061224fe72f16468ab5259f3c21f7d979c50dc7"}
Sep 19 04:30:46.270503 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:46.270059    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-8wv77"
Sep 19 04:30:46.271619 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:46.271586    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-sjhch" event={"ID":"fa5e4819-0ed6-44af-a94a-c3529a77ef4b","Type":"ContainerStarted","Data":"be2183d63b4acc86aaf98c2727552f8bfcf7d8960aed0f51467bd4c1cbe06bb2"}
Sep 19 04:30:46.271767 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:46.271746    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-sjhch"
Sep 19 04:30:46.292193 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:46.292128    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-8wv77" podStartSLOduration=1.707367533 podStartE2EDuration="3.292104267s" totalImagesPullingTime="1.584736734s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 04:30:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:30:46.290516645 +0000 UTC m=+1025.473477794" watchObservedRunningTime="2026-09-19 04:30:46.292104267 +0000 UTC m=+1025.475065416"
Sep 19 04:30:46.314739 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:46.314655    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-sjhch" podStartSLOduration=1.754870946 podStartE2EDuration="3.314637521s" totalImagesPullingTime="1.559766575s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 04:30:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:30:46.313618249 +0000 UTC m=+1025.496579426" watchObservedRunningTime="2026-09-19 04:30:46.314637521 +0000 UTC m=+1025.497598673"
Sep 19 04:30:52.306653 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:52.306617    2579 generic.go:356] "Generic (PLEG): container finished" podID="2130d608-c9dd-4f97-8348-bb1a10b7fec5" containerID="a1845151cc487d39d5e061bc7b6a9130ea5a4acd6456e9ed6457f094aa539d1b" exitCode=0
Sep 19 04:30:52.307113 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:52.306721    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jkrrv" event={"ID":"2130d608-c9dd-4f97-8348-bb1a10b7fec5","Type":"ContainerDied","Data":"a1845151cc487d39d5e061bc7b6a9130ea5a4acd6456e9ed6457f094aa539d1b"}
Sep 19 04:30:52.307171 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:52.307118    2579 scope.go:117] "RemoveContainer" containerID="a1845151cc487d39d5e061bc7b6a9130ea5a4acd6456e9ed6457f094aa539d1b"
Sep 19 04:30:52.308105 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:52.308067    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 04:30:52.308211 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:52.308150    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 04:30:52.308211 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:52.308202    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 04:30:52.591876 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:52.591779    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-pmvf4"]
Sep 19 04:30:52.595137 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:52.595112    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-pmvf4"
Sep 19 04:30:52.611491 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:52.611460    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-pmvf4"]
Sep 19 04:30:52.624452 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:52.624416    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nz685\" (UniqueName: \"kubernetes.io/projected/925187fb-6294-4733-afb7-72288f63e4d8-kube-api-access-nz685\") pod \"authorino-557f876f69-pmvf4\" (UID: \"925187fb-6294-4733-afb7-72288f63e4d8\") " pod="kuadrant-system/authorino-557f876f69-pmvf4"
Sep 19 04:30:52.725947 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:52.725909    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz685\" (UniqueName: \"kubernetes.io/projected/925187fb-6294-4733-afb7-72288f63e4d8-kube-api-access-nz685\") pod \"authorino-557f876f69-pmvf4\" (UID: \"925187fb-6294-4733-afb7-72288f63e4d8\") " pod="kuadrant-system/authorino-557f876f69-pmvf4"
Sep 19 04:30:52.765809 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:52.765779    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz685\" (UniqueName: \"kubernetes.io/projected/925187fb-6294-4733-afb7-72288f63e4d8-kube-api-access-nz685\") pod \"authorino-557f876f69-pmvf4\" (UID: \"925187fb-6294-4733-afb7-72288f63e4d8\") " pod="kuadrant-system/authorino-557f876f69-pmvf4"
Sep 19 04:30:52.807720 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:52.806418    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-pmvf4"]
Sep 19 04:30:52.807720 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:52.806786    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-pmvf4"
Sep 19 04:30:52.856993 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:52.856921    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-8zdt8"]
Sep 19 04:30:52.862112 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:52.862089    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-8zdt8"
Sep 19 04:30:52.866510 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:52.866468    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 19 04:30:52.918999 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:52.918890    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-8zdt8"]
Sep 19 04:30:52.928573 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:52.928537    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c7da955f-61a3-48c8-be9c-2b2b3b6cab45-tls-cert\") pod \"authorino-6d4dfc649f-8zdt8\" (UID: \"c7da955f-61a3-48c8-be9c-2b2b3b6cab45\") " pod="kuadrant-system/authorino-6d4dfc649f-8zdt8"
Sep 19 04:30:52.928775 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:52.928607    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjn79\" (UniqueName: \"kubernetes.io/projected/c7da955f-61a3-48c8-be9c-2b2b3b6cab45-kube-api-access-cjn79\") pod \"authorino-6d4dfc649f-8zdt8\" (UID: \"c7da955f-61a3-48c8-be9c-2b2b3b6cab45\") " pod="kuadrant-system/authorino-6d4dfc649f-8zdt8"
Sep 19 04:30:52.944599 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:52.944561    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-8zdt8"]
Sep 19 04:30:52.945018 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:30:52.944968    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-cjn79 tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-8zdt8" podUID="c7da955f-61a3-48c8-be9c-2b2b3b6cab45"
Sep 19 04:30:53.020406 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:53.020368    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6b68f67659-vrtkf"]
Sep 19 04:30:53.023911 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:53.023890    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6b68f67659-vrtkf"
Sep 19 04:30:53.029093 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:53.029067    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjn79\" (UniqueName: \"kubernetes.io/projected/c7da955f-61a3-48c8-be9c-2b2b3b6cab45-kube-api-access-cjn79\") pod \"authorino-6d4dfc649f-8zdt8\" (UID: \"c7da955f-61a3-48c8-be9c-2b2b3b6cab45\") " pod="kuadrant-system/authorino-6d4dfc649f-8zdt8"
Sep 19 04:30:53.029234 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:53.029155    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c7da955f-61a3-48c8-be9c-2b2b3b6cab45-tls-cert\") pod \"authorino-6d4dfc649f-8zdt8\" (UID: \"c7da955f-61a3-48c8-be9c-2b2b3b6cab45\") " pod="kuadrant-system/authorino-6d4dfc649f-8zdt8"
Sep 19 04:30:53.031886 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:53.031866    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c7da955f-61a3-48c8-be9c-2b2b3b6cab45-tls-cert\") pod \"authorino-6d4dfc649f-8zdt8\" (UID: \"c7da955f-61a3-48c8-be9c-2b2b3b6cab45\") " pod="kuadrant-system/authorino-6d4dfc649f-8zdt8"
Sep 19 04:30:53.043003 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:53.042975    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjn79\" (UniqueName: \"kubernetes.io/projected/c7da955f-61a3-48c8-be9c-2b2b3b6cab45-kube-api-access-cjn79\") pod \"authorino-6d4dfc649f-8zdt8\" (UID: \"c7da955f-61a3-48c8-be9c-2b2b3b6cab45\") " pod="kuadrant-system/authorino-6d4dfc649f-8zdt8"
Sep 19 04:30:53.043903 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:53.043874    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6b68f67659-vrtkf"]
Sep 19 04:30:53.108519 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:53.108447    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-pmvf4"]
Sep 19 04:30:53.109010 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:30:53.108979    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod925187fb_6294_4733_afb7_72288f63e4d8.slice/crio-da733ccd83214c9ef56b9142dd9588b1261f5880f4bba8ca3dcd4f09d61dd3ed WatchSource:0}: Error finding container da733ccd83214c9ef56b9142dd9588b1261f5880f4bba8ca3dcd4f09d61dd3ed: Status 404 returned error can't find the container with id da733ccd83214c9ef56b9142dd9588b1261f5880f4bba8ca3dcd4f09d61dd3ed
Sep 19 04:30:53.130154 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:53.130113    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8v98\" (UniqueName: \"kubernetes.io/projected/6430dec9-9e3b-4244-9280-6b000beea86c-kube-api-access-d8v98\") pod \"authorino-6b68f67659-vrtkf\" (UID: \"6430dec9-9e3b-4244-9280-6b000beea86c\") " pod="kuadrant-system/authorino-6b68f67659-vrtkf"
Sep 19 04:30:53.130323 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:53.130163    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6430dec9-9e3b-4244-9280-6b000beea86c-tls-cert\") pod \"authorino-6b68f67659-vrtkf\" (UID: \"6430dec9-9e3b-4244-9280-6b000beea86c\") " pod="kuadrant-system/authorino-6b68f67659-vrtkf"
Sep 19 04:30:53.230675 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:53.230645    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8v98\" (UniqueName: \"kubernetes.io/projected/6430dec9-9e3b-4244-9280-6b000beea86c-kube-api-access-d8v98\") pod \"authorino-6b68f67659-vrtkf\" (UID: \"6430dec9-9e3b-4244-9280-6b000beea86c\") " pod="kuadrant-system/authorino-6b68f67659-vrtkf"
Sep 19 04:30:53.230835 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:53.230687    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6430dec9-9e3b-4244-9280-6b000beea86c-tls-cert\") pod \"authorino-6b68f67659-vrtkf\" (UID: \"6430dec9-9e3b-4244-9280-6b000beea86c\") " pod="kuadrant-system/authorino-6b68f67659-vrtkf"
Sep 19 04:30:53.233251 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:53.233232    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6430dec9-9e3b-4244-9280-6b000beea86c-tls-cert\") pod \"authorino-6b68f67659-vrtkf\" (UID: \"6430dec9-9e3b-4244-9280-6b000beea86c\") " pod="kuadrant-system/authorino-6b68f67659-vrtkf"
Sep 19 04:30:53.245750 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:53.245720    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8v98\" (UniqueName: \"kubernetes.io/projected/6430dec9-9e3b-4244-9280-6b000beea86c-kube-api-access-d8v98\") pod \"authorino-6b68f67659-vrtkf\" (UID: \"6430dec9-9e3b-4244-9280-6b000beea86c\") " pod="kuadrant-system/authorino-6b68f67659-vrtkf"
Sep 19 04:30:53.314039 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:53.313995    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-pmvf4" event={"ID":"925187fb-6294-4733-afb7-72288f63e4d8","Type":"ContainerStarted","Data":"da733ccd83214c9ef56b9142dd9588b1261f5880f4bba8ca3dcd4f09d61dd3ed"}
Sep 19 04:30:53.316032 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:53.316012    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-8zdt8"
Sep 19 04:30:53.316144 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:53.316052    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jkrrv" event={"ID":"2130d608-c9dd-4f97-8348-bb1a10b7fec5","Type":"ContainerStarted","Data":"618dcea3b773e50144647aecdfaaa9b31f003b479769e4b485ea4acec5f828b9"}
Sep 19 04:30:53.324267 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:53.324248    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-8zdt8"
Sep 19 04:30:53.333197 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:53.333179    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6b68f67659-vrtkf"
Sep 19 04:30:53.432308 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:53.432278    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c7da955f-61a3-48c8-be9c-2b2b3b6cab45-tls-cert\") pod \"c7da955f-61a3-48c8-be9c-2b2b3b6cab45\" (UID: \"c7da955f-61a3-48c8-be9c-2b2b3b6cab45\") "
Sep 19 04:30:53.432486 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:53.432374    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cjn79\" (UniqueName: \"kubernetes.io/projected/c7da955f-61a3-48c8-be9c-2b2b3b6cab45-kube-api-access-cjn79\") pod \"c7da955f-61a3-48c8-be9c-2b2b3b6cab45\" (UID: \"c7da955f-61a3-48c8-be9c-2b2b3b6cab45\") "
Sep 19 04:30:53.434636 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:53.434603    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7da955f-61a3-48c8-be9c-2b2b3b6cab45-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "c7da955f-61a3-48c8-be9c-2b2b3b6cab45" (UID: "c7da955f-61a3-48c8-be9c-2b2b3b6cab45"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 04:30:53.434636 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:53.434619    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7da955f-61a3-48c8-be9c-2b2b3b6cab45-kube-api-access-cjn79" (OuterVolumeSpecName: "kube-api-access-cjn79") pod "c7da955f-61a3-48c8-be9c-2b2b3b6cab45" (UID: "c7da955f-61a3-48c8-be9c-2b2b3b6cab45"). InnerVolumeSpecName "kube-api-access-cjn79". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 04:30:53.533722 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:53.533666    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjn79\" (UniqueName: \"kubernetes.io/projected/c7da955f-61a3-48c8-be9c-2b2b3b6cab45-kube-api-access-cjn79\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:30:53.533722 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:53.533707    2579 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c7da955f-61a3-48c8-be9c-2b2b3b6cab45-tls-cert\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:30:53.631876 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:53.631855    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6b68f67659-vrtkf"]
Sep 19 04:30:53.633881 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:30:53.633847    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6430dec9_9e3b_4244_9280_6b000beea86c.slice/crio-4aeded8d3586095b30f774c5f37afaad0b0fde2e3d96b194f3e9f1898812c8b1 WatchSource:0}: Error finding container 4aeded8d3586095b30f774c5f37afaad0b0fde2e3d96b194f3e9f1898812c8b1: Status 404 returned error can't find the container with id 4aeded8d3586095b30f774c5f37afaad0b0fde2e3d96b194f3e9f1898812c8b1
Sep 19 04:30:53.646871 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:53.646849    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jkrrv"
Sep 19 04:30:53.647923 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:53.647901    2579 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-jkrrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.18:15021/healthz/ready\": dial tcp 10.134.0.18:15021: connect: connection refused" start-of-body=
Sep 19 04:30:53.648013 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:53.647944    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jkrrv" podUID="2130d608-c9dd-4f97-8348-bb1a10b7fec5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.18:15021/healthz/ready\": dial tcp 10.134.0.18:15021: connect: connection refused"
Sep 19 04:30:54.327992 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:54.327888    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-pmvf4" event={"ID":"925187fb-6294-4733-afb7-72288f63e4d8","Type":"ContainerStarted","Data":"3c3cf199b97e161875a5a10f7928b300f91c58f115358478539e46a89e5fcc05"}
Sep 19 04:30:54.328474 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:54.328082    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-557f876f69-pmvf4" podUID="925187fb-6294-4733-afb7-72288f63e4d8" containerName="authorino" containerID="cri-o://3c3cf199b97e161875a5a10f7928b300f91c58f115358478539e46a89e5fcc05" gracePeriod=30
Sep 19 04:30:54.333092 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:54.333058    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6b68f67659-vrtkf" event={"ID":"6430dec9-9e3b-4244-9280-6b000beea86c","Type":"ContainerStarted","Data":"3a1007eec828dbf95d1fa698731e81b4510087daf9507ff5f702296d04a71c49"}
Sep 19 04:30:54.333209 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:54.333123    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6b68f67659-vrtkf" event={"ID":"6430dec9-9e3b-4244-9280-6b000beea86c","Type":"ContainerStarted","Data":"4aeded8d3586095b30f774c5f37afaad0b0fde2e3d96b194f3e9f1898812c8b1"}
Sep 19 04:30:54.333209 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:54.333170    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-8zdt8"
Sep 19 04:30:54.353421 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:54.353354    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-557f876f69-pmvf4" podStartSLOduration=1.622530608 podStartE2EDuration="2.353331015s" totalImagesPullingTime="730.800407ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 04:30:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:30:54.351652596 +0000 UTC m=+1033.534613743" watchObservedRunningTime="2026-09-19 04:30:54.353331015 +0000 UTC m=+1033.536292168"
Sep 19 04:30:54.376458 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:54.376390    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6b68f67659-vrtkf" podStartSLOduration=1.963749545 podStartE2EDuration="2.376367037s" totalImagesPullingTime="412.617492ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 04:30:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:30:54.375446965 +0000 UTC m=+1033.558408113" watchObservedRunningTime="2026-09-19 04:30:54.376367037 +0000 UTC m=+1033.559328184"
Sep 19 04:30:54.417220 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:54.417168    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-rz9bs"]
Sep 19 04:30:54.417789 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:54.417398    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-rz9bs" podUID="7160552c-acfc-41be-9313-fe519bf13786" containerName="authorino" containerID="cri-o://a60075ed936653b044642e9fb91c37b422bd7b80a8ff7e95aaf28b706b5763b6" gracePeriod=30
Sep 19 04:30:54.438207 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:54.435815    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-8zdt8"]
Sep 19 04:30:54.442019 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:54.440951    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-8zdt8"]
Sep 19 04:30:54.443518 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:54.442250    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jkrrv"
Sep 19 04:30:54.646987 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:54.646946    2579 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-jkrrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.18:15021/healthz/ready\": dial tcp 10.134.0.18:15021: connect: connection refused" start-of-body=
Sep 19 04:30:54.647209 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:54.647012    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jkrrv" podUID="2130d608-c9dd-4f97-8348-bb1a10b7fec5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.18:15021/healthz/ready\": dial tcp 10.134.0.18:15021: connect: connection refused"
Sep 19 04:30:54.659840 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:54.659749    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-pmvf4"
Sep 19 04:30:54.729244 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:54.729166    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-rz9bs"
Sep 19 04:30:54.747854 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:54.747740    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nz685\" (UniqueName: \"kubernetes.io/projected/925187fb-6294-4733-afb7-72288f63e4d8-kube-api-access-nz685\") pod \"925187fb-6294-4733-afb7-72288f63e4d8\" (UID: \"925187fb-6294-4733-afb7-72288f63e4d8\") "
Sep 19 04:30:54.747854 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:54.747797    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q4chq\" (UniqueName: \"kubernetes.io/projected/7160552c-acfc-41be-9313-fe519bf13786-kube-api-access-q4chq\") pod \"7160552c-acfc-41be-9313-fe519bf13786\" (UID: \"7160552c-acfc-41be-9313-fe519bf13786\") "
Sep 19 04:30:54.750815 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:54.750779    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7160552c-acfc-41be-9313-fe519bf13786-kube-api-access-q4chq" (OuterVolumeSpecName: "kube-api-access-q4chq") pod "7160552c-acfc-41be-9313-fe519bf13786" (UID: "7160552c-acfc-41be-9313-fe519bf13786"). InnerVolumeSpecName "kube-api-access-q4chq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 04:30:54.750980 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:54.750922    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/925187fb-6294-4733-afb7-72288f63e4d8-kube-api-access-nz685" (OuterVolumeSpecName: "kube-api-access-nz685") pod "925187fb-6294-4733-afb7-72288f63e4d8" (UID: "925187fb-6294-4733-afb7-72288f63e4d8"). InnerVolumeSpecName "kube-api-access-nz685". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 04:30:54.848760 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:54.848725    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nz685\" (UniqueName: \"kubernetes.io/projected/925187fb-6294-4733-afb7-72288f63e4d8-kube-api-access-nz685\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:30:54.848760 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:54.848754    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4chq\" (UniqueName: \"kubernetes.io/projected/7160552c-acfc-41be-9313-fe519bf13786-kube-api-access-q4chq\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:30:55.338413 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:55.338378    2579 generic.go:356] "Generic (PLEG): container finished" podID="7160552c-acfc-41be-9313-fe519bf13786" containerID="a60075ed936653b044642e9fb91c37b422bd7b80a8ff7e95aaf28b706b5763b6" exitCode=0
Sep 19 04:30:55.338913 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:55.338421    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-rz9bs"
Sep 19 04:30:55.338913 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:55.338465    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-rz9bs" event={"ID":"7160552c-acfc-41be-9313-fe519bf13786","Type":"ContainerDied","Data":"a60075ed936653b044642e9fb91c37b422bd7b80a8ff7e95aaf28b706b5763b6"}
Sep 19 04:30:55.338913 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:55.338508    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-rz9bs" event={"ID":"7160552c-acfc-41be-9313-fe519bf13786","Type":"ContainerDied","Data":"b2cd62f6a63b8414291c159b77d15ad92c51ffe3a0c4a02efeda3ad109dd8277"}
Sep 19 04:30:55.338913 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:55.338524    2579 scope.go:117] "RemoveContainer" containerID="a60075ed936653b044642e9fb91c37b422bd7b80a8ff7e95aaf28b706b5763b6"
Sep 19 04:30:55.339983 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:55.339962    2579 generic.go:356] "Generic (PLEG): container finished" podID="925187fb-6294-4733-afb7-72288f63e4d8" containerID="3c3cf199b97e161875a5a10f7928b300f91c58f115358478539e46a89e5fcc05" exitCode=0
Sep 19 04:30:55.340081 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:55.340011    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-pmvf4"
Sep 19 04:30:55.340081 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:55.340047    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-pmvf4" event={"ID":"925187fb-6294-4733-afb7-72288f63e4d8","Type":"ContainerDied","Data":"3c3cf199b97e161875a5a10f7928b300f91c58f115358478539e46a89e5fcc05"}
Sep 19 04:30:55.340168 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:55.340085    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-pmvf4" event={"ID":"925187fb-6294-4733-afb7-72288f63e4d8","Type":"ContainerDied","Data":"da733ccd83214c9ef56b9142dd9588b1261f5880f4bba8ca3dcd4f09d61dd3ed"}
Sep 19 04:30:55.354648 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:55.353842    2579 scope.go:117] "RemoveContainer" containerID="a60075ed936653b044642e9fb91c37b422bd7b80a8ff7e95aaf28b706b5763b6"
Sep 19 04:30:55.354648 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:30:55.354453    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a60075ed936653b044642e9fb91c37b422bd7b80a8ff7e95aaf28b706b5763b6\": container with ID starting with a60075ed936653b044642e9fb91c37b422bd7b80a8ff7e95aaf28b706b5763b6 not found: ID does not exist" containerID="a60075ed936653b044642e9fb91c37b422bd7b80a8ff7e95aaf28b706b5763b6"
Sep 19 04:30:55.354648 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:55.354500    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a60075ed936653b044642e9fb91c37b422bd7b80a8ff7e95aaf28b706b5763b6"} err="failed to get container status \"a60075ed936653b044642e9fb91c37b422bd7b80a8ff7e95aaf28b706b5763b6\": rpc error: code = NotFound desc = could not find container \"a60075ed936653b044642e9fb91c37b422bd7b80a8ff7e95aaf28b706b5763b6\": container with ID starting with a60075ed936653b044642e9fb91c37b422bd7b80a8ff7e95aaf28b706b5763b6 not found: ID does not exist"
Sep 19 04:30:55.354648 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:55.354519    2579 scope.go:117] "RemoveContainer" containerID="3c3cf199b97e161875a5a10f7928b300f91c58f115358478539e46a89e5fcc05"
Sep 19 04:30:55.359446 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:55.359423    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7da955f-61a3-48c8-be9c-2b2b3b6cab45" path="/var/lib/kubelet/pods/c7da955f-61a3-48c8-be9c-2b2b3b6cab45/volumes"
Sep 19 04:30:55.364094 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:55.364076    2579 scope.go:117] "RemoveContainer" containerID="3c3cf199b97e161875a5a10f7928b300f91c58f115358478539e46a89e5fcc05"
Sep 19 04:30:55.364412 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:30:55.364385    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c3cf199b97e161875a5a10f7928b300f91c58f115358478539e46a89e5fcc05\": container with ID starting with 3c3cf199b97e161875a5a10f7928b300f91c58f115358478539e46a89e5fcc05 not found: ID does not exist" containerID="3c3cf199b97e161875a5a10f7928b300f91c58f115358478539e46a89e5fcc05"
Sep 19 04:30:55.364462 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:55.364430    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c3cf199b97e161875a5a10f7928b300f91c58f115358478539e46a89e5fcc05"} err="failed to get container status \"3c3cf199b97e161875a5a10f7928b300f91c58f115358478539e46a89e5fcc05\": rpc error: code = NotFound desc = could not find container \"3c3cf199b97e161875a5a10f7928b300f91c58f115358478539e46a89e5fcc05\": container with ID starting with 3c3cf199b97e161875a5a10f7928b300f91c58f115358478539e46a89e5fcc05 not found: ID does not exist"
Sep 19 04:30:55.375019 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:55.374990    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-rz9bs"]
Sep 19 04:30:55.377252 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:55.377233    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-rz9bs"]
Sep 19 04:30:55.417908 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:55.417879    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-pmvf4"]
Sep 19 04:30:55.424880 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:55.424855    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-pmvf4"]
Sep 19 04:30:55.647155 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:55.647066    2579 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-jkrrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.18:15021/healthz/ready\": dial tcp 10.134.0.18:15021: connect: connection refused" start-of-body=
Sep 19 04:30:55.647155 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:55.647144    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jkrrv" podUID="2130d608-c9dd-4f97-8348-bb1a10b7fec5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.18:15021/healthz/ready\": dial tcp 10.134.0.18:15021: connect: connection refused"
Sep 19 04:30:56.647195 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:56.647163    2579 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-jkrrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.18:15021/healthz/ready\": dial tcp 10.134.0.18:15021: connect: connection refused" start-of-body=
Sep 19 04:30:56.647605 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:56.647225    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jkrrv" podUID="2130d608-c9dd-4f97-8348-bb1a10b7fec5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.18:15021/healthz/ready\": dial tcp 10.134.0.18:15021: connect: connection refused"
Sep 19 04:30:57.279211 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:57.279182    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-sjhch"
Sep 19 04:30:57.279382 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:57.279252    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-8wv77"
Sep 19 04:30:57.353448 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:57.353416    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7160552c-acfc-41be-9313-fe519bf13786" path="/var/lib/kubelet/pods/7160552c-acfc-41be-9313-fe519bf13786/volumes"
Sep 19 04:30:57.353827 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:57.353808    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="925187fb-6294-4733-afb7-72288f63e4d8" path="/var/lib/kubelet/pods/925187fb-6294-4733-afb7-72288f63e4d8/volumes"
Sep 19 04:30:57.647182 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:57.647080    2579 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-jkrrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.18:15021/healthz/ready\": dial tcp 10.134.0.18:15021: connect: connection refused" start-of-body=
Sep 19 04:30:57.647182 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:57.647160    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jkrrv" podUID="2130d608-c9dd-4f97-8348-bb1a10b7fec5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.18:15021/healthz/ready\": dial tcp 10.134.0.18:15021: connect: connection refused"
Sep 19 04:30:58.650614 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:58.650585    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jkrrv"
Sep 19 04:30:58.651719 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:30:58.651675    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jkrrv"
Sep 19 04:31:19.340531 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:31:19.340490    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-5d57fc946d-9dls8"]
Sep 19 04:31:19.341048 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:31:19.341032    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7160552c-acfc-41be-9313-fe519bf13786" containerName="authorino"
Sep 19 04:31:19.341108 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:31:19.341051    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7160552c-acfc-41be-9313-fe519bf13786" containerName="authorino"
Sep 19 04:31:19.341108 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:31:19.341076    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="925187fb-6294-4733-afb7-72288f63e4d8" containerName="authorino"
Sep 19 04:31:19.341108 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:31:19.341084    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="925187fb-6294-4733-afb7-72288f63e4d8" containerName="authorino"
Sep 19 04:31:19.341203 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:31:19.341173    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7160552c-acfc-41be-9313-fe519bf13786" containerName="authorino"
Sep 19 04:31:19.341203 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:31:19.341185    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="925187fb-6294-4733-afb7-72288f63e4d8" containerName="authorino"
Sep 19 04:31:19.346037 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:31:19.346013    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-5d57fc946d-9dls8"
Sep 19 04:31:19.350368 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:31:19.350225    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-lbmbg\""
Sep 19 04:31:19.350368 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:31:19.350243    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 19 04:31:19.350368 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:31:19.350243    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 19 04:31:19.355223 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:31:19.351870    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Sep 19 04:31:19.364756 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:31:19.364731    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-5d57fc946d-9dls8"]
Sep 19 04:31:19.475467 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:31:19.475431    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd418b3a-d757-43bc-a02e-7d82c7356079-tmp-dir\") pod \"e2e-distinct-simulated-kserve-5d57fc946d-9dls8\" (UID: \"fd418b3a-d757-43bc-a02e-7d82c7356079\") " pod="llm/e2e-distinct-simulated-kserve-5d57fc946d-9dls8"
Sep 19 04:31:19.475667 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:31:19.475481    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd418b3a-d757-43bc-a02e-7d82c7356079-home\") pod \"e2e-distinct-simulated-kserve-5d57fc946d-9dls8\" (UID: \"fd418b3a-d757-43bc-a02e-7d82c7356079\") " pod="llm/e2e-distinct-simulated-kserve-5d57fc946d-9dls8"
Sep 19 04:31:19.475667 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:31:19.475532    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgssx\" (UniqueName: \"kubernetes.io/projected/fd418b3a-d757-43bc-a02e-7d82c7356079-kube-api-access-pgssx\") pod \"e2e-distinct-simulated-kserve-5d57fc946d-9dls8\" (UID: \"fd418b3a-d757-43bc-a02e-7d82c7356079\") " pod="llm/e2e-distinct-simulated-kserve-5d57fc946d-9dls8"
Sep 19 04:31:19.475667 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:31:19.475575    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd418b3a-d757-43bc-a02e-7d82c7356079-dshm\") pod \"e2e-distinct-simulated-kserve-5d57fc946d-9dls8\" (UID: \"fd418b3a-d757-43bc-a02e-7d82c7356079\") " pod="llm/e2e-distinct-simulated-kserve-5d57fc946d-9dls8"
Sep 19 04:31:19.475667 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:31:19.475604    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd418b3a-d757-43bc-a02e-7d82c7356079-model-cache\") pod \"e2e-distinct-simulated-kserve-5d57fc946d-9dls8\" (UID: \"fd418b3a-d757-43bc-a02e-7d82c7356079\") " pod="llm/e2e-distinct-simulated-kserve-5d57fc946d-9dls8"
Sep 19 04:31:19.475667 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:31:19.475625    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd418b3a-d757-43bc-a02e-7d82c7356079-tls-certs\") pod \"e2e-distinct-simulated-kserve-5d57fc946d-9dls8\" (UID: \"fd418b3a-d757-43bc-a02e-7d82c7356079\") " pod="llm/e2e-distinct-simulated-kserve-5d57fc946d-9dls8"
Sep 19 04:31:19.577145 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:31:19.577107    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd418b3a-d757-43bc-a02e-7d82c7356079-tmp-dir\") pod \"e2e-distinct-simulated-kserve-5d57fc946d-9dls8\" (UID: \"fd418b3a-d757-43bc-a02e-7d82c7356079\") " pod="llm/e2e-distinct-simulated-kserve-5d57fc946d-9dls8"
Sep 19 04:31:19.577318 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:31:19.577168    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd418b3a-d757-43bc-a02e-7d82c7356079-home\") pod \"e2e-distinct-simulated-kserve-5d57fc946d-9dls8\" (UID: \"fd418b3a-d757-43bc-a02e-7d82c7356079\") " pod="llm/e2e-distinct-simulated-kserve-5d57fc946d-9dls8"
Sep 19 04:31:19.577318 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:31:19.577203    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgssx\" (UniqueName: \"kubernetes.io/projected/fd418b3a-d757-43bc-a02e-7d82c7356079-kube-api-access-pgssx\") pod \"e2e-distinct-simulated-kserve-5d57fc946d-9dls8\" (UID: \"fd418b3a-d757-43bc-a02e-7d82c7356079\") " pod="llm/e2e-distinct-simulated-kserve-5d57fc946d-9dls8"
Sep 19 04:31:19.577318 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:31:19.577237    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd418b3a-d757-43bc-a02e-7d82c7356079-dshm\") pod \"e2e-distinct-simulated-kserve-5d57fc946d-9dls8\" (UID: \"fd418b3a-d757-43bc-a02e-7d82c7356079\") " pod="llm/e2e-distinct-simulated-kserve-5d57fc946d-9dls8"
Sep 19 04:31:19.577318 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:31:19.577259    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd418b3a-d757-43bc-a02e-7d82c7356079-model-cache\") pod \"e2e-distinct-simulated-kserve-5d57fc946d-9dls8\" (UID: \"fd418b3a-d757-43bc-a02e-7d82c7356079\") " pod="llm/e2e-distinct-simulated-kserve-5d57fc946d-9dls8"
Sep 19 04:31:19.577318 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:31:19.577288    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd418b3a-d757-43bc-a02e-7d82c7356079-tls-certs\") pod \"e2e-distinct-simulated-kserve-5d57fc946d-9dls8\" (UID: \"fd418b3a-d757-43bc-a02e-7d82c7356079\") " pod="llm/e2e-distinct-simulated-kserve-5d57fc946d-9dls8"
Sep 19 04:31:19.577606 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:31:19.577556    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd418b3a-d757-43bc-a02e-7d82c7356079-tmp-dir\") pod \"e2e-distinct-simulated-kserve-5d57fc946d-9dls8\" (UID: \"fd418b3a-d757-43bc-a02e-7d82c7356079\") " pod="llm/e2e-distinct-simulated-kserve-5d57fc946d-9dls8"
Sep 19 04:31:19.577660 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:31:19.577632    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd418b3a-d757-43bc-a02e-7d82c7356079-home\") pod \"e2e-distinct-simulated-kserve-5d57fc946d-9dls8\" (UID: \"fd418b3a-d757-43bc-a02e-7d82c7356079\") " pod="llm/e2e-distinct-simulated-kserve-5d57fc946d-9dls8"
Sep 19 04:31:19.577744 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:31:19.577651    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd418b3a-d757-43bc-a02e-7d82c7356079-model-cache\") pod \"e2e-distinct-simulated-kserve-5d57fc946d-9dls8\" (UID: \"fd418b3a-d757-43bc-a02e-7d82c7356079\") " pod="llm/e2e-distinct-simulated-kserve-5d57fc946d-9dls8"
Sep 19 04:31:19.579796 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:31:19.579772    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd418b3a-d757-43bc-a02e-7d82c7356079-dshm\") pod \"e2e-distinct-simulated-kserve-5d57fc946d-9dls8\" (UID: \"fd418b3a-d757-43bc-a02e-7d82c7356079\") " pod="llm/e2e-distinct-simulated-kserve-5d57fc946d-9dls8"
Sep 19 04:31:19.580036 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:31:19.580017    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd418b3a-d757-43bc-a02e-7d82c7356079-tls-certs\") pod \"e2e-distinct-simulated-kserve-5d57fc946d-9dls8\" (UID: \"fd418b3a-d757-43bc-a02e-7d82c7356079\") " pod="llm/e2e-distinct-simulated-kserve-5d57fc946d-9dls8"
Sep 19 04:31:19.586486 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:31:19.586456    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgssx\" (UniqueName: \"kubernetes.io/projected/fd418b3a-d757-43bc-a02e-7d82c7356079-kube-api-access-pgssx\") pod \"e2e-distinct-simulated-kserve-5d57fc946d-9dls8\" (UID: \"fd418b3a-d757-43bc-a02e-7d82c7356079\") " pod="llm/e2e-distinct-simulated-kserve-5d57fc946d-9dls8"
Sep 19 04:31:19.665263 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:31:19.665224    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-5d57fc946d-9dls8"
Sep 19 04:31:19.949443 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:31:19.949413    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-5d57fc946d-9dls8"]
Sep 19 04:31:19.951521 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:31:19.951496    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd418b3a_d757_43bc_a02e_7d82c7356079.slice/crio-f45b5aca1cbe1b5b071eb8552a7e070665ea6a963e12a2310f4ee47a6d63f583 WatchSource:0}: Error finding container f45b5aca1cbe1b5b071eb8552a7e070665ea6a963e12a2310f4ee47a6d63f583: Status 404 returned error can't find the container with id f45b5aca1cbe1b5b071eb8552a7e070665ea6a963e12a2310f4ee47a6d63f583
Sep 19 04:31:20.476233 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:31:20.476193    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-5d57fc946d-9dls8" event={"ID":"fd418b3a-d757-43bc-a02e-7d82c7356079","Type":"ContainerStarted","Data":"f45b5aca1cbe1b5b071eb8552a7e070665ea6a963e12a2310f4ee47a6d63f583"}
Sep 19 04:31:21.487455 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:31:21.486617    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-5d57fc946d-9dls8" event={"ID":"fd418b3a-d757-43bc-a02e-7d82c7356079","Type":"ContainerStarted","Data":"15bcdd17d0c5fcc36d53934864473819a701189ad50427bd122eb353dd00b3c0"}
Sep 19 04:31:21.511802 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:31:21.511727    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-5d57fc946d-9dls8" podStartSLOduration=1.482315323 podStartE2EDuration="2.511666966s" totalImagesPullingTime="1.029351643s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 04:31:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:31:21.509983877 +0000 UTC m=+1060.692945048" watchObservedRunningTime="2026-09-19 04:31:21.511666966 +0000 UTC m=+1060.694628115"
Sep 19 04:31:21.666309 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:31:21.666269    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-5d57fc946d-9dls8"
Sep 19 04:31:29.665624 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:31:29.665593    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-5d57fc946d-9dls8"
Sep 19 04:31:29.678724 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:31:29.678670    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-5d57fc946d-9dls8"
Sep 19 04:31:30.547829 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:31:30.547799    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-5d57fc946d-9dls8"
Sep 19 04:32:05.444490 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:32:05.444459    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-89945f574-l5xsz"]
Sep 19 04:32:05.449575 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:32:05.449553    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-89945f574-l5xsz"
Sep 19 04:32:05.454817 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:32:05.454794    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 19 04:32:05.463088 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:32:05.463063    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-89945f574-l5xsz"]
Sep 19 04:32:05.604027 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:32:05.603988    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd909eb5-27da-46c3-a4b8-1ed1265a2a0b-dshm\") pod \"facebook-opt-125m-simulated-kserve-89945f574-l5xsz\" (UID: \"dd909eb5-27da-46c3-a4b8-1ed1265a2a0b\") " pod="llm/facebook-opt-125m-simulated-kserve-89945f574-l5xsz"
Sep 19 04:32:05.604205 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:32:05.604056    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7vwv\" (UniqueName: \"kubernetes.io/projected/dd909eb5-27da-46c3-a4b8-1ed1265a2a0b-kube-api-access-f7vwv\") pod \"facebook-opt-125m-simulated-kserve-89945f574-l5xsz\" (UID: \"dd909eb5-27da-46c3-a4b8-1ed1265a2a0b\") " pod="llm/facebook-opt-125m-simulated-kserve-89945f574-l5xsz"
Sep 19 04:32:05.604205 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:32:05.604093    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd909eb5-27da-46c3-a4b8-1ed1265a2a0b-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-89945f574-l5xsz\" (UID: \"dd909eb5-27da-46c3-a4b8-1ed1265a2a0b\") " pod="llm/facebook-opt-125m-simulated-kserve-89945f574-l5xsz"
Sep 19 04:32:05.604205 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:32:05.604119    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd909eb5-27da-46c3-a4b8-1ed1265a2a0b-model-cache\") pod \"facebook-opt-125m-simulated-kserve-89945f574-l5xsz\" (UID: \"dd909eb5-27da-46c3-a4b8-1ed1265a2a0b\") " pod="llm/facebook-opt-125m-simulated-kserve-89945f574-l5xsz"
Sep 19 04:32:05.604317 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:32:05.604210    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd909eb5-27da-46c3-a4b8-1ed1265a2a0b-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-89945f574-l5xsz\" (UID: \"dd909eb5-27da-46c3-a4b8-1ed1265a2a0b\") " pod="llm/facebook-opt-125m-simulated-kserve-89945f574-l5xsz"
Sep 19 04:32:05.604317 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:32:05.604255    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd909eb5-27da-46c3-a4b8-1ed1265a2a0b-home\") pod \"facebook-opt-125m-simulated-kserve-89945f574-l5xsz\" (UID: \"dd909eb5-27da-46c3-a4b8-1ed1265a2a0b\") " pod="llm/facebook-opt-125m-simulated-kserve-89945f574-l5xsz"
Sep 19 04:32:05.705421 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:32:05.705334    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd909eb5-27da-46c3-a4b8-1ed1265a2a0b-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-89945f574-l5xsz\" (UID: \"dd909eb5-27da-46c3-a4b8-1ed1265a2a0b\") " pod="llm/facebook-opt-125m-simulated-kserve-89945f574-l5xsz"
Sep 19 04:32:05.705421 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:32:05.705375    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd909eb5-27da-46c3-a4b8-1ed1265a2a0b-home\") pod \"facebook-opt-125m-simulated-kserve-89945f574-l5xsz\" (UID: \"dd909eb5-27da-46c3-a4b8-1ed1265a2a0b\") " pod="llm/facebook-opt-125m-simulated-kserve-89945f574-l5xsz"
Sep 19 04:32:05.705611 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:32:05.705504    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd909eb5-27da-46c3-a4b8-1ed1265a2a0b-dshm\") pod \"facebook-opt-125m-simulated-kserve-89945f574-l5xsz\" (UID: \"dd909eb5-27da-46c3-a4b8-1ed1265a2a0b\") " pod="llm/facebook-opt-125m-simulated-kserve-89945f574-l5xsz"
Sep 19 04:32:05.705611 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:32:05.705588    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7vwv\" (UniqueName: \"kubernetes.io/projected/dd909eb5-27da-46c3-a4b8-1ed1265a2a0b-kube-api-access-f7vwv\") pod \"facebook-opt-125m-simulated-kserve-89945f574-l5xsz\" (UID: \"dd909eb5-27da-46c3-a4b8-1ed1265a2a0b\") " pod="llm/facebook-opt-125m-simulated-kserve-89945f574-l5xsz"
Sep 19 04:32:05.705676 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:32:05.705618    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd909eb5-27da-46c3-a4b8-1ed1265a2a0b-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-89945f574-l5xsz\" (UID: \"dd909eb5-27da-46c3-a4b8-1ed1265a2a0b\") " pod="llm/facebook-opt-125m-simulated-kserve-89945f574-l5xsz"
Sep 19 04:32:05.705676 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:32:05.705658    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd909eb5-27da-46c3-a4b8-1ed1265a2a0b-model-cache\") pod \"facebook-opt-125m-simulated-kserve-89945f574-l5xsz\" (UID: \"dd909eb5-27da-46c3-a4b8-1ed1265a2a0b\") " pod="llm/facebook-opt-125m-simulated-kserve-89945f574-l5xsz"
Sep 19 04:32:05.705809 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:32:05.705747    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd909eb5-27da-46c3-a4b8-1ed1265a2a0b-home\") pod \"facebook-opt-125m-simulated-kserve-89945f574-l5xsz\" (UID: \"dd909eb5-27da-46c3-a4b8-1ed1265a2a0b\") " pod="llm/facebook-opt-125m-simulated-kserve-89945f574-l5xsz"
Sep 19 04:32:05.705860 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:32:05.705820    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd909eb5-27da-46c3-a4b8-1ed1265a2a0b-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-89945f574-l5xsz\" (UID: \"dd909eb5-27da-46c3-a4b8-1ed1265a2a0b\") " pod="llm/facebook-opt-125m-simulated-kserve-89945f574-l5xsz"
Sep 19 04:32:05.705977 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:32:05.705961    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd909eb5-27da-46c3-a4b8-1ed1265a2a0b-model-cache\") pod \"facebook-opt-125m-simulated-kserve-89945f574-l5xsz\" (UID: \"dd909eb5-27da-46c3-a4b8-1ed1265a2a0b\") " pod="llm/facebook-opt-125m-simulated-kserve-89945f574-l5xsz"
Sep 19 04:32:05.707899 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:32:05.707874    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd909eb5-27da-46c3-a4b8-1ed1265a2a0b-dshm\") pod \"facebook-opt-125m-simulated-kserve-89945f574-l5xsz\" (UID: \"dd909eb5-27da-46c3-a4b8-1ed1265a2a0b\") " pod="llm/facebook-opt-125m-simulated-kserve-89945f574-l5xsz"
Sep 19 04:32:05.708273 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:32:05.708253    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd909eb5-27da-46c3-a4b8-1ed1265a2a0b-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-89945f574-l5xsz\" (UID: \"dd909eb5-27da-46c3-a4b8-1ed1265a2a0b\") " pod="llm/facebook-opt-125m-simulated-kserve-89945f574-l5xsz"
Sep 19 04:32:05.715237 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:32:05.715215    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7vwv\" (UniqueName: \"kubernetes.io/projected/dd909eb5-27da-46c3-a4b8-1ed1265a2a0b-kube-api-access-f7vwv\") pod \"facebook-opt-125m-simulated-kserve-89945f574-l5xsz\" (UID: \"dd909eb5-27da-46c3-a4b8-1ed1265a2a0b\") " pod="llm/facebook-opt-125m-simulated-kserve-89945f574-l5xsz"
Sep 19 04:32:05.759215 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:32:05.759181    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-89945f574-l5xsz"
Sep 19 04:32:06.048272 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:32:06.048194    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-89945f574-l5xsz"]
Sep 19 04:32:06.051879 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:32:06.051842    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd909eb5_27da_46c3_a4b8_1ed1265a2a0b.slice/crio-0f0a5242f68ef077187b84780bd0de4e19d168729ca29edc05c147bcfd21ee0b WatchSource:0}: Error finding container 0f0a5242f68ef077187b84780bd0de4e19d168729ca29edc05c147bcfd21ee0b: Status 404 returned error can't find the container with id 0f0a5242f68ef077187b84780bd0de4e19d168729ca29edc05c147bcfd21ee0b
Sep 19 04:32:06.731901 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:32:06.731865    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-89945f574-l5xsz" event={"ID":"dd909eb5-27da-46c3-a4b8-1ed1265a2a0b","Type":"ContainerStarted","Data":"a8894ca5286889336028bc1881d79db313256e78cb30edd0829097e759c0f6aa"}
Sep 19 04:32:06.732277 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:32:06.731921    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-89945f574-l5xsz" event={"ID":"dd909eb5-27da-46c3-a4b8-1ed1265a2a0b","Type":"ContainerStarted","Data":"0f0a5242f68ef077187b84780bd0de4e19d168729ca29edc05c147bcfd21ee0b"}
Sep 19 04:32:06.753536 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:32:06.753485    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-89945f574-l5xsz" podStartSLOduration=1.57263901 podStartE2EDuration="1.753468075s" totalImagesPullingTime="180.829065ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 04:32:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:32:06.75235379 +0000 UTC m=+1105.935314930" watchObservedRunningTime="2026-09-19 04:32:06.753468075 +0000 UTC m=+1105.936429223"
Sep 19 04:32:06.759992 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:32:06.759967    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-89945f574-l5xsz"
Sep 19 04:32:15.759752 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:32:15.759686    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-89945f574-l5xsz"
Sep 19 04:32:15.772173 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:32:15.772145    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-89945f574-l5xsz"
Sep 19 04:32:15.790975 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:32:15.790944    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-89945f574-l5xsz"
Sep 19 04:33:22.165195 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:22.165156    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-bd667f646-lx5br"]
Sep 19 04:33:22.168521 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:22.168503    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-bd667f646-lx5br"
Sep 19 04:33:22.179599 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:22.179569    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-bd667f646-lx5br"]
Sep 19 04:33:22.187642 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:22.187607    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0b3506fd-4ba6-4525-a63f-c0aff0f06b90-tls-cert\") pod \"authorino-bd667f646-lx5br\" (UID: \"0b3506fd-4ba6-4525-a63f-c0aff0f06b90\") " pod="kuadrant-system/authorino-bd667f646-lx5br"
Sep 19 04:33:22.187824 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:22.187661    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29rqg\" (UniqueName: \"kubernetes.io/projected/0b3506fd-4ba6-4525-a63f-c0aff0f06b90-kube-api-access-29rqg\") pod \"authorino-bd667f646-lx5br\" (UID: \"0b3506fd-4ba6-4525-a63f-c0aff0f06b90\") " pod="kuadrant-system/authorino-bd667f646-lx5br"
Sep 19 04:33:22.289221 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:22.289187    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0b3506fd-4ba6-4525-a63f-c0aff0f06b90-tls-cert\") pod \"authorino-bd667f646-lx5br\" (UID: \"0b3506fd-4ba6-4525-a63f-c0aff0f06b90\") " pod="kuadrant-system/authorino-bd667f646-lx5br"
Sep 19 04:33:22.289221 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:22.289226    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29rqg\" (UniqueName: \"kubernetes.io/projected/0b3506fd-4ba6-4525-a63f-c0aff0f06b90-kube-api-access-29rqg\") pod \"authorino-bd667f646-lx5br\" (UID: \"0b3506fd-4ba6-4525-a63f-c0aff0f06b90\") " pod="kuadrant-system/authorino-bd667f646-lx5br"
Sep 19 04:33:22.292238 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:22.292207    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0b3506fd-4ba6-4525-a63f-c0aff0f06b90-tls-cert\") pod \"authorino-bd667f646-lx5br\" (UID: \"0b3506fd-4ba6-4525-a63f-c0aff0f06b90\") " pod="kuadrant-system/authorino-bd667f646-lx5br"
Sep 19 04:33:22.298044 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:22.298004    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29rqg\" (UniqueName: \"kubernetes.io/projected/0b3506fd-4ba6-4525-a63f-c0aff0f06b90-kube-api-access-29rqg\") pod \"authorino-bd667f646-lx5br\" (UID: \"0b3506fd-4ba6-4525-a63f-c0aff0f06b90\") " pod="kuadrant-system/authorino-bd667f646-lx5br"
Sep 19 04:33:22.477320 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:22.477235    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-bd667f646-lx5br"
Sep 19 04:33:22.756259 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:22.756231    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-bd667f646-lx5br"]
Sep 19 04:33:22.758142 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:33:22.758116    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0b3506fd_4ba6_4525_a63f_c0aff0f06b90.slice/crio-a9fffb8a2c17edb0a9dc40e22b4f9ffef258db313bb5bae9bb3fa5f2ae6b8d0f WatchSource:0}: Error finding container a9fffb8a2c17edb0a9dc40e22b4f9ffef258db313bb5bae9bb3fa5f2ae6b8d0f: Status 404 returned error can't find the container with id a9fffb8a2c17edb0a9dc40e22b4f9ffef258db313bb5bae9bb3fa5f2ae6b8d0f
Sep 19 04:33:23.152419 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:23.152386    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-bd667f646-lx5br" event={"ID":"0b3506fd-4ba6-4525-a63f-c0aff0f06b90","Type":"ContainerStarted","Data":"a9fffb8a2c17edb0a9dc40e22b4f9ffef258db313bb5bae9bb3fa5f2ae6b8d0f"}
Sep 19 04:33:24.162570 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:24.162535    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-bd667f646-lx5br" event={"ID":"0b3506fd-4ba6-4525-a63f-c0aff0f06b90","Type":"ContainerStarted","Data":"47f1b3275bba81f59dbff43b84826ca35c22197b3c2833e52edf3f97462bdf54"}
Sep 19 04:33:24.224264 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:24.224207    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-bd667f646-lx5br" podStartSLOduration=1.477426214 podStartE2EDuration="2.224188045s" totalImagesPullingTime="746.761831ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 04:33:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:33:24.222092129 +0000 UTC m=+1183.405053277" watchObservedRunningTime="2026-09-19 04:33:24.224188045 +0000 UTC m=+1183.407149204"
Sep 19 04:33:24.289209 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:24.289165    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6b68f67659-vrtkf"]
Sep 19 04:33:24.289430 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:24.289400    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-6b68f67659-vrtkf" podUID="6430dec9-9e3b-4244-9280-6b000beea86c" containerName="authorino" containerID="cri-o://3a1007eec828dbf95d1fa698731e81b4510087daf9507ff5f702296d04a71c49" gracePeriod=30
Sep 19 04:33:24.568291 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:24.568263    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6b68f67659-vrtkf"
Sep 19 04:33:24.611464 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:24.611431    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6430dec9-9e3b-4244-9280-6b000beea86c-tls-cert\") pod \"6430dec9-9e3b-4244-9280-6b000beea86c\" (UID: \"6430dec9-9e3b-4244-9280-6b000beea86c\") "
Sep 19 04:33:24.611667 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:24.611485    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d8v98\" (UniqueName: \"kubernetes.io/projected/6430dec9-9e3b-4244-9280-6b000beea86c-kube-api-access-d8v98\") pod \"6430dec9-9e3b-4244-9280-6b000beea86c\" (UID: \"6430dec9-9e3b-4244-9280-6b000beea86c\") "
Sep 19 04:33:24.614445 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:24.614407    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6430dec9-9e3b-4244-9280-6b000beea86c-kube-api-access-d8v98" (OuterVolumeSpecName: "kube-api-access-d8v98") pod "6430dec9-9e3b-4244-9280-6b000beea86c" (UID: "6430dec9-9e3b-4244-9280-6b000beea86c"). InnerVolumeSpecName "kube-api-access-d8v98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 04:33:24.623024 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:24.622990    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6430dec9-9e3b-4244-9280-6b000beea86c-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "6430dec9-9e3b-4244-9280-6b000beea86c" (UID: "6430dec9-9e3b-4244-9280-6b000beea86c"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 04:33:24.713214 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:24.713175    2579 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6430dec9-9e3b-4244-9280-6b000beea86c-tls-cert\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:33:24.713214 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:24.713209    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8v98\" (UniqueName: \"kubernetes.io/projected/6430dec9-9e3b-4244-9280-6b000beea86c-kube-api-access-d8v98\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:33:25.168981 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:25.168937    2579 generic.go:356] "Generic (PLEG): container finished" podID="6430dec9-9e3b-4244-9280-6b000beea86c" containerID="3a1007eec828dbf95d1fa698731e81b4510087daf9507ff5f702296d04a71c49" exitCode=0
Sep 19 04:33:25.169474 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:25.169002    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6b68f67659-vrtkf"
Sep 19 04:33:25.169474 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:25.169026    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6b68f67659-vrtkf" event={"ID":"6430dec9-9e3b-4244-9280-6b000beea86c","Type":"ContainerDied","Data":"3a1007eec828dbf95d1fa698731e81b4510087daf9507ff5f702296d04a71c49"}
Sep 19 04:33:25.169474 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:25.169067    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6b68f67659-vrtkf" event={"ID":"6430dec9-9e3b-4244-9280-6b000beea86c","Type":"ContainerDied","Data":"4aeded8d3586095b30f774c5f37afaad0b0fde2e3d96b194f3e9f1898812c8b1"}
Sep 19 04:33:25.169474 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:25.169086    2579 scope.go:117] "RemoveContainer" containerID="3a1007eec828dbf95d1fa698731e81b4510087daf9507ff5f702296d04a71c49"
Sep 19 04:33:25.178895 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:25.178875    2579 scope.go:117] "RemoveContainer" containerID="3a1007eec828dbf95d1fa698731e81b4510087daf9507ff5f702296d04a71c49"
Sep 19 04:33:25.179222 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:33:25.179205    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a1007eec828dbf95d1fa698731e81b4510087daf9507ff5f702296d04a71c49\": container with ID starting with 3a1007eec828dbf95d1fa698731e81b4510087daf9507ff5f702296d04a71c49 not found: ID does not exist" containerID="3a1007eec828dbf95d1fa698731e81b4510087daf9507ff5f702296d04a71c49"
Sep 19 04:33:25.179280 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:25.179231    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a1007eec828dbf95d1fa698731e81b4510087daf9507ff5f702296d04a71c49"} err="failed to get container status \"3a1007eec828dbf95d1fa698731e81b4510087daf9507ff5f702296d04a71c49\": rpc error: code = NotFound desc = could not find container \"3a1007eec828dbf95d1fa698731e81b4510087daf9507ff5f702296d04a71c49\": container with ID starting with 3a1007eec828dbf95d1fa698731e81b4510087daf9507ff5f702296d04a71c49 not found: ID does not exist"
Sep 19 04:33:25.342057 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:25.342019    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6b68f67659-vrtkf"]
Sep 19 04:33:25.373333 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:25.373301    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6b68f67659-vrtkf"]
Sep 19 04:33:27.351101 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:27.351064    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6430dec9-9e3b-4244-9280-6b000beea86c" path="/var/lib/kubelet/pods/6430dec9-9e3b-4244-9280-6b000beea86c/volumes"
Sep 19 04:33:46.824748 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:46.824713    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-87e3e6-openshift-default-8989bfbdd-fwgnd"]
Sep 19 04:33:46.825141 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:46.825105    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6430dec9-9e3b-4244-9280-6b000beea86c" containerName="authorino"
Sep 19 04:33:46.825141 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:46.825117    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6430dec9-9e3b-4244-9280-6b000beea86c" containerName="authorino"
Sep 19 04:33:46.825220 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:46.825188    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6430dec9-9e3b-4244-9280-6b000beea86c" containerName="authorino"
Sep 19 04:33:46.829859 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:46.829837    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-87e3e6-openshift-default-8989bfbdd-fwgnd"
Sep 19 04:33:46.834998 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:46.834973    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-87e3e6-openshift-default-dockercfg-xbcpl\""
Sep 19 04:33:46.869327 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:46.869297    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-87e3e6-openshift-default-8989bfbdd-fwgnd"]
Sep 19 04:33:46.922073 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:46.922034    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39081cb6-0611-41e5-b50e-bfe7759ef529-istiod-ca-cert\") pod \"e2e-worker-w0-87e3e6-openshift-default-8989bfbdd-fwgnd\" (UID: \"39081cb6-0611-41e5-b50e-bfe7759ef529\") " pod="openshift-ingress/e2e-worker-w0-87e3e6-openshift-default-8989bfbdd-fwgnd"
Sep 19 04:33:46.922242 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:46.922083    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39081cb6-0611-41e5-b50e-bfe7759ef529-credential-socket\") pod \"e2e-worker-w0-87e3e6-openshift-default-8989bfbdd-fwgnd\" (UID: \"39081cb6-0611-41e5-b50e-bfe7759ef529\") " pod="openshift-ingress/e2e-worker-w0-87e3e6-openshift-default-8989bfbdd-fwgnd"
Sep 19 04:33:46.922242 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:46.922157    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39081cb6-0611-41e5-b50e-bfe7759ef529-istio-envoy\") pod \"e2e-worker-w0-87e3e6-openshift-default-8989bfbdd-fwgnd\" (UID: \"39081cb6-0611-41e5-b50e-bfe7759ef529\") " pod="openshift-ingress/e2e-worker-w0-87e3e6-openshift-default-8989bfbdd-fwgnd"
Sep 19 04:33:46.922242 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:46.922206    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39081cb6-0611-41e5-b50e-bfe7759ef529-workload-certs\") pod \"e2e-worker-w0-87e3e6-openshift-default-8989bfbdd-fwgnd\" (UID: \"39081cb6-0611-41e5-b50e-bfe7759ef529\") " pod="openshift-ingress/e2e-worker-w0-87e3e6-openshift-default-8989bfbdd-fwgnd"
Sep 19 04:33:46.922242 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:46.922236    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39081cb6-0611-41e5-b50e-bfe7759ef529-workload-socket\") pod \"e2e-worker-w0-87e3e6-openshift-default-8989bfbdd-fwgnd\" (UID: \"39081cb6-0611-41e5-b50e-bfe7759ef529\") " pod="openshift-ingress/e2e-worker-w0-87e3e6-openshift-default-8989bfbdd-fwgnd"
Sep 19 04:33:46.922380 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:46.922287    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39081cb6-0611-41e5-b50e-bfe7759ef529-istio-token\") pod \"e2e-worker-w0-87e3e6-openshift-default-8989bfbdd-fwgnd\" (UID: \"39081cb6-0611-41e5-b50e-bfe7759ef529\") " pod="openshift-ingress/e2e-worker-w0-87e3e6-openshift-default-8989bfbdd-fwgnd"
Sep 19 04:33:46.922380 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:46.922304    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39081cb6-0611-41e5-b50e-bfe7759ef529-istio-data\") pod \"e2e-worker-w0-87e3e6-openshift-default-8989bfbdd-fwgnd\" (UID: \"39081cb6-0611-41e5-b50e-bfe7759ef529\") " pod="openshift-ingress/e2e-worker-w0-87e3e6-openshift-default-8989bfbdd-fwgnd"
Sep 19 04:33:46.922380 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:46.922319    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39081cb6-0611-41e5-b50e-bfe7759ef529-istio-podinfo\") pod \"e2e-worker-w0-87e3e6-openshift-default-8989bfbdd-fwgnd\" (UID: \"39081cb6-0611-41e5-b50e-bfe7759ef529\") " pod="openshift-ingress/e2e-worker-w0-87e3e6-openshift-default-8989bfbdd-fwgnd"
Sep 19 04:33:46.922380 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:46.922333    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44rjb\" (UniqueName: \"kubernetes.io/projected/39081cb6-0611-41e5-b50e-bfe7759ef529-kube-api-access-44rjb\") pod \"e2e-worker-w0-87e3e6-openshift-default-8989bfbdd-fwgnd\" (UID: \"39081cb6-0611-41e5-b50e-bfe7759ef529\") " pod="openshift-ingress/e2e-worker-w0-87e3e6-openshift-default-8989bfbdd-fwgnd"
Sep 19 04:33:46.968823 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:46.968788    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk"]
Sep 19 04:33:46.972271 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:46.972246    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk"
Sep 19 04:33:46.976434 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:46.976410    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-d0f416-openshift-default-dockercfg-6mpgc\""
Sep 19 04:33:46.994109 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:46.994078    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk"]
Sep 19 04:33:47.023447 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:47.023401    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39081cb6-0611-41e5-b50e-bfe7759ef529-credential-socket\") pod \"e2e-worker-w0-87e3e6-openshift-default-8989bfbdd-fwgnd\" (UID: \"39081cb6-0611-41e5-b50e-bfe7759ef529\") " pod="openshift-ingress/e2e-worker-w0-87e3e6-openshift-default-8989bfbdd-fwgnd"
Sep 19 04:33:47.023647 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:47.023450    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39081cb6-0611-41e5-b50e-bfe7759ef529-istio-envoy\") pod \"e2e-worker-w0-87e3e6-openshift-default-8989bfbdd-fwgnd\" (UID: \"39081cb6-0611-41e5-b50e-bfe7759ef529\") " pod="openshift-ingress/e2e-worker-w0-87e3e6-openshift-default-8989bfbdd-fwgnd"
Sep 19 04:33:47.023647 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:47.023504    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39081cb6-0611-41e5-b50e-bfe7759ef529-workload-certs\") pod \"e2e-worker-w0-87e3e6-openshift-default-8989bfbdd-fwgnd\" (UID: \"39081cb6-0611-41e5-b50e-bfe7759ef529\") " pod="openshift-ingress/e2e-worker-w0-87e3e6-openshift-default-8989bfbdd-fwgnd"
Sep 19 04:33:47.023647 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:47.023538    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39081cb6-0611-41e5-b50e-bfe7759ef529-workload-socket\") pod \"e2e-worker-w0-87e3e6-openshift-default-8989bfbdd-fwgnd\" (UID: \"39081cb6-0611-41e5-b50e-bfe7759ef529\") " pod="openshift-ingress/e2e-worker-w0-87e3e6-openshift-default-8989bfbdd-fwgnd"
Sep 19 04:33:47.023647 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:47.023607    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39081cb6-0611-41e5-b50e-bfe7759ef529-istio-token\") pod \"e2e-worker-w0-87e3e6-openshift-default-8989bfbdd-fwgnd\" (UID: \"39081cb6-0611-41e5-b50e-bfe7759ef529\") " pod="openshift-ingress/e2e-worker-w0-87e3e6-openshift-default-8989bfbdd-fwgnd"
Sep 19 04:33:47.023647 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:47.023636    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39081cb6-0611-41e5-b50e-bfe7759ef529-istio-data\") pod \"e2e-worker-w0-87e3e6-openshift-default-8989bfbdd-fwgnd\" (UID: \"39081cb6-0611-41e5-b50e-bfe7759ef529\") " pod="openshift-ingress/e2e-worker-w0-87e3e6-openshift-default-8989bfbdd-fwgnd"
Sep 19 04:33:47.023931 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:47.023659    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39081cb6-0611-41e5-b50e-bfe7759ef529-istio-podinfo\") pod \"e2e-worker-w0-87e3e6-openshift-default-8989bfbdd-fwgnd\" (UID: \"39081cb6-0611-41e5-b50e-bfe7759ef529\") " pod="openshift-ingress/e2e-worker-w0-87e3e6-openshift-default-8989bfbdd-fwgnd"
Sep 19 04:33:47.023931 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:47.023686    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44rjb\" (UniqueName: \"kubernetes.io/projected/39081cb6-0611-41e5-b50e-bfe7759ef529-kube-api-access-44rjb\") pod \"e2e-worker-w0-87e3e6-openshift-default-8989bfbdd-fwgnd\" (UID: \"39081cb6-0611-41e5-b50e-bfe7759ef529\") " pod="openshift-ingress/e2e-worker-w0-87e3e6-openshift-default-8989bfbdd-fwgnd"
Sep 19 04:33:47.023931 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:47.023743    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39081cb6-0611-41e5-b50e-bfe7759ef529-istiod-ca-cert\") pod \"e2e-worker-w0-87e3e6-openshift-default-8989bfbdd-fwgnd\" (UID: \"39081cb6-0611-41e5-b50e-bfe7759ef529\") " pod="openshift-ingress/e2e-worker-w0-87e3e6-openshift-default-8989bfbdd-fwgnd"
Sep 19 04:33:47.023931 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:47.023894    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39081cb6-0611-41e5-b50e-bfe7759ef529-credential-socket\") pod \"e2e-worker-w0-87e3e6-openshift-default-8989bfbdd-fwgnd\" (UID: \"39081cb6-0611-41e5-b50e-bfe7759ef529\") " pod="openshift-ingress/e2e-worker-w0-87e3e6-openshift-default-8989bfbdd-fwgnd"
Sep 19 04:33:47.024144 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:47.024046    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39081cb6-0611-41e5-b50e-bfe7759ef529-istio-data\") pod \"e2e-worker-w0-87e3e6-openshift-default-8989bfbdd-fwgnd\" (UID: \"39081cb6-0611-41e5-b50e-bfe7759ef529\") " pod="openshift-ingress/e2e-worker-w0-87e3e6-openshift-default-8989bfbdd-fwgnd"
Sep 19 04:33:47.024355 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:47.024296    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39081cb6-0611-41e5-b50e-bfe7759ef529-workload-socket\") pod \"e2e-worker-w0-87e3e6-openshift-default-8989bfbdd-fwgnd\" (UID: \"39081cb6-0611-41e5-b50e-bfe7759ef529\") " pod="openshift-ingress/e2e-worker-w0-87e3e6-openshift-default-8989bfbdd-fwgnd"
Sep 19 04:33:47.024355 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:47.024332    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39081cb6-0611-41e5-b50e-bfe7759ef529-workload-certs\") pod \"e2e-worker-w0-87e3e6-openshift-default-8989bfbdd-fwgnd\" (UID: \"39081cb6-0611-41e5-b50e-bfe7759ef529\") " pod="openshift-ingress/e2e-worker-w0-87e3e6-openshift-default-8989bfbdd-fwgnd"
Sep 19 04:33:47.024508 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:47.024430    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39081cb6-0611-41e5-b50e-bfe7759ef529-istiod-ca-cert\") pod \"e2e-worker-w0-87e3e6-openshift-default-8989bfbdd-fwgnd\" (UID: \"39081cb6-0611-41e5-b50e-bfe7759ef529\") " pod="openshift-ingress/e2e-worker-w0-87e3e6-openshift-default-8989bfbdd-fwgnd"
Sep 19 04:33:47.026071 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:47.026050    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39081cb6-0611-41e5-b50e-bfe7759ef529-istio-envoy\") pod \"e2e-worker-w0-87e3e6-openshift-default-8989bfbdd-fwgnd\" (UID: \"39081cb6-0611-41e5-b50e-bfe7759ef529\") " pod="openshift-ingress/e2e-worker-w0-87e3e6-openshift-default-8989bfbdd-fwgnd"
Sep 19 04:33:47.026709 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:47.026675    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39081cb6-0611-41e5-b50e-bfe7759ef529-istio-podinfo\") pod \"e2e-worker-w0-87e3e6-openshift-default-8989bfbdd-fwgnd\" (UID: \"39081cb6-0611-41e5-b50e-bfe7759ef529\") " pod="openshift-ingress/e2e-worker-w0-87e3e6-openshift-default-8989bfbdd-fwgnd"
Sep 19 04:33:47.042336 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:47.042294    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39081cb6-0611-41e5-b50e-bfe7759ef529-istio-token\") pod \"e2e-worker-w0-87e3e6-openshift-default-8989bfbdd-fwgnd\" (UID: \"39081cb6-0611-41e5-b50e-bfe7759ef529\") " pod="openshift-ingress/e2e-worker-w0-87e3e6-openshift-default-8989bfbdd-fwgnd"
Sep 19 04:33:47.073985 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:47.073947    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44rjb\" (UniqueName: \"kubernetes.io/projected/39081cb6-0611-41e5-b50e-bfe7759ef529-kube-api-access-44rjb\") pod \"e2e-worker-w0-87e3e6-openshift-default-8989bfbdd-fwgnd\" (UID: \"39081cb6-0611-41e5-b50e-bfe7759ef529\") " pod="openshift-ingress/e2e-worker-w0-87e3e6-openshift-default-8989bfbdd-fwgnd"
Sep 19 04:33:47.124324 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:47.124208    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6239a719-1815-4ef5-8b52-a488185c44a7-workload-socket\") pod \"e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk\" (UID: \"6239a719-1815-4ef5-8b52-a488185c44a7\") " pod="openshift-ingress/e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk"
Sep 19 04:33:47.124324 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:47.124293    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6239a719-1815-4ef5-8b52-a488185c44a7-istio-podinfo\") pod \"e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk\" (UID: \"6239a719-1815-4ef5-8b52-a488185c44a7\") " pod="openshift-ingress/e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk"
Sep 19 04:33:47.124559 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:47.124332    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6239a719-1815-4ef5-8b52-a488185c44a7-istio-envoy\") pod \"e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk\" (UID: \"6239a719-1815-4ef5-8b52-a488185c44a7\") " pod="openshift-ingress/e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk"
Sep 19 04:33:47.124559 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:47.124357    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6239a719-1815-4ef5-8b52-a488185c44a7-credential-socket\") pod \"e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk\" (UID: \"6239a719-1815-4ef5-8b52-a488185c44a7\") " pod="openshift-ingress/e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk"
Sep 19 04:33:47.124559 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:47.124471    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6239a719-1815-4ef5-8b52-a488185c44a7-istio-token\") pod \"e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk\" (UID: \"6239a719-1815-4ef5-8b52-a488185c44a7\") " pod="openshift-ingress/e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk"
Sep 19 04:33:47.124559 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:47.124502    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6239a719-1815-4ef5-8b52-a488185c44a7-istio-data\") pod \"e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk\" (UID: \"6239a719-1815-4ef5-8b52-a488185c44a7\") " pod="openshift-ingress/e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk"
Sep 19 04:33:47.124559 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:47.124532    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6239a719-1815-4ef5-8b52-a488185c44a7-workload-certs\") pod \"e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk\" (UID: \"6239a719-1815-4ef5-8b52-a488185c44a7\") " pod="openshift-ingress/e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk"
Sep 19 04:33:47.124834 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:47.124565    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6239a719-1815-4ef5-8b52-a488185c44a7-istiod-ca-cert\") pod \"e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk\" (UID: \"6239a719-1815-4ef5-8b52-a488185c44a7\") " pod="openshift-ingress/e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk"
Sep 19 04:33:47.124834 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:47.124605    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf6bp\" (UniqueName: \"kubernetes.io/projected/6239a719-1815-4ef5-8b52-a488185c44a7-kube-api-access-zf6bp\") pod \"e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk\" (UID: \"6239a719-1815-4ef5-8b52-a488185c44a7\") " pod="openshift-ingress/e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk"
Sep 19 04:33:47.140037 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:47.140008    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-87e3e6-openshift-default-8989bfbdd-fwgnd"
Sep 19 04:33:47.225686 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:47.225638    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6239a719-1815-4ef5-8b52-a488185c44a7-workload-certs\") pod \"e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk\" (UID: \"6239a719-1815-4ef5-8b52-a488185c44a7\") " pod="openshift-ingress/e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk"
Sep 19 04:33:47.225912 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:47.225686    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6239a719-1815-4ef5-8b52-a488185c44a7-istiod-ca-cert\") pod \"e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk\" (UID: \"6239a719-1815-4ef5-8b52-a488185c44a7\") " pod="openshift-ingress/e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk"
Sep 19 04:33:47.225912 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:47.225801    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf6bp\" (UniqueName: \"kubernetes.io/projected/6239a719-1815-4ef5-8b52-a488185c44a7-kube-api-access-zf6bp\") pod \"e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk\" (UID: \"6239a719-1815-4ef5-8b52-a488185c44a7\") " pod="openshift-ingress/e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk"
Sep 19 04:33:47.225912 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:47.225879    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6239a719-1815-4ef5-8b52-a488185c44a7-workload-socket\") pod \"e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk\" (UID: \"6239a719-1815-4ef5-8b52-a488185c44a7\") " pod="openshift-ingress/e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk"
Sep 19 04:33:47.226080 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:47.225940    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6239a719-1815-4ef5-8b52-a488185c44a7-istio-podinfo\") pod \"e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk\" (UID: \"6239a719-1815-4ef5-8b52-a488185c44a7\") " pod="openshift-ingress/e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk"
Sep 19 04:33:47.226080 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:47.225981    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6239a719-1815-4ef5-8b52-a488185c44a7-istio-envoy\") pod \"e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk\" (UID: \"6239a719-1815-4ef5-8b52-a488185c44a7\") " pod="openshift-ingress/e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk"
Sep 19 04:33:47.226080 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:47.226019    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6239a719-1815-4ef5-8b52-a488185c44a7-credential-socket\") pod \"e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk\" (UID: \"6239a719-1815-4ef5-8b52-a488185c44a7\") " pod="openshift-ingress/e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk"
Sep 19 04:33:47.226080 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:47.226047    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6239a719-1815-4ef5-8b52-a488185c44a7-istio-token\") pod \"e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk\" (UID: \"6239a719-1815-4ef5-8b52-a488185c44a7\") " pod="openshift-ingress/e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk"
Sep 19 04:33:47.226080 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:47.226076    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6239a719-1815-4ef5-8b52-a488185c44a7-istio-data\") pod \"e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk\" (UID: \"6239a719-1815-4ef5-8b52-a488185c44a7\") " pod="openshift-ingress/e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk"
Sep 19 04:33:47.226748 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:47.226534    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6239a719-1815-4ef5-8b52-a488185c44a7-istio-data\") pod \"e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk\" (UID: \"6239a719-1815-4ef5-8b52-a488185c44a7\") " pod="openshift-ingress/e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk"
Sep 19 04:33:47.226878 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:47.226857    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6239a719-1815-4ef5-8b52-a488185c44a7-workload-socket\") pod \"e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk\" (UID: \"6239a719-1815-4ef5-8b52-a488185c44a7\") " pod="openshift-ingress/e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk"
Sep 19 04:33:47.228076 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:47.227160    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6239a719-1815-4ef5-8b52-a488185c44a7-credential-socket\") pod \"e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk\" (UID: \"6239a719-1815-4ef5-8b52-a488185c44a7\") " pod="openshift-ingress/e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk"
Sep 19 04:33:47.228076 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:47.227229    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6239a719-1815-4ef5-8b52-a488185c44a7-workload-certs\") pod \"e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk\" (UID: \"6239a719-1815-4ef5-8b52-a488185c44a7\") " pod="openshift-ingress/e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk"
Sep 19 04:33:47.228076 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:47.227853    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6239a719-1815-4ef5-8b52-a488185c44a7-istiod-ca-cert\") pod \"e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk\" (UID: \"6239a719-1815-4ef5-8b52-a488185c44a7\") " pod="openshift-ingress/e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk"
Sep 19 04:33:47.229964 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:47.229914    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6239a719-1815-4ef5-8b52-a488185c44a7-istio-envoy\") pod \"e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk\" (UID: \"6239a719-1815-4ef5-8b52-a488185c44a7\") " pod="openshift-ingress/e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk"
Sep 19 04:33:47.230288 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:47.230263    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6239a719-1815-4ef5-8b52-a488185c44a7-istio-podinfo\") pod \"e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk\" (UID: \"6239a719-1815-4ef5-8b52-a488185c44a7\") " pod="openshift-ingress/e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk"
Sep 19 04:33:47.251343 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:47.251308    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf6bp\" (UniqueName: \"kubernetes.io/projected/6239a719-1815-4ef5-8b52-a488185c44a7-kube-api-access-zf6bp\") pod \"e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk\" (UID: \"6239a719-1815-4ef5-8b52-a488185c44a7\") " pod="openshift-ingress/e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk"
Sep 19 04:33:47.262946 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:47.262912    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6239a719-1815-4ef5-8b52-a488185c44a7-istio-token\") pod \"e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk\" (UID: \"6239a719-1815-4ef5-8b52-a488185c44a7\") " pod="openshift-ingress/e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk"
Sep 19 04:33:47.283564 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:47.283531    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk"
Sep 19 04:33:47.454053 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:47.454022    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-87e3e6-openshift-default-8989bfbdd-fwgnd"]
Sep 19 04:33:47.455188 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:33:47.455159    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39081cb6_0611_41e5_b50e_bfe7759ef529.slice/crio-bfcf9ee85a7595e5fd6c96f103ba105618ffcd540435bca411ac82b62d02c43a WatchSource:0}: Error finding container bfcf9ee85a7595e5fd6c96f103ba105618ffcd540435bca411ac82b62d02c43a: Status 404 returned error can't find the container with id bfcf9ee85a7595e5fd6c96f103ba105618ffcd540435bca411ac82b62d02c43a
Sep 19 04:33:47.459810 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:47.459775    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 04:33:47.459890 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:47.459849    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 04:33:47.459890 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:47.459878    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 04:33:47.588869 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:47.588834    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk"]
Sep 19 04:33:47.593173 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:33:47.593137    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6239a719_1815_4ef5_8b52_a488185c44a7.slice/crio-7163390a983df51ec232e9daac7d3634a5e5b6ed7065ed9cf91df63b1312c442 WatchSource:0}: Error finding container 7163390a983df51ec232e9daac7d3634a5e5b6ed7065ed9cf91df63b1312c442: Status 404 returned error can't find the container with id 7163390a983df51ec232e9daac7d3634a5e5b6ed7065ed9cf91df63b1312c442
Sep 19 04:33:47.596174 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:47.596126    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 04:33:47.596250 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:47.596232    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 04:33:47.596296 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:47.596284    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 04:33:48.294180 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:48.294139    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-87e3e6-openshift-default-8989bfbdd-fwgnd" event={"ID":"39081cb6-0611-41e5-b50e-bfe7759ef529","Type":"ContainerStarted","Data":"40df442f47aa4c4fbcbd07b89753e89ee38a79d4b0ce19eed2f6d50264f2d0cc"}
Sep 19 04:33:48.294659 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:48.294205    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-87e3e6-openshift-default-8989bfbdd-fwgnd" event={"ID":"39081cb6-0611-41e5-b50e-bfe7759ef529","Type":"ContainerStarted","Data":"bfcf9ee85a7595e5fd6c96f103ba105618ffcd540435bca411ac82b62d02c43a"}
Sep 19 04:33:48.296024 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:48.296003    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk" event={"ID":"6239a719-1815-4ef5-8b52-a488185c44a7","Type":"ContainerStarted","Data":"d57b524321eff87e26ffa888663d27890113781fdb13fea27b1883045f22b8e9"}
Sep 19 04:33:48.296106 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:48.296040    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk" event={"ID":"6239a719-1815-4ef5-8b52-a488185c44a7","Type":"ContainerStarted","Data":"7163390a983df51ec232e9daac7d3634a5e5b6ed7065ed9cf91df63b1312c442"}
Sep 19 04:33:48.325357 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:48.325308    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-87e3e6-openshift-default-8989bfbdd-fwgnd" podStartSLOduration=2.325293138 podStartE2EDuration="2.325293138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 04:33:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:33:48.322438425 +0000 UTC m=+1207.505399574" watchObservedRunningTime="2026-09-19 04:33:48.325293138 +0000 UTC m=+1207.508254287"
Sep 19 04:33:48.348670 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:48.348613    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk" podStartSLOduration=2.348595766 podStartE2EDuration="2.348595766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 04:33:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:33:48.346899966 +0000 UTC m=+1207.529861115" watchObservedRunningTime="2026-09-19 04:33:48.348595766 +0000 UTC m=+1207.531556980"
Sep 19 04:33:49.140642 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:49.140604    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-87e3e6-openshift-default-8989bfbdd-fwgnd"
Sep 19 04:33:49.146150 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:49.146122    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-87e3e6-openshift-default-8989bfbdd-fwgnd"
Sep 19 04:33:49.284407 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:49.284359    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk"
Sep 19 04:33:49.289792 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:49.289767    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk"
Sep 19 04:33:49.302422 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:49.302389    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-87e3e6-openshift-default-8989bfbdd-fwgnd"
Sep 19 04:33:49.302872 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:49.302442    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk"
Sep 19 04:33:49.303075 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:49.303050    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-87e3e6-openshift-default-8989bfbdd-fwgnd"
Sep 19 04:33:49.303186 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:49.303140    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk"
Sep 19 04:33:52.324070 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:52.324037    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-87e3e6-7f7f6b4cc9-p22q5"]
Sep 19 04:33:52.328910 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:52.328893    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-87e3e6-7f7f6b4cc9-p22q5"
Sep 19 04:33:52.332069 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:52.332044    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-87e3e6\""
Sep 19 04:33:52.332069 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:52.332044    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-87e3e6-dockercfg-hrzc7\""
Sep 19 04:33:52.342566 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:52.342540    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-87e3e6-7f7f6b4cc9-p22q5"]
Sep 19 04:33:52.365906 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:52.365872    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-87e3e6-749ffc4498-lv2h9"]
Sep 19 04:33:52.369454 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:52.369438    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-87e3e6-749ffc4498-lv2h9"
Sep 19 04:33:52.380189 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:52.380158    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-87e3e6-749ffc4498-lv2h9"]
Sep 19 04:33:52.482522 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:52.482480    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snfkz\" (UniqueName: \"kubernetes.io/projected/be72f493-a1a7-4627-8130-cc828a2fdd82-kube-api-access-snfkz\") pod \"payload-pre-processing-e2e-worker-w0-87e3e6-749ffc4498-lv2h9\" (UID: \"be72f493-a1a7-4627-8130-cc828a2fdd82\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-87e3e6-749ffc4498-lv2h9"
Sep 19 04:33:52.482741 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:52.482538    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77vbk\" (UniqueName: \"kubernetes.io/projected/ee412ff9-2199-4f68-a185-fe1b4c59cf78-kube-api-access-77vbk\") pod \"payload-processing-e2e-worker-w0-87e3e6-7f7f6b4cc9-p22q5\" (UID: \"ee412ff9-2199-4f68-a185-fe1b4c59cf78\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-87e3e6-7f7f6b4cc9-p22q5"
Sep 19 04:33:52.482741 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:52.482568    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be72f493-a1a7-4627-8130-cc828a2fdd82-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-87e3e6-749ffc4498-lv2h9\" (UID: \"be72f493-a1a7-4627-8130-cc828a2fdd82\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-87e3e6-749ffc4498-lv2h9"
Sep 19 04:33:52.482875 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:52.482725    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee412ff9-2199-4f68-a185-fe1b4c59cf78-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-87e3e6-7f7f6b4cc9-p22q5\" (UID: \"ee412ff9-2199-4f68-a185-fe1b4c59cf78\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-87e3e6-7f7f6b4cc9-p22q5"
Sep 19 04:33:52.584030 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:52.583932    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snfkz\" (UniqueName: \"kubernetes.io/projected/be72f493-a1a7-4627-8130-cc828a2fdd82-kube-api-access-snfkz\") pod \"payload-pre-processing-e2e-worker-w0-87e3e6-749ffc4498-lv2h9\" (UID: \"be72f493-a1a7-4627-8130-cc828a2fdd82\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-87e3e6-749ffc4498-lv2h9"
Sep 19 04:33:52.584275 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:52.584254    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77vbk\" (UniqueName: \"kubernetes.io/projected/ee412ff9-2199-4f68-a185-fe1b4c59cf78-kube-api-access-77vbk\") pod \"payload-processing-e2e-worker-w0-87e3e6-7f7f6b4cc9-p22q5\" (UID: \"ee412ff9-2199-4f68-a185-fe1b4c59cf78\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-87e3e6-7f7f6b4cc9-p22q5"
Sep 19 04:33:52.584401 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:52.584385    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be72f493-a1a7-4627-8130-cc828a2fdd82-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-87e3e6-749ffc4498-lv2h9\" (UID: \"be72f493-a1a7-4627-8130-cc828a2fdd82\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-87e3e6-749ffc4498-lv2h9"
Sep 19 04:33:52.584552 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:52.584537    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee412ff9-2199-4f68-a185-fe1b4c59cf78-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-87e3e6-7f7f6b4cc9-p22q5\" (UID: \"ee412ff9-2199-4f68-a185-fe1b4c59cf78\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-87e3e6-7f7f6b4cc9-p22q5"
Sep 19 04:33:52.585434 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:52.585400    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be72f493-a1a7-4627-8130-cc828a2fdd82-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-87e3e6-749ffc4498-lv2h9\" (UID: \"be72f493-a1a7-4627-8130-cc828a2fdd82\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-87e3e6-749ffc4498-lv2h9"
Sep 19 04:33:52.585562 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:52.585463    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee412ff9-2199-4f68-a185-fe1b4c59cf78-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-87e3e6-7f7f6b4cc9-p22q5\" (UID: \"ee412ff9-2199-4f68-a185-fe1b4c59cf78\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-87e3e6-7f7f6b4cc9-p22q5"
Sep 19 04:33:52.598372 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:52.598340    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-snfkz\" (UniqueName: \"kubernetes.io/projected/be72f493-a1a7-4627-8130-cc828a2fdd82-kube-api-access-snfkz\") pod \"payload-pre-processing-e2e-worker-w0-87e3e6-749ffc4498-lv2h9\" (UID: \"be72f493-a1a7-4627-8130-cc828a2fdd82\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-87e3e6-749ffc4498-lv2h9"
Sep 19 04:33:52.598535 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:52.598404    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77vbk\" (UniqueName: \"kubernetes.io/projected/ee412ff9-2199-4f68-a185-fe1b4c59cf78-kube-api-access-77vbk\") pod \"payload-processing-e2e-worker-w0-87e3e6-7f7f6b4cc9-p22q5\" (UID: \"ee412ff9-2199-4f68-a185-fe1b4c59cf78\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-87e3e6-7f7f6b4cc9-p22q5"
Sep 19 04:33:52.640433 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:52.640392    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-87e3e6-7f7f6b4cc9-p22q5"
Sep 19 04:33:52.680536 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:52.680500    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-87e3e6-749ffc4498-lv2h9"
Sep 19 04:33:52.945758 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:33:52.945723    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee412ff9_2199_4f68_a185_fe1b4c59cf78.slice/crio-ee209eef5c41d59796dd5b4e4fd7e9eb1aaf2690457e882f943353edf7651bf9 WatchSource:0}: Error finding container ee209eef5c41d59796dd5b4e4fd7e9eb1aaf2690457e882f943353edf7651bf9: Status 404 returned error can't find the container with id ee209eef5c41d59796dd5b4e4fd7e9eb1aaf2690457e882f943353edf7651bf9
Sep 19 04:33:52.946321 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:52.946292    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-87e3e6-7f7f6b4cc9-p22q5"]
Sep 19 04:33:52.989517 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:33:52.989485    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe72f493_a1a7_4627_8130_cc828a2fdd82.slice/crio-db7eafb5a5122746f3f2bf9648211af6714fc7cbeda3ceeba985c621c379a86b WatchSource:0}: Error finding container db7eafb5a5122746f3f2bf9648211af6714fc7cbeda3ceeba985c621c379a86b: Status 404 returned error can't find the container with id db7eafb5a5122746f3f2bf9648211af6714fc7cbeda3ceeba985c621c379a86b
Sep 19 04:33:52.989943 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:52.989917    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-87e3e6-749ffc4498-lv2h9"]
Sep 19 04:33:53.326207 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:53.326114    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-87e3e6-749ffc4498-lv2h9" event={"ID":"be72f493-a1a7-4627-8130-cc828a2fdd82","Type":"ContainerStarted","Data":"0cefaf98f7ea4927e376d3bcc9161b443f26e971bbdfff1514c34bacf07ce862"}
Sep 19 04:33:53.326207 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:53.326173    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-87e3e6-749ffc4498-lv2h9" event={"ID":"be72f493-a1a7-4627-8130-cc828a2fdd82","Type":"ContainerStarted","Data":"db7eafb5a5122746f3f2bf9648211af6714fc7cbeda3ceeba985c621c379a86b"}
Sep 19 04:33:53.326885 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:53.326239    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-87e3e6-749ffc4498-lv2h9"
Sep 19 04:33:53.328128 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:53.328102    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-87e3e6-7f7f6b4cc9-p22q5" event={"ID":"ee412ff9-2199-4f68-a185-fe1b4c59cf78","Type":"ContainerStarted","Data":"0f8224599559efb6ebfd193dbc1f309df9047b3fc23f2893773e2d4ca8b3bc8f"}
Sep 19 04:33:53.328254 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:53.328147    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-87e3e6-7f7f6b4cc9-p22q5" event={"ID":"ee412ff9-2199-4f68-a185-fe1b4c59cf78","Type":"ContainerStarted","Data":"ee209eef5c41d59796dd5b4e4fd7e9eb1aaf2690457e882f943353edf7651bf9"}
Sep 19 04:33:53.328311 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:53.328274    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-87e3e6-7f7f6b4cc9-p22q5"
Sep 19 04:33:53.352089 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:53.352033    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-87e3e6-749ffc4498-lv2h9" podStartSLOduration=1.35201328 podStartE2EDuration="1.35201328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 04:33:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:33:53.348333384 +0000 UTC m=+1212.531294532" watchObservedRunningTime="2026-09-19 04:33:53.35201328 +0000 UTC m=+1212.534974429"
Sep 19 04:33:53.370322 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:53.370269    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-87e3e6-7f7f6b4cc9-p22q5" podStartSLOduration=1.370251339 podStartE2EDuration="1.370251339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 04:33:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:33:53.369111491 +0000 UTC m=+1212.552072655" watchObservedRunningTime="2026-09-19 04:33:53.370251339 +0000 UTC m=+1212.553212524"
Sep 19 04:33:55.142048 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:55.142007    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-d0f416-6448755f7-d7wgd"]
Sep 19 04:33:55.147318 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:55.147289    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-d0f416-6448755f7-d7wgd"
Sep 19 04:33:55.151224 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:55.151197    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-d0f416\""
Sep 19 04:33:55.152273 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:55.152246    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-d0f416-dockercfg-g5rkk\""
Sep 19 04:33:55.162342 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:55.162317    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-d0f416-6448755f7-d7wgd"]
Sep 19 04:33:55.227196 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:55.227155    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc7r7\" (UniqueName: \"kubernetes.io/projected/13dcb6c5-a168-4fbf-8a3f-177d77dea22a-kube-api-access-zc7r7\") pod \"payload-processing-e2e-worker-w4-d0f416-6448755f7-d7wgd\" (UID: \"13dcb6c5-a168-4fbf-8a3f-177d77dea22a\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-d0f416-6448755f7-d7wgd"
Sep 19 04:33:55.227392 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:55.227319    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13dcb6c5-a168-4fbf-8a3f-177d77dea22a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-d0f416-6448755f7-d7wgd\" (UID: \"13dcb6c5-a168-4fbf-8a3f-177d77dea22a\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-d0f416-6448755f7-d7wgd"
Sep 19 04:33:55.328612 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:55.328567    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13dcb6c5-a168-4fbf-8a3f-177d77dea22a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-d0f416-6448755f7-d7wgd\" (UID: \"13dcb6c5-a168-4fbf-8a3f-177d77dea22a\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-d0f416-6448755f7-d7wgd"
Sep 19 04:33:55.328796 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:55.328654    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc7r7\" (UniqueName: \"kubernetes.io/projected/13dcb6c5-a168-4fbf-8a3f-177d77dea22a-kube-api-access-zc7r7\") pod \"payload-processing-e2e-worker-w4-d0f416-6448755f7-d7wgd\" (UID: \"13dcb6c5-a168-4fbf-8a3f-177d77dea22a\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-d0f416-6448755f7-d7wgd"
Sep 19 04:33:55.329551 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:55.329525    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13dcb6c5-a168-4fbf-8a3f-177d77dea22a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-d0f416-6448755f7-d7wgd\" (UID: \"13dcb6c5-a168-4fbf-8a3f-177d77dea22a\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-d0f416-6448755f7-d7wgd"
Sep 19 04:33:55.340081 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:55.340048    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc7r7\" (UniqueName: \"kubernetes.io/projected/13dcb6c5-a168-4fbf-8a3f-177d77dea22a-kube-api-access-zc7r7\") pod \"payload-processing-e2e-worker-w4-d0f416-6448755f7-d7wgd\" (UID: \"13dcb6c5-a168-4fbf-8a3f-177d77dea22a\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-d0f416-6448755f7-d7wgd"
Sep 19 04:33:55.459621 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:55.459583    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-d0f416-6448755f7-d7wgd"
Sep 19 04:33:55.757513 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:55.757458    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-d0f416-6448755f7-d7wgd"]
Sep 19 04:33:55.760550 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:33:55.760509    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13dcb6c5_a168_4fbf_8a3f_177d77dea22a.slice/crio-c14e7a1836bebc07a9adbbdd34230d2c740e6e7c1684ac17794df169b8dac221 WatchSource:0}: Error finding container c14e7a1836bebc07a9adbbdd34230d2c740e6e7c1684ac17794df169b8dac221: Status 404 returned error can't find the container with id c14e7a1836bebc07a9adbbdd34230d2c740e6e7c1684ac17794df169b8dac221
Sep 19 04:33:56.348370 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:56.348324    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-d0f416-6448755f7-d7wgd" event={"ID":"13dcb6c5-a168-4fbf-8a3f-177d77dea22a","Type":"ContainerStarted","Data":"956b7f471e2a7ce3e7a1d2858703f3b6063f6f17654dbd9ff7420d4f6a362d35"}
Sep 19 04:33:56.348887 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:56.348383    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-d0f416-6448755f7-d7wgd" event={"ID":"13dcb6c5-a168-4fbf-8a3f-177d77dea22a","Type":"ContainerStarted","Data":"c14e7a1836bebc07a9adbbdd34230d2c740e6e7c1684ac17794df169b8dac221"}
Sep 19 04:33:56.348887 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:56.348469    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-d0f416-6448755f7-d7wgd"
Sep 19 04:33:56.397336 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:33:56.397279    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-d0f416-6448755f7-d7wgd" podStartSLOduration=1.397260288 podStartE2EDuration="1.397260288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 04:33:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:33:56.396863833 +0000 UTC m=+1215.579824976" watchObservedRunningTime="2026-09-19 04:33:56.397260288 +0000 UTC m=+1215.580221437"
Sep 19 04:34:02.412532 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:02.412494    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-a42681-7c7b879c89-9gjfv"]
Sep 19 04:34:02.417419 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:02.417398    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-a42681-7c7b879c89-9gjfv"
Sep 19 04:34:02.420582 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:02.420563    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-a42681\""
Sep 19 04:34:02.433590 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:02.433564    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-a42681-7c7b879c89-9gjfv"]
Sep 19 04:34:02.495832 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:02.495749    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ae90dc0-c97f-4b94-be86-c434a89d100e-plugins-config-volume\") pod \"payload-processing-e2e-autores-a42681-7c7b879c89-9gjfv\" (UID: \"8ae90dc0-c97f-4b94-be86-c434a89d100e\") " pod="openshift-ingress/payload-processing-e2e-autores-a42681-7c7b879c89-9gjfv"
Sep 19 04:34:02.495832 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:02.495800    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8btwk\" (UniqueName: \"kubernetes.io/projected/8ae90dc0-c97f-4b94-be86-c434a89d100e-kube-api-access-8btwk\") pod \"payload-processing-e2e-autores-a42681-7c7b879c89-9gjfv\" (UID: \"8ae90dc0-c97f-4b94-be86-c434a89d100e\") " pod="openshift-ingress/payload-processing-e2e-autores-a42681-7c7b879c89-9gjfv"
Sep 19 04:34:02.596433 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:02.596383    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ae90dc0-c97f-4b94-be86-c434a89d100e-plugins-config-volume\") pod \"payload-processing-e2e-autores-a42681-7c7b879c89-9gjfv\" (UID: \"8ae90dc0-c97f-4b94-be86-c434a89d100e\") " pod="openshift-ingress/payload-processing-e2e-autores-a42681-7c7b879c89-9gjfv"
Sep 19 04:34:02.596433 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:02.596432    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8btwk\" (UniqueName: \"kubernetes.io/projected/8ae90dc0-c97f-4b94-be86-c434a89d100e-kube-api-access-8btwk\") pod \"payload-processing-e2e-autores-a42681-7c7b879c89-9gjfv\" (UID: \"8ae90dc0-c97f-4b94-be86-c434a89d100e\") " pod="openshift-ingress/payload-processing-e2e-autores-a42681-7c7b879c89-9gjfv"
Sep 19 04:34:02.597274 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:02.597252    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ae90dc0-c97f-4b94-be86-c434a89d100e-plugins-config-volume\") pod \"payload-processing-e2e-autores-a42681-7c7b879c89-9gjfv\" (UID: \"8ae90dc0-c97f-4b94-be86-c434a89d100e\") " pod="openshift-ingress/payload-processing-e2e-autores-a42681-7c7b879c89-9gjfv"
Sep 19 04:34:02.612571 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:02.612540    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8btwk\" (UniqueName: \"kubernetes.io/projected/8ae90dc0-c97f-4b94-be86-c434a89d100e-kube-api-access-8btwk\") pod \"payload-processing-e2e-autores-a42681-7c7b879c89-9gjfv\" (UID: \"8ae90dc0-c97f-4b94-be86-c434a89d100e\") " pod="openshift-ingress/payload-processing-e2e-autores-a42681-7c7b879c89-9gjfv"
Sep 19 04:34:02.727299 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:02.727204    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-a42681-7c7b879c89-9gjfv"
Sep 19 04:34:03.013379 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:03.013166    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-a42681-7c7b879c89-9gjfv"]
Sep 19 04:34:03.015732 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:34:03.015674    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ae90dc0_c97f_4b94_be86_c434a89d100e.slice/crio-2616500cac48e914490dc2e4a124b416577bce721bfe31de47ae338f88a10518 WatchSource:0}: Error finding container 2616500cac48e914490dc2e4a124b416577bce721bfe31de47ae338f88a10518: Status 404 returned error can't find the container with id 2616500cac48e914490dc2e4a124b416577bce721bfe31de47ae338f88a10518
Sep 19 04:34:03.391617 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:03.391515    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-a42681-7c7b879c89-9gjfv" event={"ID":"8ae90dc0-c97f-4b94-be86-c434a89d100e","Type":"ContainerStarted","Data":"e28ff8512773cf948391e67cfe5731b9621b5fcc219651b5fca070748abd7ae1"}
Sep 19 04:34:03.391617 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:03.391580    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-a42681-7c7b879c89-9gjfv" event={"ID":"8ae90dc0-c97f-4b94-be86-c434a89d100e","Type":"ContainerStarted","Data":"2616500cac48e914490dc2e4a124b416577bce721bfe31de47ae338f88a10518"}
Sep 19 04:34:03.391895 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:03.391662    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-a42681-7c7b879c89-9gjfv"
Sep 19 04:34:03.439061 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:03.438992    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-a42681-7c7b879c89-9gjfv" podStartSLOduration=1.438970436 podStartE2EDuration="1.438970436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 04:34:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:34:03.432719002 +0000 UTC m=+1222.615680147" watchObservedRunningTime="2026-09-19 04:34:03.438970436 +0000 UTC m=+1222.621931587"
Sep 19 04:34:04.336405 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:04.336376    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-87e3e6-7f7f6b4cc9-p22q5"
Sep 19 04:34:04.336603 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:04.336451    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-87e3e6-749ffc4498-lv2h9"
Sep 19 04:34:07.361433 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:07.361402    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-d0f416-6448755f7-d7wgd"
Sep 19 04:34:09.116721 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:09.116071    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e8aaa06f-b598d6fdc-hqplp"]
Sep 19 04:34:09.130301 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:09.130270    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e8aaa06f-b598d6fdc-hqplp"
Sep 19 04:34:09.135211 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:09.134757    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-e8aaa06f\""
Sep 19 04:34:09.138919 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:09.138891    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e8aaa06f-b598d6fdc-hqplp"]
Sep 19 04:34:09.172651 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:09.172620    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hm7xk\" (UniqueName: \"kubernetes.io/projected/e4173388-3e28-4437-9044-491413e66ace-kube-api-access-hm7xk\") pod \"payload-processing-e2e-ait-e8aaa06f-b598d6fdc-hqplp\" (UID: \"e4173388-3e28-4437-9044-491413e66ace\") " pod="openshift-ingress/payload-processing-e2e-ait-e8aaa06f-b598d6fdc-hqplp"
Sep 19 04:34:09.172874 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:09.172684    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4173388-3e28-4437-9044-491413e66ace-plugins-config-volume\") pod \"payload-processing-e2e-ait-e8aaa06f-b598d6fdc-hqplp\" (UID: \"e4173388-3e28-4437-9044-491413e66ace\") " pod="openshift-ingress/payload-processing-e2e-ait-e8aaa06f-b598d6fdc-hqplp"
Sep 19 04:34:09.273474 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:09.273442    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hm7xk\" (UniqueName: \"kubernetes.io/projected/e4173388-3e28-4437-9044-491413e66ace-kube-api-access-hm7xk\") pod \"payload-processing-e2e-ait-e8aaa06f-b598d6fdc-hqplp\" (UID: \"e4173388-3e28-4437-9044-491413e66ace\") " pod="openshift-ingress/payload-processing-e2e-ait-e8aaa06f-b598d6fdc-hqplp"
Sep 19 04:34:09.273636 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:09.273512    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4173388-3e28-4437-9044-491413e66ace-plugins-config-volume\") pod \"payload-processing-e2e-ait-e8aaa06f-b598d6fdc-hqplp\" (UID: \"e4173388-3e28-4437-9044-491413e66ace\") " pod="openshift-ingress/payload-processing-e2e-ait-e8aaa06f-b598d6fdc-hqplp"
Sep 19 04:34:09.274405 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:09.274387    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4173388-3e28-4437-9044-491413e66ace-plugins-config-volume\") pod \"payload-processing-e2e-ait-e8aaa06f-b598d6fdc-hqplp\" (UID: \"e4173388-3e28-4437-9044-491413e66ace\") " pod="openshift-ingress/payload-processing-e2e-ait-e8aaa06f-b598d6fdc-hqplp"
Sep 19 04:34:09.301900 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:09.301866    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hm7xk\" (UniqueName: \"kubernetes.io/projected/e4173388-3e28-4437-9044-491413e66ace-kube-api-access-hm7xk\") pod \"payload-processing-e2e-ait-e8aaa06f-b598d6fdc-hqplp\" (UID: \"e4173388-3e28-4437-9044-491413e66ace\") " pod="openshift-ingress/payload-processing-e2e-ait-e8aaa06f-b598d6fdc-hqplp"
Sep 19 04:34:09.448499 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:09.448470    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e8aaa06f-b598d6fdc-hqplp"
Sep 19 04:34:09.798774 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:09.798605    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e8aaa06f-b598d6fdc-hqplp"]
Sep 19 04:34:09.818381 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:34:09.818035    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4173388_3e28_4437_9044_491413e66ace.slice/crio-24d6cfaae0a2fb2f356a4aeaf8fb6e8f8d75164d89e05fc834c41d2cb8261090 WatchSource:0}: Error finding container 24d6cfaae0a2fb2f356a4aeaf8fb6e8f8d75164d89e05fc834c41d2cb8261090: Status 404 returned error can't find the container with id 24d6cfaae0a2fb2f356a4aeaf8fb6e8f8d75164d89e05fc834c41d2cb8261090
Sep 19 04:34:10.473895 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:10.473864    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e8aaa06f-b598d6fdc-hqplp" event={"ID":"e4173388-3e28-4437-9044-491413e66ace","Type":"ContainerStarted","Data":"f3a9f170c7756e569479208b1ab6eb4fab6cda8abfe4da9c4b7187caaac32ae1"}
Sep 19 04:34:10.474338 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:10.473922    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e8aaa06f-b598d6fdc-hqplp" event={"ID":"e4173388-3e28-4437-9044-491413e66ace","Type":"ContainerStarted","Data":"24d6cfaae0a2fb2f356a4aeaf8fb6e8f8d75164d89e05fc834c41d2cb8261090"}
Sep 19 04:34:10.474338 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:10.474000    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-e8aaa06f-b598d6fdc-hqplp"
Sep 19 04:34:14.400080 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:14.400048    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-a42681-7c7b879c89-9gjfv"
Sep 19 04:34:14.441612 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:14.441548    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-e8aaa06f-b598d6fdc-hqplp" podStartSLOduration=5.441526216 podStartE2EDuration="5.441526216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 04:34:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:34:10.523114611 +0000 UTC m=+1229.706075761" watchObservedRunningTime="2026-09-19 04:34:14.441526216 +0000 UTC m=+1233.624487365"
Sep 19 04:34:14.517763 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:14.517729    2579 generic.go:356] "Generic (PLEG): container finished" podID="6239a719-1815-4ef5-8b52-a488185c44a7" containerID="d57b524321eff87e26ffa888663d27890113781fdb13fea27b1883045f22b8e9" exitCode=0
Sep 19 04:34:14.517935 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:14.517813    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk" event={"ID":"6239a719-1815-4ef5-8b52-a488185c44a7","Type":"ContainerDied","Data":"d57b524321eff87e26ffa888663d27890113781fdb13fea27b1883045f22b8e9"}
Sep 19 04:34:14.518413 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:14.518388    2579 scope.go:117] "RemoveContainer" containerID="d57b524321eff87e26ffa888663d27890113781fdb13fea27b1883045f22b8e9"
Sep 19 04:34:14.520117 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:14.520075    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 04:34:14.520204 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:14.520168    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 04:34:14.520243 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:14.520214    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 04:34:15.530336 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:15.530294    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk" event={"ID":"6239a719-1815-4ef5-8b52-a488185c44a7","Type":"ContainerStarted","Data":"d8ddcbc8882c2d447c6ec7997a8923646042ddb37493699e66940f73c0982e2b"}
Sep 19 04:34:16.284481 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:16.284366    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk"
Sep 19 04:34:16.286369 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:16.286333    2579 patch_prober.go:28] interesting pod/e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Sep 19 04:34:16.286522 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:16.286401    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk" podUID="6239a719-1815-4ef5-8b52-a488185c44a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Sep 19 04:34:16.537644 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:16.537553    2579 generic.go:356] "Generic (PLEG): container finished" podID="39081cb6-0611-41e5-b50e-bfe7759ef529" containerID="40df442f47aa4c4fbcbd07b89753e89ee38a79d4b0ce19eed2f6d50264f2d0cc" exitCode=0
Sep 19 04:34:16.537644 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:16.537625    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-87e3e6-openshift-default-8989bfbdd-fwgnd" event={"ID":"39081cb6-0611-41e5-b50e-bfe7759ef529","Type":"ContainerDied","Data":"40df442f47aa4c4fbcbd07b89753e89ee38a79d4b0ce19eed2f6d50264f2d0cc"}
Sep 19 04:34:16.538065 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:16.538050    2579 scope.go:117] "RemoveContainer" containerID="40df442f47aa4c4fbcbd07b89753e89ee38a79d4b0ce19eed2f6d50264f2d0cc"
Sep 19 04:34:16.539192 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:16.539163    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 04:34:16.539297 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:16.539242    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 04:34:16.539297 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:16.539289    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 04:34:17.285497 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:17.285012    2579 patch_prober.go:28] interesting pod/e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Sep 19 04:34:17.285497 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:17.285101    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk" podUID="6239a719-1815-4ef5-8b52-a488185c44a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Sep 19 04:34:17.552227 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:17.552126    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-87e3e6-openshift-default-8989bfbdd-fwgnd" event={"ID":"39081cb6-0611-41e5-b50e-bfe7759ef529","Type":"ContainerStarted","Data":"7767b9aaefb207bc507d2ed7cf4cf100de783b3e1c49dc120633ec33b925c219"}
Sep 19 04:34:18.140843 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:18.140808    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-87e3e6-openshift-default-8989bfbdd-fwgnd"
Sep 19 04:34:18.141920 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:18.141885    2579 patch_prober.go:28] interesting pod/e2e-worker-w0-87e3e6-openshift-default-8989bfbdd-fwgnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Sep 19 04:34:18.142065 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:18.141949    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-87e3e6-openshift-default-8989bfbdd-fwgnd" podUID="39081cb6-0611-41e5-b50e-bfe7759ef529" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Sep 19 04:34:18.285428 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:18.285389    2579 patch_prober.go:28] interesting pod/e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Sep 19 04:34:18.285751 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:18.285715    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk" podUID="6239a719-1815-4ef5-8b52-a488185c44a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Sep 19 04:34:18.400604 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:18.400483    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e8aaa06f-pzrrk"]
Sep 19 04:34:18.405189 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:18.405162    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e8aaa06f-pzrrk"
Sep 19 04:34:18.411444 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:18.411418    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s6xbq\""
Sep 19 04:34:18.411734 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:18.411707    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 19 04:34:18.418126 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:18.418101    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e8aaa06f-pzrrk"]
Sep 19 04:34:18.486354 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:18.486304    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1de0ac98-7c69-4e78-9fec-663ea21715f7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e8aaa06f-pzrrk\" (UID: \"1de0ac98-7c69-4e78-9fec-663ea21715f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e8aaa06f-pzrrk"
Sep 19 04:34:18.588912 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:18.587967    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1de0ac98-7c69-4e78-9fec-663ea21715f7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e8aaa06f-pzrrk\" (UID: \"1de0ac98-7c69-4e78-9fec-663ea21715f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e8aaa06f-pzrrk"
Sep 19 04:34:18.588912 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:18.588829    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1de0ac98-7c69-4e78-9fec-663ea21715f7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e8aaa06f-pzrrk\" (UID: \"1de0ac98-7c69-4e78-9fec-663ea21715f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e8aaa06f-pzrrk"
Sep 19 04:34:18.726150 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:18.726112    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e8aaa06f-pzrrk"
Sep 19 04:34:19.146186 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:19.144117    2579 patch_prober.go:28] interesting pod/e2e-worker-w0-87e3e6-openshift-default-8989bfbdd-fwgnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Sep 19 04:34:19.146186 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:19.144184    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-87e3e6-openshift-default-8989bfbdd-fwgnd" podUID="39081cb6-0611-41e5-b50e-bfe7759ef529" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Sep 19 04:34:19.208719 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:19.205323    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e8aaa06f-pzrrk"]
Sep 19 04:34:19.291721 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:19.290134    2579 patch_prober.go:28] interesting pod/e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Sep 19 04:34:19.291721 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:19.290191    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk" podUID="6239a719-1815-4ef5-8b52-a488185c44a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Sep 19 04:34:19.306722 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:19.306643    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk"
Sep 19 04:34:19.310721 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:19.307510    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-87e3e6-openshift-default-8989bfbdd-fwgnd"
Sep 19 04:34:19.565992 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:19.565955    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e8aaa06f-pzrrk" event={"ID":"1de0ac98-7c69-4e78-9fec-663ea21715f7","Type":"ContainerStarted","Data":"8a27be5347994cc54454058ae59800a091a8e29483f3b57a5b22d25f0ff64ea7"}
Sep 19 04:34:20.140473 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:20.140427    2579 patch_prober.go:28] interesting pod/e2e-worker-w0-87e3e6-openshift-default-8989bfbdd-fwgnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Sep 19 04:34:20.141056 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:20.140496    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-87e3e6-openshift-default-8989bfbdd-fwgnd" podUID="39081cb6-0611-41e5-b50e-bfe7759ef529" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Sep 19 04:34:20.284642 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:20.284600    2579 patch_prober.go:28] interesting pod/e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Sep 19 04:34:20.284865 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:20.284674    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk" podUID="6239a719-1815-4ef5-8b52-a488185c44a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Sep 19 04:34:21.141417 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:21.141370    2579 patch_prober.go:28] interesting pod/e2e-worker-w0-87e3e6-openshift-default-8989bfbdd-fwgnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Sep 19 04:34:21.141964 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:21.141439    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-87e3e6-openshift-default-8989bfbdd-fwgnd" podUID="39081cb6-0611-41e5-b50e-bfe7759ef529" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Sep 19 04:34:21.333237 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:21.333200    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk"
Sep 19 04:34:21.337350 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:21.337321    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk"
Sep 19 04:34:21.485882 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:21.485845    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-e8aaa06f-b598d6fdc-hqplp"
Sep 19 04:34:22.140444 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:22.140403    2579 patch_prober.go:28] interesting pod/e2e-worker-w0-87e3e6-openshift-default-8989bfbdd-fwgnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Sep 19 04:34:22.140624 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:22.140462    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-87e3e6-openshift-default-8989bfbdd-fwgnd" podUID="39081cb6-0611-41e5-b50e-bfe7759ef529" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Sep 19 04:34:23.207046 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:23.207007    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-87e3e6-openshift-default-8989bfbdd-fwgnd"
Sep 19 04:34:23.209013 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:23.208986    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-87e3e6-openshift-default-8989bfbdd-fwgnd"
Sep 19 04:34:28.627819 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:28.627784    2579 generic.go:356] "Generic (PLEG): container finished" podID="1de0ac98-7c69-4e78-9fec-663ea21715f7" containerID="8fb84c42cd3afb7dfaca1eab1468ff6df01d5c5707db9eda2bf176694a5cb98d" exitCode=0
Sep 19 04:34:28.628224 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:28.627865    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e8aaa06f-pzrrk" event={"ID":"1de0ac98-7c69-4e78-9fec-663ea21715f7","Type":"ContainerDied","Data":"8fb84c42cd3afb7dfaca1eab1468ff6df01d5c5707db9eda2bf176694a5cb98d"}
Sep 19 04:34:29.773925 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:29.773901    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e8aaa06f-pzrrk"
Sep 19 04:34:29.935343 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:29.935311    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1de0ac98-7c69-4e78-9fec-663ea21715f7-maas-api-service-ca\") pod \"1de0ac98-7c69-4e78-9fec-663ea21715f7\" (UID: \"1de0ac98-7c69-4e78-9fec-663ea21715f7\") "
Sep 19 04:34:29.935674 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:29.935649    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1de0ac98-7c69-4e78-9fec-663ea21715f7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1de0ac98-7c69-4e78-9fec-663ea21715f7" (UID: "1de0ac98-7c69-4e78-9fec-663ea21715f7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 04:34:30.036644 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:30.036610    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1de0ac98-7c69-4e78-9fec-663ea21715f7-maas-api-service-ca\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:34:30.642461 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:30.642431    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e8aaa06f-pzrrk"
Sep 19 04:34:30.642625 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:30.642462    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e8aaa06f-pzrrk" event={"ID":"1de0ac98-7c69-4e78-9fec-663ea21715f7","Type":"ContainerDied","Data":"8a27be5347994cc54454058ae59800a091a8e29483f3b57a5b22d25f0ff64ea7"}
Sep 19 04:34:30.642625 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:30.642515    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a27be5347994cc54454058ae59800a091a8e29483f3b57a5b22d25f0ff64ea7"
Sep 19 04:34:33.198098 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:33.198006    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6b1935-7b7d4bc579-l6sgg"]
Sep 19 04:34:33.198650 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:33.198629    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1de0ac98-7c69-4e78-9fec-663ea21715f7" containerName="revoke-keys"
Sep 19 04:34:33.198746 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:33.198654    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1de0ac98-7c69-4e78-9fec-663ea21715f7" containerName="revoke-keys"
Sep 19 04:34:33.198810 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:33.198786    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1de0ac98-7c69-4e78-9fec-663ea21715f7" containerName="revoke-keys"
Sep 19 04:34:33.201811 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:33.201787    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-6b1935-7b7d4bc579-l6sgg"
Sep 19 04:34:33.207284 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:33.207263    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-6b1935\""
Sep 19 04:34:33.207459 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:33.207290    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-6b1935-dockercfg-6mst5\""
Sep 19 04:34:33.216566 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:33.216519    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6b1935-7b7d4bc579-l6sgg"]
Sep 19 04:34:33.369558 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:33.369528    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec508902-bbc1-4675-85c4-535eb8ba6d99-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-6b1935-7b7d4bc579-l6sgg\" (UID: \"ec508902-bbc1-4675-85c4-535eb8ba6d99\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6b1935-7b7d4bc579-l6sgg"
Sep 19 04:34:33.369847 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:33.369595    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h58v7\" (UniqueName: \"kubernetes.io/projected/ec508902-bbc1-4675-85c4-535eb8ba6d99-kube-api-access-h58v7\") pod \"payload-processing-e2e-inf-b-6b1935-7b7d4bc579-l6sgg\" (UID: \"ec508902-bbc1-4675-85c4-535eb8ba6d99\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6b1935-7b7d4bc579-l6sgg"
Sep 19 04:34:33.471297 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:33.471189    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h58v7\" (UniqueName: \"kubernetes.io/projected/ec508902-bbc1-4675-85c4-535eb8ba6d99-kube-api-access-h58v7\") pod \"payload-processing-e2e-inf-b-6b1935-7b7d4bc579-l6sgg\" (UID: \"ec508902-bbc1-4675-85c4-535eb8ba6d99\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6b1935-7b7d4bc579-l6sgg"
Sep 19 04:34:33.471487 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:33.471349    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec508902-bbc1-4675-85c4-535eb8ba6d99-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-6b1935-7b7d4bc579-l6sgg\" (UID: \"ec508902-bbc1-4675-85c4-535eb8ba6d99\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6b1935-7b7d4bc579-l6sgg"
Sep 19 04:34:33.472147 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:33.472122    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec508902-bbc1-4675-85c4-535eb8ba6d99-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-6b1935-7b7d4bc579-l6sgg\" (UID: \"ec508902-bbc1-4675-85c4-535eb8ba6d99\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6b1935-7b7d4bc579-l6sgg"
Sep 19 04:34:33.480447 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:33.480415    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h58v7\" (UniqueName: \"kubernetes.io/projected/ec508902-bbc1-4675-85c4-535eb8ba6d99-kube-api-access-h58v7\") pod \"payload-processing-e2e-inf-b-6b1935-7b7d4bc579-l6sgg\" (UID: \"ec508902-bbc1-4675-85c4-535eb8ba6d99\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6b1935-7b7d4bc579-l6sgg"
Sep 19 04:34:33.513554 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:33.513511    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-6b1935-7b7d4bc579-l6sgg"
Sep 19 04:34:33.811311 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:34:33.811273    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec508902_bbc1_4675_85c4_535eb8ba6d99.slice/crio-16108035eaa11f242a59192071fffc2951f199b6784e883cf4ccd6eca96c8d77 WatchSource:0}: Error finding container 16108035eaa11f242a59192071fffc2951f199b6784e883cf4ccd6eca96c8d77: Status 404 returned error can't find the container with id 16108035eaa11f242a59192071fffc2951f199b6784e883cf4ccd6eca96c8d77
Sep 19 04:34:33.813929 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:33.813905    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6b1935-7b7d4bc579-l6sgg"]
Sep 19 04:34:34.667229 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:34.667188    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-6b1935-7b7d4bc579-l6sgg" event={"ID":"ec508902-bbc1-4675-85c4-535eb8ba6d99","Type":"ContainerStarted","Data":"86e5a34bdf13674ee28fefc4183ec01bc9018bf06c77ec306fd688c1174468fb"}
Sep 19 04:34:34.667651 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:34.667253    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-6b1935-7b7d4bc579-l6sgg" event={"ID":"ec508902-bbc1-4675-85c4-535eb8ba6d99","Type":"ContainerStarted","Data":"16108035eaa11f242a59192071fffc2951f199b6784e883cf4ccd6eca96c8d77"}
Sep 19 04:34:34.667651 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:34.667405    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-6b1935-7b7d4bc579-l6sgg"
Sep 19 04:34:34.695410 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:34.695360    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-6b1935-7b7d4bc579-l6sgg" podStartSLOduration=1.695344035 podStartE2EDuration="1.695344035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 04:34:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:34:34.69513442 +0000 UTC m=+1253.878095579" watchObservedRunningTime="2026-09-19 04:34:34.695344035 +0000 UTC m=+1253.878305184"
Sep 19 04:34:39.455795 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:39.455755    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e8aaa06f-b598d6fdc-hqplp"]
Sep 19 04:34:39.456287 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:39.456009    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-e8aaa06f-b598d6fdc-hqplp" podUID="e4173388-3e28-4437-9044-491413e66ace" containerName="payload-processing" containerID="cri-o://f3a9f170c7756e569479208b1ab6eb4fab6cda8abfe4da9c4b7187caaac32ae1" gracePeriod=30
Sep 19 04:34:39.655126 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:39.655039    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e8aaa06f-pzrrk"]
Sep 19 04:34:39.657789 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:39.657762    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e8aaa06f-pzrrk"]
Sep 19 04:34:39.703443 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:39.703406    2579 generic.go:356] "Generic (PLEG): container finished" podID="e4173388-3e28-4437-9044-491413e66ace" containerID="f3a9f170c7756e569479208b1ab6eb4fab6cda8abfe4da9c4b7187caaac32ae1" exitCode=0
Sep 19 04:34:39.703673 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:39.703656    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e8aaa06f-b598d6fdc-hqplp" event={"ID":"e4173388-3e28-4437-9044-491413e66ace","Type":"ContainerDied","Data":"f3a9f170c7756e569479208b1ab6eb4fab6cda8abfe4da9c4b7187caaac32ae1"}
Sep 19 04:34:39.746570 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:39.746546    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e8aaa06f-b598d6fdc-hqplp"
Sep 19 04:34:39.937155 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:39.937124    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4173388-3e28-4437-9044-491413e66ace-plugins-config-volume\") pod \"e4173388-3e28-4437-9044-491413e66ace\" (UID: \"e4173388-3e28-4437-9044-491413e66ace\") "
Sep 19 04:34:39.937155 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:39.937170    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hm7xk\" (UniqueName: \"kubernetes.io/projected/e4173388-3e28-4437-9044-491413e66ace-kube-api-access-hm7xk\") pod \"e4173388-3e28-4437-9044-491413e66ace\" (UID: \"e4173388-3e28-4437-9044-491413e66ace\") "
Sep 19 04:34:39.937551 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:39.937530    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4173388-3e28-4437-9044-491413e66ace-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e4173388-3e28-4437-9044-491413e66ace" (UID: "e4173388-3e28-4437-9044-491413e66ace"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 04:34:39.939726 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:39.939670    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4173388-3e28-4437-9044-491413e66ace-kube-api-access-hm7xk" (OuterVolumeSpecName: "kube-api-access-hm7xk") pod "e4173388-3e28-4437-9044-491413e66ace" (UID: "e4173388-3e28-4437-9044-491413e66ace"). InnerVolumeSpecName "kube-api-access-hm7xk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 04:34:40.038177 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:40.038095    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4173388-3e28-4437-9044-491413e66ace-plugins-config-volume\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:34:40.038177 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:40.038124    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hm7xk\" (UniqueName: \"kubernetes.io/projected/e4173388-3e28-4437-9044-491413e66ace-kube-api-access-hm7xk\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:34:40.710507 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:40.710466    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e8aaa06f-b598d6fdc-hqplp" event={"ID":"e4173388-3e28-4437-9044-491413e66ace","Type":"ContainerDied","Data":"24d6cfaae0a2fb2f356a4aeaf8fb6e8f8d75164d89e05fc834c41d2cb8261090"}
Sep 19 04:34:40.710507 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:40.710482    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e8aaa06f-b598d6fdc-hqplp"
Sep 19 04:34:40.711080 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:40.710526    2579 scope.go:117] "RemoveContainer" containerID="f3a9f170c7756e569479208b1ab6eb4fab6cda8abfe4da9c4b7187caaac32ae1"
Sep 19 04:34:40.741580 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:40.741550    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e8aaa06f-b598d6fdc-hqplp"]
Sep 19 04:34:40.748551 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:40.748525    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e8aaa06f-b598d6fdc-hqplp"]
Sep 19 04:34:41.351794 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:41.351766    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1de0ac98-7c69-4e78-9fec-663ea21715f7" path="/var/lib/kubelet/pods/1de0ac98-7c69-4e78-9fec-663ea21715f7/volumes"
Sep 19 04:34:41.352124 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:41.352112    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4173388-3e28-4437-9044-491413e66ace" path="/var/lib/kubelet/pods/e4173388-3e28-4437-9044-491413e66ace/volumes"
Sep 19 04:34:45.675521 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:45.675477    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-6b1935-7b7d4bc579-l6sgg"
Sep 19 04:34:47.812719 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:47.811349    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-1b8ef317-7cfcb5f846-glpws"]
Sep 19 04:34:47.812719 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:47.812517    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4173388-3e28-4437-9044-491413e66ace" containerName="payload-processing"
Sep 19 04:34:47.812719 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:47.812542    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4173388-3e28-4437-9044-491413e66ace" containerName="payload-processing"
Sep 19 04:34:47.813276 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:47.812736    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4173388-3e28-4437-9044-491413e66ace" containerName="payload-processing"
Sep 19 04:34:47.821061 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:47.821032    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-1b8ef317-7cfcb5f846-glpws"
Sep 19 04:34:47.824351 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:47.824325    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-1b8ef317\""
Sep 19 04:34:47.824507 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:47.824325    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-1b8ef317-dockercfg-c6776\""
Sep 19 04:34:47.825957 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:47.825932    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-1b8ef317-7cfcb5f846-glpws"]
Sep 19 04:34:47.919263 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:47.918476    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzlrt\" (UniqueName: \"kubernetes.io/projected/c2dffc0d-3824-4e6a-8c0d-fe001f05566e-kube-api-access-vzlrt\") pod \"payload-processing-e2e-migrate-1b8ef317-7cfcb5f846-glpws\" (UID: \"c2dffc0d-3824-4e6a-8c0d-fe001f05566e\") " pod="openshift-ingress/payload-processing-e2e-migrate-1b8ef317-7cfcb5f846-glpws"
Sep 19 04:34:47.919263 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:47.918618    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2dffc0d-3824-4e6a-8c0d-fe001f05566e-plugins-config-volume\") pod \"payload-processing-e2e-migrate-1b8ef317-7cfcb5f846-glpws\" (UID: \"c2dffc0d-3824-4e6a-8c0d-fe001f05566e\") " pod="openshift-ingress/payload-processing-e2e-migrate-1b8ef317-7cfcb5f846-glpws"
Sep 19 04:34:48.019766 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:48.019728    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2dffc0d-3824-4e6a-8c0d-fe001f05566e-plugins-config-volume\") pod \"payload-processing-e2e-migrate-1b8ef317-7cfcb5f846-glpws\" (UID: \"c2dffc0d-3824-4e6a-8c0d-fe001f05566e\") " pod="openshift-ingress/payload-processing-e2e-migrate-1b8ef317-7cfcb5f846-glpws"
Sep 19 04:34:48.019959 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:48.019819    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzlrt\" (UniqueName: \"kubernetes.io/projected/c2dffc0d-3824-4e6a-8c0d-fe001f05566e-kube-api-access-vzlrt\") pod \"payload-processing-e2e-migrate-1b8ef317-7cfcb5f846-glpws\" (UID: \"c2dffc0d-3824-4e6a-8c0d-fe001f05566e\") " pod="openshift-ingress/payload-processing-e2e-migrate-1b8ef317-7cfcb5f846-glpws"
Sep 19 04:34:48.020436 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:48.020411    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2dffc0d-3824-4e6a-8c0d-fe001f05566e-plugins-config-volume\") pod \"payload-processing-e2e-migrate-1b8ef317-7cfcb5f846-glpws\" (UID: \"c2dffc0d-3824-4e6a-8c0d-fe001f05566e\") " pod="openshift-ingress/payload-processing-e2e-migrate-1b8ef317-7cfcb5f846-glpws"
Sep 19 04:34:48.029461 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:48.029437    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzlrt\" (UniqueName: \"kubernetes.io/projected/c2dffc0d-3824-4e6a-8c0d-fe001f05566e-kube-api-access-vzlrt\") pod \"payload-processing-e2e-migrate-1b8ef317-7cfcb5f846-glpws\" (UID: \"c2dffc0d-3824-4e6a-8c0d-fe001f05566e\") " pod="openshift-ingress/payload-processing-e2e-migrate-1b8ef317-7cfcb5f846-glpws"
Sep 19 04:34:48.133911 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:48.133806    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-1b8ef317-7cfcb5f846-glpws"
Sep 19 04:34:48.416917 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:48.416889    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-1b8ef317-7cfcb5f846-glpws"]
Sep 19 04:34:48.418370 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:34:48.418343    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2dffc0d_3824_4e6a_8c0d_fe001f05566e.slice/crio-8eae88fe22ce1b914e01c4f011a979e8894e00e3cef9af578109852c851660f5 WatchSource:0}: Error finding container 8eae88fe22ce1b914e01c4f011a979e8894e00e3cef9af578109852c851660f5: Status 404 returned error can't find the container with id 8eae88fe22ce1b914e01c4f011a979e8894e00e3cef9af578109852c851660f5
Sep 19 04:34:48.763197 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:48.763094    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-1b8ef317-7cfcb5f846-glpws" event={"ID":"c2dffc0d-3824-4e6a-8c0d-fe001f05566e","Type":"ContainerStarted","Data":"da9ea87684227ecc35ad941f1b9238ae5ef851df185c6f2aa7fe5af2eefdcd80"}
Sep 19 04:34:48.763197 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:48.763154    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-1b8ef317-7cfcb5f846-glpws" event={"ID":"c2dffc0d-3824-4e6a-8c0d-fe001f05566e","Type":"ContainerStarted","Data":"8eae88fe22ce1b914e01c4f011a979e8894e00e3cef9af578109852c851660f5"}
Sep 19 04:34:48.763197 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:48.763180    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-1b8ef317-7cfcb5f846-glpws"
Sep 19 04:34:48.781801 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:48.781743    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-1b8ef317-7cfcb5f846-glpws" podStartSLOduration=1.781726769 podStartE2EDuration="1.781726769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 04:34:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:34:48.780675303 +0000 UTC m=+1267.963636452" watchObservedRunningTime="2026-09-19 04:34:48.781726769 +0000 UTC m=+1267.964687911"
Sep 19 04:34:59.771447 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:34:59.771419    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-1b8ef317-7cfcb5f846-glpws"
Sep 19 04:35:13.621255 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:13.621224    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1b8ef317-n2kdh"]
Sep 19 04:35:13.624753 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:13.624733    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1b8ef317-n2kdh"
Sep 19 04:35:13.627546 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:13.627520    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s6xbq\""
Sep 19 04:35:13.628139 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:13.628119    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 19 04:35:13.653718 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:13.650659    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1b8ef317-n2kdh"]
Sep 19 04:35:13.782925 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:13.782887    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fca33bc8-f0aa-4df9-983e-814371ad3e71-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1b8ef317-n2kdh\" (UID: \"fca33bc8-f0aa-4df9-983e-814371ad3e71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1b8ef317-n2kdh"
Sep 19 04:35:13.884062 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:13.883954    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fca33bc8-f0aa-4df9-983e-814371ad3e71-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1b8ef317-n2kdh\" (UID: \"fca33bc8-f0aa-4df9-983e-814371ad3e71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1b8ef317-n2kdh"
Sep 19 04:35:13.884755 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:13.884730    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fca33bc8-f0aa-4df9-983e-814371ad3e71-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1b8ef317-n2kdh\" (UID: \"fca33bc8-f0aa-4df9-983e-814371ad3e71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1b8ef317-n2kdh"
Sep 19 04:35:13.934587 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:13.934544    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1b8ef317-n2kdh"
Sep 19 04:35:14.209207 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:14.209179    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1b8ef317-n2kdh"]
Sep 19 04:35:14.211337 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:35:14.211278    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfca33bc8_f0aa_4df9_983e_814371ad3e71.slice/crio-1265372ad911e06c0d05a285f856e3ad1f3aabcfef95998692952af26544dc73 WatchSource:0}: Error finding container 1265372ad911e06c0d05a285f856e3ad1f3aabcfef95998692952af26544dc73: Status 404 returned error can't find the container with id 1265372ad911e06c0d05a285f856e3ad1f3aabcfef95998692952af26544dc73
Sep 19 04:35:14.933000 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:14.932964    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1b8ef317-n2kdh" event={"ID":"fca33bc8-f0aa-4df9-983e-814371ad3e71","Type":"ContainerStarted","Data":"16d4d7f362a14ee5a553a5dab2ca5c86a66df5738414a04a8bba3883af04d53a"}
Sep 19 04:35:14.933399 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:14.933013    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1b8ef317-n2kdh" event={"ID":"fca33bc8-f0aa-4df9-983e-814371ad3e71","Type":"ContainerStarted","Data":"1265372ad911e06c0d05a285f856e3ad1f3aabcfef95998692952af26544dc73"}
Sep 19 04:35:14.952890 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:14.952841    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1b8ef317-n2kdh" podStartSLOduration=1.9528257629999999 podStartE2EDuration="1.952825763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 04:35:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:35:14.949895482 +0000 UTC m=+1294.132856639" watchObservedRunningTime="2026-09-19 04:35:14.952825763 +0000 UTC m=+1294.135786911"
Sep 19 04:35:15.940554 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:15.940523    2579 generic.go:356] "Generic (PLEG): container finished" podID="fca33bc8-f0aa-4df9-983e-814371ad3e71" containerID="16d4d7f362a14ee5a553a5dab2ca5c86a66df5738414a04a8bba3883af04d53a" exitCode=0
Sep 19 04:35:15.941024 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:15.940616    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1b8ef317-n2kdh" event={"ID":"fca33bc8-f0aa-4df9-983e-814371ad3e71","Type":"ContainerDied","Data":"16d4d7f362a14ee5a553a5dab2ca5c86a66df5738414a04a8bba3883af04d53a"}
Sep 19 04:35:17.085941 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:17.085912    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1b8ef317-n2kdh"
Sep 19 04:35:17.211141 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:17.211056    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fca33bc8-f0aa-4df9-983e-814371ad3e71-maas-api-service-ca\") pod \"fca33bc8-f0aa-4df9-983e-814371ad3e71\" (UID: \"fca33bc8-f0aa-4df9-983e-814371ad3e71\") "
Sep 19 04:35:17.211382 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:17.211360    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fca33bc8-f0aa-4df9-983e-814371ad3e71-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fca33bc8-f0aa-4df9-983e-814371ad3e71" (UID: "fca33bc8-f0aa-4df9-983e-814371ad3e71"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 04:35:17.312108 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:17.312075    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fca33bc8-f0aa-4df9-983e-814371ad3e71-maas-api-service-ca\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:35:17.953518 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:17.953482    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1b8ef317-n2kdh" event={"ID":"fca33bc8-f0aa-4df9-983e-814371ad3e71","Type":"ContainerDied","Data":"1265372ad911e06c0d05a285f856e3ad1f3aabcfef95998692952af26544dc73"}
Sep 19 04:35:17.953719 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:17.953530    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1265372ad911e06c0d05a285f856e3ad1f3aabcfef95998692952af26544dc73"
Sep 19 04:35:17.953719 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:17.953494    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1b8ef317-n2kdh"
Sep 19 04:35:25.833434 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:25.833398    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-1b8ef317-7cfcb5f846-glpws"]
Sep 19 04:35:25.834071 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:25.834004    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-1b8ef317-7cfcb5f846-glpws" podUID="c2dffc0d-3824-4e6a-8c0d-fe001f05566e" containerName="payload-processing" containerID="cri-o://da9ea87684227ecc35ad941f1b9238ae5ef851df185c6f2aa7fe5af2eefdcd80" gracePeriod=30
Sep 19 04:35:26.002500 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:26.002469    2579 generic.go:356] "Generic (PLEG): container finished" podID="c2dffc0d-3824-4e6a-8c0d-fe001f05566e" containerID="da9ea87684227ecc35ad941f1b9238ae5ef851df185c6f2aa7fe5af2eefdcd80" exitCode=0
Sep 19 04:35:26.002732 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:26.002547    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-1b8ef317-7cfcb5f846-glpws" event={"ID":"c2dffc0d-3824-4e6a-8c0d-fe001f05566e","Type":"ContainerDied","Data":"da9ea87684227ecc35ad941f1b9238ae5ef851df185c6f2aa7fe5af2eefdcd80"}
Sep 19 04:35:26.017663 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:26.017632    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1b8ef317-n2kdh"]
Sep 19 04:35:26.021826 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:26.021797    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1b8ef317-n2kdh"]
Sep 19 04:35:26.109244 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:26.109126    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-1b8ef317-7cfcb5f846-glpws"
Sep 19 04:35:26.197057 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:26.197020    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vzlrt\" (UniqueName: \"kubernetes.io/projected/c2dffc0d-3824-4e6a-8c0d-fe001f05566e-kube-api-access-vzlrt\") pod \"c2dffc0d-3824-4e6a-8c0d-fe001f05566e\" (UID: \"c2dffc0d-3824-4e6a-8c0d-fe001f05566e\") "
Sep 19 04:35:26.197264 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:26.197132    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2dffc0d-3824-4e6a-8c0d-fe001f05566e-plugins-config-volume\") pod \"c2dffc0d-3824-4e6a-8c0d-fe001f05566e\" (UID: \"c2dffc0d-3824-4e6a-8c0d-fe001f05566e\") "
Sep 19 04:35:26.197422 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:26.197398    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2dffc0d-3824-4e6a-8c0d-fe001f05566e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c2dffc0d-3824-4e6a-8c0d-fe001f05566e" (UID: "c2dffc0d-3824-4e6a-8c0d-fe001f05566e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 04:35:26.199305 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:26.199279    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2dffc0d-3824-4e6a-8c0d-fe001f05566e-kube-api-access-vzlrt" (OuterVolumeSpecName: "kube-api-access-vzlrt") pod "c2dffc0d-3824-4e6a-8c0d-fe001f05566e" (UID: "c2dffc0d-3824-4e6a-8c0d-fe001f05566e"). InnerVolumeSpecName "kube-api-access-vzlrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 04:35:26.298155 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:26.298117    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzlrt\" (UniqueName: \"kubernetes.io/projected/c2dffc0d-3824-4e6a-8c0d-fe001f05566e-kube-api-access-vzlrt\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:35:26.298155 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:26.298152    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2dffc0d-3824-4e6a-8c0d-fe001f05566e-plugins-config-volume\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:35:27.010248 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:27.010213    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-1b8ef317-7cfcb5f846-glpws"
Sep 19 04:35:27.010248 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:27.010237    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-1b8ef317-7cfcb5f846-glpws" event={"ID":"c2dffc0d-3824-4e6a-8c0d-fe001f05566e","Type":"ContainerDied","Data":"8eae88fe22ce1b914e01c4f011a979e8894e00e3cef9af578109852c851660f5"}
Sep 19 04:35:27.010749 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:27.010299    2579 scope.go:117] "RemoveContainer" containerID="da9ea87684227ecc35ad941f1b9238ae5ef851df185c6f2aa7fe5af2eefdcd80"
Sep 19 04:35:27.039045 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:27.039006    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-1b8ef317-7cfcb5f846-glpws"]
Sep 19 04:35:27.042619 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:27.042588    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-1b8ef317-7cfcb5f846-glpws"]
Sep 19 04:35:27.352097 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:27.352006    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2dffc0d-3824-4e6a-8c0d-fe001f05566e" path="/var/lib/kubelet/pods/c2dffc0d-3824-4e6a-8c0d-fe001f05566e/volumes"
Sep 19 04:35:27.352475 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:27.352454    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fca33bc8-f0aa-4df9-983e-814371ad3e71" path="/var/lib/kubelet/pods/fca33bc8-f0aa-4df9-983e-814371ad3e71/volumes"
Sep 19 04:35:46.847038 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:46.847000    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-d0f416/simulator-w4-d0f416-kserve-7c6665bd67-kqvrt"]
Sep 19 04:35:46.847521 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:46.847506    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fca33bc8-f0aa-4df9-983e-814371ad3e71" containerName="revoke-keys"
Sep 19 04:35:46.847569 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:46.847524    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="fca33bc8-f0aa-4df9-983e-814371ad3e71" containerName="revoke-keys"
Sep 19 04:35:46.847569 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:46.847549    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2dffc0d-3824-4e6a-8c0d-fe001f05566e" containerName="payload-processing"
Sep 19 04:35:46.847569 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:46.847555    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2dffc0d-3824-4e6a-8c0d-fe001f05566e" containerName="payload-processing"
Sep 19 04:35:46.847661 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:46.847611    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="fca33bc8-f0aa-4df9-983e-814371ad3e71" containerName="revoke-keys"
Sep 19 04:35:46.847661 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:46.847620    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2dffc0d-3824-4e6a-8c0d-fe001f05566e" containerName="payload-processing"
Sep 19 04:35:46.851935 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:46.851921    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d0f416/simulator-w4-d0f416-kserve-7c6665bd67-kqvrt"
Sep 19 04:35:46.856074 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:46.856054    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-d0f416\"/\"default-dockercfg-tg7hm\""
Sep 19 04:35:46.856074 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:46.856062    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-d0f416\"/\"kube-root-ca.crt\""
Sep 19 04:35:46.856241 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:46.856054    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-d0f416\"/\"simulator-w4-d0f416-kserve-self-signed-certs\""
Sep 19 04:35:46.856241 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:46.856111    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-d0f416\"/\"openshift-service-ca.crt\""
Sep 19 04:35:46.862769 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:46.862746    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-d0f416/simulator-w4-d0f416-kserve-7c6665bd67-kqvrt"]
Sep 19 04:35:47.008214 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:47.008180    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ca22b40-223c-472d-879a-0194160787f0-dshm\") pod \"simulator-w4-d0f416-kserve-7c6665bd67-kqvrt\" (UID: \"3ca22b40-223c-472d-879a-0194160787f0\") " pod="e2e-models-e2e-worker-w4-d0f416/simulator-w4-d0f416-kserve-7c6665bd67-kqvrt"
Sep 19 04:35:47.008214 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:47.008215    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ca22b40-223c-472d-879a-0194160787f0-model-cache\") pod \"simulator-w4-d0f416-kserve-7c6665bd67-kqvrt\" (UID: \"3ca22b40-223c-472d-879a-0194160787f0\") " pod="e2e-models-e2e-worker-w4-d0f416/simulator-w4-d0f416-kserve-7c6665bd67-kqvrt"
Sep 19 04:35:47.008433 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:47.008247    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ca22b40-223c-472d-879a-0194160787f0-tls-certs\") pod \"simulator-w4-d0f416-kserve-7c6665bd67-kqvrt\" (UID: \"3ca22b40-223c-472d-879a-0194160787f0\") " pod="e2e-models-e2e-worker-w4-d0f416/simulator-w4-d0f416-kserve-7c6665bd67-kqvrt"
Sep 19 04:35:47.008433 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:47.008367    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ca22b40-223c-472d-879a-0194160787f0-tmp-dir\") pod \"simulator-w4-d0f416-kserve-7c6665bd67-kqvrt\" (UID: \"3ca22b40-223c-472d-879a-0194160787f0\") " pod="e2e-models-e2e-worker-w4-d0f416/simulator-w4-d0f416-kserve-7c6665bd67-kqvrt"
Sep 19 04:35:47.008507 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:47.008427    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntk78\" (UniqueName: \"kubernetes.io/projected/3ca22b40-223c-472d-879a-0194160787f0-kube-api-access-ntk78\") pod \"simulator-w4-d0f416-kserve-7c6665bd67-kqvrt\" (UID: \"3ca22b40-223c-472d-879a-0194160787f0\") " pod="e2e-models-e2e-worker-w4-d0f416/simulator-w4-d0f416-kserve-7c6665bd67-kqvrt"
Sep 19 04:35:47.008507 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:47.008470    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ca22b40-223c-472d-879a-0194160787f0-home\") pod \"simulator-w4-d0f416-kserve-7c6665bd67-kqvrt\" (UID: \"3ca22b40-223c-472d-879a-0194160787f0\") " pod="e2e-models-e2e-worker-w4-d0f416/simulator-w4-d0f416-kserve-7c6665bd67-kqvrt"
Sep 19 04:35:47.109792 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:47.109672    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntk78\" (UniqueName: \"kubernetes.io/projected/3ca22b40-223c-472d-879a-0194160787f0-kube-api-access-ntk78\") pod \"simulator-w4-d0f416-kserve-7c6665bd67-kqvrt\" (UID: \"3ca22b40-223c-472d-879a-0194160787f0\") " pod="e2e-models-e2e-worker-w4-d0f416/simulator-w4-d0f416-kserve-7c6665bd67-kqvrt"
Sep 19 04:35:47.110058 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:47.110036    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ca22b40-223c-472d-879a-0194160787f0-home\") pod \"simulator-w4-d0f416-kserve-7c6665bd67-kqvrt\" (UID: \"3ca22b40-223c-472d-879a-0194160787f0\") " pod="e2e-models-e2e-worker-w4-d0f416/simulator-w4-d0f416-kserve-7c6665bd67-kqvrt"
Sep 19 04:35:47.110256 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:47.110230    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ca22b40-223c-472d-879a-0194160787f0-dshm\") pod \"simulator-w4-d0f416-kserve-7c6665bd67-kqvrt\" (UID: \"3ca22b40-223c-472d-879a-0194160787f0\") " pod="e2e-models-e2e-worker-w4-d0f416/simulator-w4-d0f416-kserve-7c6665bd67-kqvrt"
Sep 19 04:35:47.110392 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:47.110275    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ca22b40-223c-472d-879a-0194160787f0-model-cache\") pod \"simulator-w4-d0f416-kserve-7c6665bd67-kqvrt\" (UID: \"3ca22b40-223c-472d-879a-0194160787f0\") " pod="e2e-models-e2e-worker-w4-d0f416/simulator-w4-d0f416-kserve-7c6665bd67-kqvrt"
Sep 19 04:35:47.110392 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:47.110324    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ca22b40-223c-472d-879a-0194160787f0-tls-certs\") pod \"simulator-w4-d0f416-kserve-7c6665bd67-kqvrt\" (UID: \"3ca22b40-223c-472d-879a-0194160787f0\") " pod="e2e-models-e2e-worker-w4-d0f416/simulator-w4-d0f416-kserve-7c6665bd67-kqvrt"
Sep 19 04:35:47.110574 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:47.110420    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ca22b40-223c-472d-879a-0194160787f0-tmp-dir\") pod \"simulator-w4-d0f416-kserve-7c6665bd67-kqvrt\" (UID: \"3ca22b40-223c-472d-879a-0194160787f0\") " pod="e2e-models-e2e-worker-w4-d0f416/simulator-w4-d0f416-kserve-7c6665bd67-kqvrt"
Sep 19 04:35:47.110791 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:47.110743    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ca22b40-223c-472d-879a-0194160787f0-model-cache\") pod \"simulator-w4-d0f416-kserve-7c6665bd67-kqvrt\" (UID: \"3ca22b40-223c-472d-879a-0194160787f0\") " pod="e2e-models-e2e-worker-w4-d0f416/simulator-w4-d0f416-kserve-7c6665bd67-kqvrt"
Sep 19 04:35:47.111016 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:47.110995    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ca22b40-223c-472d-879a-0194160787f0-home\") pod \"simulator-w4-d0f416-kserve-7c6665bd67-kqvrt\" (UID: \"3ca22b40-223c-472d-879a-0194160787f0\") " pod="e2e-models-e2e-worker-w4-d0f416/simulator-w4-d0f416-kserve-7c6665bd67-kqvrt"
Sep 19 04:35:47.111135 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:47.111013    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ca22b40-223c-472d-879a-0194160787f0-tmp-dir\") pod \"simulator-w4-d0f416-kserve-7c6665bd67-kqvrt\" (UID: \"3ca22b40-223c-472d-879a-0194160787f0\") " pod="e2e-models-e2e-worker-w4-d0f416/simulator-w4-d0f416-kserve-7c6665bd67-kqvrt"
Sep 19 04:35:47.114005 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:47.113977    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ca22b40-223c-472d-879a-0194160787f0-dshm\") pod \"simulator-w4-d0f416-kserve-7c6665bd67-kqvrt\" (UID: \"3ca22b40-223c-472d-879a-0194160787f0\") " pod="e2e-models-e2e-worker-w4-d0f416/simulator-w4-d0f416-kserve-7c6665bd67-kqvrt"
Sep 19 04:35:47.114633 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:47.114606    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ca22b40-223c-472d-879a-0194160787f0-tls-certs\") pod \"simulator-w4-d0f416-kserve-7c6665bd67-kqvrt\" (UID: \"3ca22b40-223c-472d-879a-0194160787f0\") " pod="e2e-models-e2e-worker-w4-d0f416/simulator-w4-d0f416-kserve-7c6665bd67-kqvrt"
Sep 19 04:35:47.119284 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:47.119259    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntk78\" (UniqueName: \"kubernetes.io/projected/3ca22b40-223c-472d-879a-0194160787f0-kube-api-access-ntk78\") pod \"simulator-w4-d0f416-kserve-7c6665bd67-kqvrt\" (UID: \"3ca22b40-223c-472d-879a-0194160787f0\") " pod="e2e-models-e2e-worker-w4-d0f416/simulator-w4-d0f416-kserve-7c6665bd67-kqvrt"
Sep 19 04:35:47.160766 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:47.160741    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d0f416/simulator-w4-d0f416-kserve-7c6665bd67-kqvrt"
Sep 19 04:35:47.447670 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:47.447628    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-d0f416/simulator-w4-d0f416-kserve-7c6665bd67-kqvrt"]
Sep 19 04:35:47.448846 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:35:47.448821    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3ca22b40_223c_472d_879a_0194160787f0.slice/crio-ba5b7a9afdc026543bf4a545fccf2218c2d556c76d4160c821afac6ebedde0e7 WatchSource:0}: Error finding container ba5b7a9afdc026543bf4a545fccf2218c2d556c76d4160c821afac6ebedde0e7: Status 404 returned error can't find the container with id ba5b7a9afdc026543bf4a545fccf2218c2d556c76d4160c821afac6ebedde0e7
Sep 19 04:35:48.133350 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:48.133311    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d0f416/simulator-w4-d0f416-kserve-7c6665bd67-kqvrt" event={"ID":"3ca22b40-223c-472d-879a-0194160787f0","Type":"ContainerStarted","Data":"32c8f8982b2a3c3f58a7051aedb84b6b991164f992d5f832b9dac038203d045c"}
Sep 19 04:35:48.133755 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:48.133360    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d0f416/simulator-w4-d0f416-kserve-7c6665bd67-kqvrt" event={"ID":"3ca22b40-223c-472d-879a-0194160787f0","Type":"ContainerStarted","Data":"ba5b7a9afdc026543bf4a545fccf2218c2d556c76d4160c821afac6ebedde0e7"}
Sep 19 04:35:48.154321 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:48.154272    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-d0f416/simulator-w4-d0f416-kserve-7c6665bd67-kqvrt" podStartSLOduration=2.15425552 podStartE2EDuration="2.15425552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 04:35:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:35:48.152886185 +0000 UTC m=+1327.335847336" watchObservedRunningTime="2026-09-19 04:35:48.15425552 +0000 UTC m=+1327.337216665"
Sep 19 04:35:48.161139 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:48.161104    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d0f416/simulator-w4-d0f416-kserve-7c6665bd67-kqvrt"
Sep 19 04:35:55.717958 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:55.717924    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33718462-xnd9l"]
Sep 19 04:35:55.721594 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:55.721575    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33718462-xnd9l"
Sep 19 04:35:55.727436 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:55.727415    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s6xbq\""
Sep 19 04:35:55.728844 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:55.728819    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 19 04:35:55.771329 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:55.771296    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33718462-xnd9l"]
Sep 19 04:35:55.788845 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:55.788817    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4aaf1bfd-0af9-46a7-a4a4-0e80e8bccebd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-33718462-xnd9l\" (UID: \"4aaf1bfd-0af9-46a7-a4a4-0e80e8bccebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33718462-xnd9l"
Sep 19 04:35:55.889462 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:55.889419    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4aaf1bfd-0af9-46a7-a4a4-0e80e8bccebd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-33718462-xnd9l\" (UID: \"4aaf1bfd-0af9-46a7-a4a4-0e80e8bccebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33718462-xnd9l"
Sep 19 04:35:55.890257 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:55.890232    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4aaf1bfd-0af9-46a7-a4a4-0e80e8bccebd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-33718462-xnd9l\" (UID: \"4aaf1bfd-0af9-46a7-a4a4-0e80e8bccebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33718462-xnd9l"
Sep 19 04:35:56.032586 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:56.032488    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33718462-xnd9l"
Sep 19 04:35:56.323687 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:56.323658    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33718462-xnd9l"]
Sep 19 04:35:56.325612 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:35:56.325586    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4aaf1bfd_0af9_46a7_a4a4_0e80e8bccebd.slice/crio-1379c74391e235edd05bde110d0ed8d15d873143d802fc7ee31aff6b44b12e27 WatchSource:0}: Error finding container 1379c74391e235edd05bde110d0ed8d15d873143d802fc7ee31aff6b44b12e27: Status 404 returned error can't find the container with id 1379c74391e235edd05bde110d0ed8d15d873143d802fc7ee31aff6b44b12e27
Sep 19 04:35:57.161770 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:57.161725    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-d0f416/simulator-w4-d0f416-kserve-7c6665bd67-kqvrt"
Sep 19 04:35:57.174425 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:57.174398    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-d0f416/simulator-w4-d0f416-kserve-7c6665bd67-kqvrt"
Sep 19 04:35:57.192734 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:57.192677    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33718462-xnd9l" event={"ID":"4aaf1bfd-0af9-46a7-a4a4-0e80e8bccebd","Type":"ContainerStarted","Data":"ef52a76e4019c044f63ea87f72fe83b713593d68619abcd7bdbc83f38af0facb"}
Sep 19 04:35:57.192909 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:57.192759    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33718462-xnd9l" event={"ID":"4aaf1bfd-0af9-46a7-a4a4-0e80e8bccebd","Type":"ContainerStarted","Data":"1379c74391e235edd05bde110d0ed8d15d873143d802fc7ee31aff6b44b12e27"}
Sep 19 04:35:57.207665 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:57.207641    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d0f416/simulator-w4-d0f416-kserve-7c6665bd67-kqvrt"
Sep 19 04:35:57.290153 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:57.290091    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33718462-xnd9l" podStartSLOduration=2.2900692879999998 podStartE2EDuration="2.290069288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 04:35:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:35:57.288000053 +0000 UTC m=+1336.470961202" watchObservedRunningTime="2026-09-19 04:35:57.290069288 +0000 UTC m=+1336.473030438"
Sep 19 04:35:58.201120 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:58.201088    2579 generic.go:356] "Generic (PLEG): container finished" podID="4aaf1bfd-0af9-46a7-a4a4-0e80e8bccebd" containerID="ef52a76e4019c044f63ea87f72fe83b713593d68619abcd7bdbc83f38af0facb" exitCode=0
Sep 19 04:35:58.201621 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:58.201162    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33718462-xnd9l" event={"ID":"4aaf1bfd-0af9-46a7-a4a4-0e80e8bccebd","Type":"ContainerDied","Data":"ef52a76e4019c044f63ea87f72fe83b713593d68619abcd7bdbc83f38af0facb"}
Sep 19 04:35:59.355420 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:59.355393    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33718462-xnd9l"
Sep 19 04:35:59.423379 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:59.423353    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4aaf1bfd-0af9-46a7-a4a4-0e80e8bccebd-maas-api-service-ca\") pod \"4aaf1bfd-0af9-46a7-a4a4-0e80e8bccebd\" (UID: \"4aaf1bfd-0af9-46a7-a4a4-0e80e8bccebd\") "
Sep 19 04:35:59.423763 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:59.423741    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4aaf1bfd-0af9-46a7-a4a4-0e80e8bccebd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4aaf1bfd-0af9-46a7-a4a4-0e80e8bccebd" (UID: "4aaf1bfd-0af9-46a7-a4a4-0e80e8bccebd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 04:35:59.524595 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:35:59.524507    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4aaf1bfd-0af9-46a7-a4a4-0e80e8bccebd-maas-api-service-ca\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:36:00.216327 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:00.216300    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33718462-xnd9l"
Sep 19 04:36:00.216502 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:00.216340    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33718462-xnd9l" event={"ID":"4aaf1bfd-0af9-46a7-a4a4-0e80e8bccebd","Type":"ContainerDied","Data":"1379c74391e235edd05bde110d0ed8d15d873143d802fc7ee31aff6b44b12e27"}
Sep 19 04:36:00.216502 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:00.216392    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1379c74391e235edd05bde110d0ed8d15d873143d802fc7ee31aff6b44b12e27"
Sep 19 04:36:02.808228 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:02.808145    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33718462-xnd9l"]
Sep 19 04:36:02.812123 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:02.812096    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33718462-xnd9l"]
Sep 19 04:36:03.352300 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:03.352257    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4aaf1bfd-0af9-46a7-a4a4-0e80e8bccebd" path="/var/lib/kubelet/pods/4aaf1bfd-0af9-46a7-a4a4-0e80e8bccebd/volumes"
Sep 19 04:36:05.564263 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:05.564230    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-a42681/autores-model-a42681-kserve-6887d75cd4-k4fws"]
Sep 19 04:36:05.564774 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:05.564757    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4aaf1bfd-0af9-46a7-a4a4-0e80e8bccebd" containerName="revoke-keys"
Sep 19 04:36:05.564774 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:05.564775    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4aaf1bfd-0af9-46a7-a4a4-0e80e8bccebd" containerName="revoke-keys"
Sep 19 04:36:05.564862 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:05.564853    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4aaf1bfd-0af9-46a7-a4a4-0e80e8bccebd" containerName="revoke-keys"
Sep 19 04:36:05.568142 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:05.568127    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-a42681/autores-model-a42681-kserve-6887d75cd4-k4fws"
Sep 19 04:36:05.572631 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:05.572608    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-a42681\"/\"autores-model-a42681-kserve-self-signed-certs\""
Sep 19 04:36:05.572782 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:05.572658    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-a42681\"/\"openshift-service-ca.crt\""
Sep 19 04:36:05.572782 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:05.572688    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-a42681\"/\"kube-root-ca.crt\""
Sep 19 04:36:05.573568 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:05.573550    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-a42681\"/\"default-dockercfg-5knpr\""
Sep 19 04:36:05.581404 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:05.581383    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-a42681/autores-model-a42681-kserve-6887d75cd4-k4fws"]
Sep 19 04:36:05.696120 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:05.696085    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0af35b4a-3ddf-4931-bdd7-890c22863222-tls-certs\") pod \"autores-model-a42681-kserve-6887d75cd4-k4fws\" (UID: \"0af35b4a-3ddf-4931-bdd7-890c22863222\") " pod="ai-tenant-e2e-autores-a42681/autores-model-a42681-kserve-6887d75cd4-k4fws"
Sep 19 04:36:05.696120 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:05.696121    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0af35b4a-3ddf-4931-bdd7-890c22863222-tmp-dir\") pod \"autores-model-a42681-kserve-6887d75cd4-k4fws\" (UID: \"0af35b4a-3ddf-4931-bdd7-890c22863222\") " pod="ai-tenant-e2e-autores-a42681/autores-model-a42681-kserve-6887d75cd4-k4fws"
Sep 19 04:36:05.696370 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:05.696278    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkqpd\" (UniqueName: \"kubernetes.io/projected/0af35b4a-3ddf-4931-bdd7-890c22863222-kube-api-access-kkqpd\") pod \"autores-model-a42681-kserve-6887d75cd4-k4fws\" (UID: \"0af35b4a-3ddf-4931-bdd7-890c22863222\") " pod="ai-tenant-e2e-autores-a42681/autores-model-a42681-kserve-6887d75cd4-k4fws"
Sep 19 04:36:05.696370 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:05.696309    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0af35b4a-3ddf-4931-bdd7-890c22863222-home\") pod \"autores-model-a42681-kserve-6887d75cd4-k4fws\" (UID: \"0af35b4a-3ddf-4931-bdd7-890c22863222\") " pod="ai-tenant-e2e-autores-a42681/autores-model-a42681-kserve-6887d75cd4-k4fws"
Sep 19 04:36:05.696370 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:05.696335    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0af35b4a-3ddf-4931-bdd7-890c22863222-model-cache\") pod \"autores-model-a42681-kserve-6887d75cd4-k4fws\" (UID: \"0af35b4a-3ddf-4931-bdd7-890c22863222\") " pod="ai-tenant-e2e-autores-a42681/autores-model-a42681-kserve-6887d75cd4-k4fws"
Sep 19 04:36:05.696370 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:05.696363    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0af35b4a-3ddf-4931-bdd7-890c22863222-dshm\") pod \"autores-model-a42681-kserve-6887d75cd4-k4fws\" (UID: \"0af35b4a-3ddf-4931-bdd7-890c22863222\") " pod="ai-tenant-e2e-autores-a42681/autores-model-a42681-kserve-6887d75cd4-k4fws"
Sep 19 04:36:05.797002 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:05.796966    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkqpd\" (UniqueName: \"kubernetes.io/projected/0af35b4a-3ddf-4931-bdd7-890c22863222-kube-api-access-kkqpd\") pod \"autores-model-a42681-kserve-6887d75cd4-k4fws\" (UID: \"0af35b4a-3ddf-4931-bdd7-890c22863222\") " pod="ai-tenant-e2e-autores-a42681/autores-model-a42681-kserve-6887d75cd4-k4fws"
Sep 19 04:36:05.797002 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:05.797002    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0af35b4a-3ddf-4931-bdd7-890c22863222-home\") pod \"autores-model-a42681-kserve-6887d75cd4-k4fws\" (UID: \"0af35b4a-3ddf-4931-bdd7-890c22863222\") " pod="ai-tenant-e2e-autores-a42681/autores-model-a42681-kserve-6887d75cd4-k4fws"
Sep 19 04:36:05.797274 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:05.797027    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0af35b4a-3ddf-4931-bdd7-890c22863222-model-cache\") pod \"autores-model-a42681-kserve-6887d75cd4-k4fws\" (UID: \"0af35b4a-3ddf-4931-bdd7-890c22863222\") " pod="ai-tenant-e2e-autores-a42681/autores-model-a42681-kserve-6887d75cd4-k4fws"
Sep 19 04:36:05.797274 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:05.797045    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0af35b4a-3ddf-4931-bdd7-890c22863222-dshm\") pod \"autores-model-a42681-kserve-6887d75cd4-k4fws\" (UID: \"0af35b4a-3ddf-4931-bdd7-890c22863222\") " pod="ai-tenant-e2e-autores-a42681/autores-model-a42681-kserve-6887d75cd4-k4fws"
Sep 19 04:36:05.797274 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:05.797076    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0af35b4a-3ddf-4931-bdd7-890c22863222-tls-certs\") pod \"autores-model-a42681-kserve-6887d75cd4-k4fws\" (UID: \"0af35b4a-3ddf-4931-bdd7-890c22863222\") " pod="ai-tenant-e2e-autores-a42681/autores-model-a42681-kserve-6887d75cd4-k4fws"
Sep 19 04:36:05.797274 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:05.797100    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0af35b4a-3ddf-4931-bdd7-890c22863222-tmp-dir\") pod \"autores-model-a42681-kserve-6887d75cd4-k4fws\" (UID: \"0af35b4a-3ddf-4931-bdd7-890c22863222\") " pod="ai-tenant-e2e-autores-a42681/autores-model-a42681-kserve-6887d75cd4-k4fws"
Sep 19 04:36:05.797493 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:05.797431    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0af35b4a-3ddf-4931-bdd7-890c22863222-home\") pod \"autores-model-a42681-kserve-6887d75cd4-k4fws\" (UID: \"0af35b4a-3ddf-4931-bdd7-890c22863222\") " pod="ai-tenant-e2e-autores-a42681/autores-model-a42681-kserve-6887d75cd4-k4fws"
Sep 19 04:36:05.797532 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:05.797484    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0af35b4a-3ddf-4931-bdd7-890c22863222-model-cache\") pod \"autores-model-a42681-kserve-6887d75cd4-k4fws\" (UID: \"0af35b4a-3ddf-4931-bdd7-890c22863222\") " pod="ai-tenant-e2e-autores-a42681/autores-model-a42681-kserve-6887d75cd4-k4fws"
Sep 19 04:36:05.797604 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:05.797588    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0af35b4a-3ddf-4931-bdd7-890c22863222-tmp-dir\") pod \"autores-model-a42681-kserve-6887d75cd4-k4fws\" (UID: \"0af35b4a-3ddf-4931-bdd7-890c22863222\") " pod="ai-tenant-e2e-autores-a42681/autores-model-a42681-kserve-6887d75cd4-k4fws"
Sep 19 04:36:05.799684 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:05.799655    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0af35b4a-3ddf-4931-bdd7-890c22863222-dshm\") pod \"autores-model-a42681-kserve-6887d75cd4-k4fws\" (UID: \"0af35b4a-3ddf-4931-bdd7-890c22863222\") " pod="ai-tenant-e2e-autores-a42681/autores-model-a42681-kserve-6887d75cd4-k4fws"
Sep 19 04:36:05.799921 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:05.799902    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0af35b4a-3ddf-4931-bdd7-890c22863222-tls-certs\") pod \"autores-model-a42681-kserve-6887d75cd4-k4fws\" (UID: \"0af35b4a-3ddf-4931-bdd7-890c22863222\") " pod="ai-tenant-e2e-autores-a42681/autores-model-a42681-kserve-6887d75cd4-k4fws"
Sep 19 04:36:05.809828 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:05.809801    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkqpd\" (UniqueName: \"kubernetes.io/projected/0af35b4a-3ddf-4931-bdd7-890c22863222-kube-api-access-kkqpd\") pod \"autores-model-a42681-kserve-6887d75cd4-k4fws\" (UID: \"0af35b4a-3ddf-4931-bdd7-890c22863222\") " pod="ai-tenant-e2e-autores-a42681/autores-model-a42681-kserve-6887d75cd4-k4fws"
Sep 19 04:36:05.876919 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:05.876840    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-a42681/autores-model-a42681-kserve-6887d75cd4-k4fws"
Sep 19 04:36:06.160258 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:06.160228    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-a42681/autores-model-a42681-kserve-6887d75cd4-k4fws"]
Sep 19 04:36:06.164154 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:36:06.164120    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0af35b4a_3ddf_4931_bdd7_890c22863222.slice/crio-65cc5754183827b58e810ab6caf217eb6aed9f19f439134be84e88079fe3851c WatchSource:0}: Error finding container 65cc5754183827b58e810ab6caf217eb6aed9f19f439134be84e88079fe3851c: Status 404 returned error can't find the container with id 65cc5754183827b58e810ab6caf217eb6aed9f19f439134be84e88079fe3851c
Sep 19 04:36:06.252522 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:06.252487    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-a42681/autores-model-a42681-kserve-6887d75cd4-k4fws" event={"ID":"0af35b4a-3ddf-4931-bdd7-890c22863222","Type":"ContainerStarted","Data":"2c6eccf8150f61deba47c2d91e9fb454c4d7ba46e1ca58444163e6d8897a8d3c"}
Sep 19 04:36:06.252668 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:06.252536    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-a42681/autores-model-a42681-kserve-6887d75cd4-k4fws" event={"ID":"0af35b4a-3ddf-4931-bdd7-890c22863222","Type":"ContainerStarted","Data":"65cc5754183827b58e810ab6caf217eb6aed9f19f439134be84e88079fe3851c"}
Sep 19 04:36:06.279540 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:06.279488    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-a42681/autores-model-a42681-kserve-6887d75cd4-k4fws" podStartSLOduration=1.279470748 podStartE2EDuration="1.279470748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 04:36:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:36:06.277649459 +0000 UTC m=+1345.460610606" watchObservedRunningTime="2026-09-19 04:36:06.279470748 +0000 UTC m=+1345.462431895"
Sep 19 04:36:06.877763 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:06.877725    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-a42681/autores-model-a42681-kserve-6887d75cd4-k4fws"
Sep 19 04:36:15.877208 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:15.877171    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-a42681/autores-model-a42681-kserve-6887d75cd4-k4fws"
Sep 19 04:36:15.892194 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:15.892171    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-a42681/autores-model-a42681-kserve-6887d75cd4-k4fws"
Sep 19 04:36:16.323501 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:16.323472    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-a42681/autores-model-a42681-kserve-6887d75cd4-k4fws"
Sep 19 04:36:32.863852 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:32.863819    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2a94cba7-42fqk"]
Sep 19 04:36:32.867512 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:32.867495    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2a94cba7-42fqk"
Sep 19 04:36:32.871370 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:32.871350    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s6xbq\""
Sep 19 04:36:32.871370 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:32.871360    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 19 04:36:32.879835 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:32.879807    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2a94cba7-42fqk"]
Sep 19 04:36:32.970493 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:32.970460    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd4e5831-d1b1-4050-b712-3f0a3155c948-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-2a94cba7-42fqk\" (UID: \"fd4e5831-d1b1-4050-b712-3f0a3155c948\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2a94cba7-42fqk"
Sep 19 04:36:33.071145 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:33.071110    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd4e5831-d1b1-4050-b712-3f0a3155c948-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-2a94cba7-42fqk\" (UID: \"fd4e5831-d1b1-4050-b712-3f0a3155c948\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2a94cba7-42fqk"
Sep 19 04:36:33.071815 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:33.071791    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd4e5831-d1b1-4050-b712-3f0a3155c948-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-2a94cba7-42fqk\" (UID: \"fd4e5831-d1b1-4050-b712-3f0a3155c948\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2a94cba7-42fqk"
Sep 19 04:36:33.178477 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:33.178445    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2a94cba7-42fqk"
Sep 19 04:36:33.458455 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:33.458426    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2a94cba7-42fqk"]
Sep 19 04:36:33.460207 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:36:33.460170    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd4e5831_d1b1_4050_b712_3f0a3155c948.slice/crio-26f58d7aaa421bedf31ea3fde136e4d82bef62729faac9bcdfb2ea9ae3eef4e0 WatchSource:0}: Error finding container 26f58d7aaa421bedf31ea3fde136e4d82bef62729faac9bcdfb2ea9ae3eef4e0: Status 404 returned error can't find the container with id 26f58d7aaa421bedf31ea3fde136e4d82bef62729faac9bcdfb2ea9ae3eef4e0
Sep 19 04:36:34.425155 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:34.425121    2579 generic.go:356] "Generic (PLEG): container finished" podID="fd4e5831-d1b1-4050-b712-3f0a3155c948" containerID="2183a12d05180f60cd0b279966f0e861750bebc9f5140d7cb691e4a22ff214e1" exitCode=0
Sep 19 04:36:34.425524 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:34.425193    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2a94cba7-42fqk" event={"ID":"fd4e5831-d1b1-4050-b712-3f0a3155c948","Type":"ContainerDied","Data":"2183a12d05180f60cd0b279966f0e861750bebc9f5140d7cb691e4a22ff214e1"}
Sep 19 04:36:34.425524 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:34.425246    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2a94cba7-42fqk" event={"ID":"fd4e5831-d1b1-4050-b712-3f0a3155c948","Type":"ContainerStarted","Data":"26f58d7aaa421bedf31ea3fde136e4d82bef62729faac9bcdfb2ea9ae3eef4e0"}
Sep 19 04:36:35.566615 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:35.566590    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2a94cba7-42fqk"
Sep 19 04:36:35.596896 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:35.596869    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd4e5831-d1b1-4050-b712-3f0a3155c948-maas-api-service-ca\") pod \"fd4e5831-d1b1-4050-b712-3f0a3155c948\" (UID: \"fd4e5831-d1b1-4050-b712-3f0a3155c948\") "
Sep 19 04:36:35.597318 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:35.597289    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd4e5831-d1b1-4050-b712-3f0a3155c948-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fd4e5831-d1b1-4050-b712-3f0a3155c948" (UID: "fd4e5831-d1b1-4050-b712-3f0a3155c948"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 04:36:35.698219 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:35.698120    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd4e5831-d1b1-4050-b712-3f0a3155c948-maas-api-service-ca\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:36:36.438311 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:36.438282    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2a94cba7-42fqk"
Sep 19 04:36:36.438481 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:36.438325    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2a94cba7-42fqk" event={"ID":"fd4e5831-d1b1-4050-b712-3f0a3155c948","Type":"ContainerDied","Data":"26f58d7aaa421bedf31ea3fde136e4d82bef62729faac9bcdfb2ea9ae3eef4e0"}
Sep 19 04:36:36.438481 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:36.438370    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26f58d7aaa421bedf31ea3fde136e4d82bef62729faac9bcdfb2ea9ae3eef4e0"
Sep 19 04:36:38.880213 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:38.880177    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a42681-7nkqg"]
Sep 19 04:36:38.880737 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:38.880647    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd4e5831-d1b1-4050-b712-3f0a3155c948" containerName="revoke-keys"
Sep 19 04:36:38.880737 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:38.880660    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd4e5831-d1b1-4050-b712-3f0a3155c948" containerName="revoke-keys"
Sep 19 04:36:38.880737 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:38.880734    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd4e5831-d1b1-4050-b712-3f0a3155c948" containerName="revoke-keys"
Sep 19 04:36:38.884143 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:38.884116    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a42681-7nkqg"
Sep 19 04:36:38.886922 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:38.886896    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 19 04:36:38.887045 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:38.886981    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s6xbq\""
Sep 19 04:36:38.895929 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:38.895903    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a42681-7nkqg"]
Sep 19 04:36:38.929870 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:38.929837    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a402aa5d-9335-4138-acca-ae47d3d665ba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-a42681-7nkqg\" (UID: \"a402aa5d-9335-4138-acca-ae47d3d665ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a42681-7nkqg"
Sep 19 04:36:39.031257 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:39.031218    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a402aa5d-9335-4138-acca-ae47d3d665ba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-a42681-7nkqg\" (UID: \"a402aa5d-9335-4138-acca-ae47d3d665ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a42681-7nkqg"
Sep 19 04:36:39.031876 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:39.031857    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a402aa5d-9335-4138-acca-ae47d3d665ba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-a42681-7nkqg\" (UID: \"a402aa5d-9335-4138-acca-ae47d3d665ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a42681-7nkqg"
Sep 19 04:36:39.196793 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:39.196762    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a42681-7nkqg"
Sep 19 04:36:39.483834 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:39.483802    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a42681-7nkqg"]
Sep 19 04:36:39.485265 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:36:39.485228    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda402aa5d_9335_4138_acca_ae47d3d665ba.slice/crio-423f15129117d6ae5df560e79286a30fe244304db36415f7a952fb9aa50c7b1f WatchSource:0}: Error finding container 423f15129117d6ae5df560e79286a30fe244304db36415f7a952fb9aa50c7b1f: Status 404 returned error can't find the container with id 423f15129117d6ae5df560e79286a30fe244304db36415f7a952fb9aa50c7b1f
Sep 19 04:36:40.464539 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:40.464504    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a42681-7nkqg" event={"ID":"a402aa5d-9335-4138-acca-ae47d3d665ba","Type":"ContainerStarted","Data":"a7955c60484c221712e9ab739f7c963426a9c4e6ac94ca231d83afc9a566b84f"}
Sep 19 04:36:40.465016 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:40.464558    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a42681-7nkqg" event={"ID":"a402aa5d-9335-4138-acca-ae47d3d665ba","Type":"ContainerStarted","Data":"423f15129117d6ae5df560e79286a30fe244304db36415f7a952fb9aa50c7b1f"}
Sep 19 04:36:40.486612 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:40.486555    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a42681-7nkqg" podStartSLOduration=2.486533143 podStartE2EDuration="2.486533143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 04:36:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:36:40.482440817 +0000 UTC m=+1379.665401966" watchObservedRunningTime="2026-09-19 04:36:40.486533143 +0000 UTC m=+1379.669494292"
Sep 19 04:36:41.472069 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:41.472033    2579 generic.go:356] "Generic (PLEG): container finished" podID="a402aa5d-9335-4138-acca-ae47d3d665ba" containerID="a7955c60484c221712e9ab739f7c963426a9c4e6ac94ca231d83afc9a566b84f" exitCode=0
Sep 19 04:36:41.472456 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:41.472105    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a42681-7nkqg" event={"ID":"a402aa5d-9335-4138-acca-ae47d3d665ba","Type":"ContainerDied","Data":"a7955c60484c221712e9ab739f7c963426a9c4e6ac94ca231d83afc9a566b84f"}
Sep 19 04:36:42.351392 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:42.351359    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2a94cba7-42fqk"]
Sep 19 04:36:42.355808 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:42.355781    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2a94cba7-42fqk"]
Sep 19 04:36:42.658403 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:42.658377    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a42681-7nkqg"
Sep 19 04:36:42.770070 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:42.770042    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a402aa5d-9335-4138-acca-ae47d3d665ba-maas-api-service-ca\") pod \"a402aa5d-9335-4138-acca-ae47d3d665ba\" (UID: \"a402aa5d-9335-4138-acca-ae47d3d665ba\") "
Sep 19 04:36:42.770432 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:42.770405    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a402aa5d-9335-4138-acca-ae47d3d665ba-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a402aa5d-9335-4138-acca-ae47d3d665ba" (UID: "a402aa5d-9335-4138-acca-ae47d3d665ba"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 04:36:42.871565 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:42.871527    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a402aa5d-9335-4138-acca-ae47d3d665ba-maas-api-service-ca\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:36:43.352667 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:43.352631    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd4e5831-d1b1-4050-b712-3f0a3155c948" path="/var/lib/kubelet/pods/fd4e5831-d1b1-4050-b712-3f0a3155c948/volumes"
Sep 19 04:36:43.501891 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:43.501862    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a42681-7nkqg"
Sep 19 04:36:43.502084 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:43.501896    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a42681-7nkqg" event={"ID":"a402aa5d-9335-4138-acca-ae47d3d665ba","Type":"ContainerDied","Data":"423f15129117d6ae5df560e79286a30fe244304db36415f7a952fb9aa50c7b1f"}
Sep 19 04:36:43.502084 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:43.501941    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="423f15129117d6ae5df560e79286a30fe244304db36415f7a952fb9aa50c7b1f"
Sep 19 04:36:49.900081 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:49.900048    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-a42681-7c7b879c89-9gjfv"]
Sep 19 04:36:49.900604 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:49.900307    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-a42681-7c7b879c89-9gjfv" podUID="8ae90dc0-c97f-4b94-be86-c434a89d100e" containerName="payload-processing" containerID="cri-o://e28ff8512773cf948391e67cfe5731b9621b5fcc219651b5fca070748abd7ae1" gracePeriod=30
Sep 19 04:36:50.100358 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:50.100275    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a42681-7nkqg"]
Sep 19 04:36:50.110293 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:50.110118    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a42681-7nkqg"]
Sep 19 04:36:50.181390 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:50.181360    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-a42681-7c7b879c89-9gjfv"
Sep 19 04:36:50.250162 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:50.250133    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ae90dc0-c97f-4b94-be86-c434a89d100e-plugins-config-volume\") pod \"8ae90dc0-c97f-4b94-be86-c434a89d100e\" (UID: \"8ae90dc0-c97f-4b94-be86-c434a89d100e\") "
Sep 19 04:36:50.250356 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:50.250272    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8btwk\" (UniqueName: \"kubernetes.io/projected/8ae90dc0-c97f-4b94-be86-c434a89d100e-kube-api-access-8btwk\") pod \"8ae90dc0-c97f-4b94-be86-c434a89d100e\" (UID: \"8ae90dc0-c97f-4b94-be86-c434a89d100e\") "
Sep 19 04:36:50.250584 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:50.250557    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ae90dc0-c97f-4b94-be86-c434a89d100e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8ae90dc0-c97f-4b94-be86-c434a89d100e" (UID: "8ae90dc0-c97f-4b94-be86-c434a89d100e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 04:36:50.252601 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:50.252577    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ae90dc0-c97f-4b94-be86-c434a89d100e-kube-api-access-8btwk" (OuterVolumeSpecName: "kube-api-access-8btwk") pod "8ae90dc0-c97f-4b94-be86-c434a89d100e" (UID: "8ae90dc0-c97f-4b94-be86-c434a89d100e"). InnerVolumeSpecName "kube-api-access-8btwk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 04:36:50.351220 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:50.351171    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8btwk\" (UniqueName: \"kubernetes.io/projected/8ae90dc0-c97f-4b94-be86-c434a89d100e-kube-api-access-8btwk\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:36:50.351220 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:50.351208    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ae90dc0-c97f-4b94-be86-c434a89d100e-plugins-config-volume\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:36:50.546827 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:50.546738    2579 generic.go:356] "Generic (PLEG): container finished" podID="8ae90dc0-c97f-4b94-be86-c434a89d100e" containerID="e28ff8512773cf948391e67cfe5731b9621b5fcc219651b5fca070748abd7ae1" exitCode=0
Sep 19 04:36:50.546827 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:50.546790    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-a42681-7c7b879c89-9gjfv"
Sep 19 04:36:50.547022 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:50.546788    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-a42681-7c7b879c89-9gjfv" event={"ID":"8ae90dc0-c97f-4b94-be86-c434a89d100e","Type":"ContainerDied","Data":"e28ff8512773cf948391e67cfe5731b9621b5fcc219651b5fca070748abd7ae1"}
Sep 19 04:36:50.547022 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:50.546908    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-a42681-7c7b879c89-9gjfv" event={"ID":"8ae90dc0-c97f-4b94-be86-c434a89d100e","Type":"ContainerDied","Data":"2616500cac48e914490dc2e4a124b416577bce721bfe31de47ae338f88a10518"}
Sep 19 04:36:50.547022 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:50.546930    2579 scope.go:117] "RemoveContainer" containerID="e28ff8512773cf948391e67cfe5731b9621b5fcc219651b5fca070748abd7ae1"
Sep 19 04:36:50.560416 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:50.560393    2579 scope.go:117] "RemoveContainer" containerID="e28ff8512773cf948391e67cfe5731b9621b5fcc219651b5fca070748abd7ae1"
Sep 19 04:36:50.560770 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:36:50.560751    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e28ff8512773cf948391e67cfe5731b9621b5fcc219651b5fca070748abd7ae1\": container with ID starting with e28ff8512773cf948391e67cfe5731b9621b5fcc219651b5fca070748abd7ae1 not found: ID does not exist" containerID="e28ff8512773cf948391e67cfe5731b9621b5fcc219651b5fca070748abd7ae1"
Sep 19 04:36:50.560826 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:50.560780    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e28ff8512773cf948391e67cfe5731b9621b5fcc219651b5fca070748abd7ae1"} err="failed to get container status \"e28ff8512773cf948391e67cfe5731b9621b5fcc219651b5fca070748abd7ae1\": rpc error: code = NotFound desc = could not find container \"e28ff8512773cf948391e67cfe5731b9621b5fcc219651b5fca070748abd7ae1\": container with ID starting with e28ff8512773cf948391e67cfe5731b9621b5fcc219651b5fca070748abd7ae1 not found: ID does not exist"
Sep 19 04:36:50.571942 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:50.571911    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-a42681-7c7b879c89-9gjfv"]
Sep 19 04:36:50.578095 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:50.578067    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-a42681-7c7b879c89-9gjfv"]
Sep 19 04:36:51.352025 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:51.351991    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ae90dc0-c97f-4b94-be86-c434a89d100e" path="/var/lib/kubelet/pods/8ae90dc0-c97f-4b94-be86-c434a89d100e/volumes"
Sep 19 04:36:51.352410 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:51.352371    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a402aa5d-9335-4138-acca-ae47d3d665ba" path="/var/lib/kubelet/pods/a402aa5d-9335-4138-acca-ae47d3d665ba/volumes"
Sep 19 04:36:55.470334 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:36:55.470296    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-a42681/autores-model-a42681-kserve-6887d75cd4-k4fws"]
Sep 19 04:37:10.491551 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:10.491482    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-a42681/autores-model-a42681-kserve-6887d75cd4-k4fws" podUID="0af35b4a-3ddf-4931-bdd7-890c22863222" containerName="main" containerID="cri-o://2c6eccf8150f61deba47c2d91e9fb454c4d7ba46e1ca58444163e6d8897a8d3c" gracePeriod=45
Sep 19 04:37:10.672323 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:10.672293    2579 generic.go:356] "Generic (PLEG): container finished" podID="0af35b4a-3ddf-4931-bdd7-890c22863222" containerID="2c6eccf8150f61deba47c2d91e9fb454c4d7ba46e1ca58444163e6d8897a8d3c" exitCode=0
Sep 19 04:37:10.672487 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:10.672358    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-a42681/autores-model-a42681-kserve-6887d75cd4-k4fws" event={"ID":"0af35b4a-3ddf-4931-bdd7-890c22863222","Type":"ContainerDied","Data":"2c6eccf8150f61deba47c2d91e9fb454c4d7ba46e1ca58444163e6d8897a8d3c"}
Sep 19 04:37:10.749475 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:10.749409    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-a42681/autores-model-a42681-kserve-6887d75cd4-k4fws"
Sep 19 04:37:10.859028 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:10.858984    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kkqpd\" (UniqueName: \"kubernetes.io/projected/0af35b4a-3ddf-4931-bdd7-890c22863222-kube-api-access-kkqpd\") pod \"0af35b4a-3ddf-4931-bdd7-890c22863222\" (UID: \"0af35b4a-3ddf-4931-bdd7-890c22863222\") "
Sep 19 04:37:10.859233 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:10.859040    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0af35b4a-3ddf-4931-bdd7-890c22863222-tls-certs\") pod \"0af35b4a-3ddf-4931-bdd7-890c22863222\" (UID: \"0af35b4a-3ddf-4931-bdd7-890c22863222\") "
Sep 19 04:37:10.859233 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:10.859065    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0af35b4a-3ddf-4931-bdd7-890c22863222-tmp-dir\") pod \"0af35b4a-3ddf-4931-bdd7-890c22863222\" (UID: \"0af35b4a-3ddf-4931-bdd7-890c22863222\") "
Sep 19 04:37:10.859233 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:10.859087    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0af35b4a-3ddf-4931-bdd7-890c22863222-model-cache\") pod \"0af35b4a-3ddf-4931-bdd7-890c22863222\" (UID: \"0af35b4a-3ddf-4931-bdd7-890c22863222\") "
Sep 19 04:37:10.859233 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:10.859142    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0af35b4a-3ddf-4931-bdd7-890c22863222-dshm\") pod \"0af35b4a-3ddf-4931-bdd7-890c22863222\" (UID: \"0af35b4a-3ddf-4931-bdd7-890c22863222\") "
Sep 19 04:37:10.859233 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:10.859167    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0af35b4a-3ddf-4931-bdd7-890c22863222-home\") pod \"0af35b4a-3ddf-4931-bdd7-890c22863222\" (UID: \"0af35b4a-3ddf-4931-bdd7-890c22863222\") "
Sep 19 04:37:10.859489 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:10.859359    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0af35b4a-3ddf-4931-bdd7-890c22863222-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0af35b4a-3ddf-4931-bdd7-890c22863222" (UID: "0af35b4a-3ddf-4931-bdd7-890c22863222"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:37:10.859489 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:10.859424    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0af35b4a-3ddf-4931-bdd7-890c22863222-model-cache" (OuterVolumeSpecName: "model-cache") pod "0af35b4a-3ddf-4931-bdd7-890c22863222" (UID: "0af35b4a-3ddf-4931-bdd7-890c22863222"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:37:10.859489 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:10.859434    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0af35b4a-3ddf-4931-bdd7-890c22863222-home" (OuterVolumeSpecName: "home") pod "0af35b4a-3ddf-4931-bdd7-890c22863222" (UID: "0af35b4a-3ddf-4931-bdd7-890c22863222"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:37:10.859656 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:10.859583    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0af35b4a-3ddf-4931-bdd7-890c22863222-tmp-dir\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:37:10.859656 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:10.859603    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0af35b4a-3ddf-4931-bdd7-890c22863222-model-cache\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:37:10.859656 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:10.859613    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0af35b4a-3ddf-4931-bdd7-890c22863222-home\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:37:10.861492 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:10.861460    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0af35b4a-3ddf-4931-bdd7-890c22863222-dshm" (OuterVolumeSpecName: "dshm") pod "0af35b4a-3ddf-4931-bdd7-890c22863222" (UID: "0af35b4a-3ddf-4931-bdd7-890c22863222"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:37:10.861616 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:10.861470    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0af35b4a-3ddf-4931-bdd7-890c22863222-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0af35b4a-3ddf-4931-bdd7-890c22863222" (UID: "0af35b4a-3ddf-4931-bdd7-890c22863222"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 04:37:10.861616 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:10.861543    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0af35b4a-3ddf-4931-bdd7-890c22863222-kube-api-access-kkqpd" (OuterVolumeSpecName: "kube-api-access-kkqpd") pod "0af35b4a-3ddf-4931-bdd7-890c22863222" (UID: "0af35b4a-3ddf-4931-bdd7-890c22863222"). InnerVolumeSpecName "kube-api-access-kkqpd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 04:37:10.960279 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:10.960241    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkqpd\" (UniqueName: \"kubernetes.io/projected/0af35b4a-3ddf-4931-bdd7-890c22863222-kube-api-access-kkqpd\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:37:10.960279 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:10.960276    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0af35b4a-3ddf-4931-bdd7-890c22863222-tls-certs\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:37:10.960279 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:10.960285    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0af35b4a-3ddf-4931-bdd7-890c22863222-dshm\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:37:11.679547 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:11.679511    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-a42681/autores-model-a42681-kserve-6887d75cd4-k4fws" event={"ID":"0af35b4a-3ddf-4931-bdd7-890c22863222","Type":"ContainerDied","Data":"65cc5754183827b58e810ab6caf217eb6aed9f19f439134be84e88079fe3851c"}
Sep 19 04:37:11.679547 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:11.679539    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-a42681/autores-model-a42681-kserve-6887d75cd4-k4fws"
Sep 19 04:37:11.680080 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:11.679565    2579 scope.go:117] "RemoveContainer" containerID="2c6eccf8150f61deba47c2d91e9fb454c4d7ba46e1ca58444163e6d8897a8d3c"
Sep 19 04:37:11.698005 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:11.697965    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-a42681/autores-model-a42681-kserve-6887d75cd4-k4fws"]
Sep 19 04:37:11.702880 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:11.702848    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-a42681/autores-model-a42681-kserve-6887d75cd4-k4fws"]
Sep 19 04:37:13.353051 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:13.353020    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0af35b4a-3ddf-4931-bdd7-890c22863222" path="/var/lib/kubelet/pods/0af35b4a-3ddf-4931-bdd7-890c22863222/volumes"
Sep 19 04:37:16.166714 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:16.166667    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d0f416-rvkt8"]
Sep 19 04:37:16.167118 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:16.167103    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a402aa5d-9335-4138-acca-ae47d3d665ba" containerName="revoke-keys"
Sep 19 04:37:16.167157 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:16.167119    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a402aa5d-9335-4138-acca-ae47d3d665ba" containerName="revoke-keys"
Sep 19 04:37:16.167157 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:16.167134    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ae90dc0-c97f-4b94-be86-c434a89d100e" containerName="payload-processing"
Sep 19 04:37:16.167157 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:16.167140    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ae90dc0-c97f-4b94-be86-c434a89d100e" containerName="payload-processing"
Sep 19 04:37:16.167255 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:16.167157    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0af35b4a-3ddf-4931-bdd7-890c22863222" containerName="main"
Sep 19 04:37:16.167255 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:16.167163    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0af35b4a-3ddf-4931-bdd7-890c22863222" containerName="main"
Sep 19 04:37:16.167255 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:16.167225    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a402aa5d-9335-4138-acca-ae47d3d665ba" containerName="revoke-keys"
Sep 19 04:37:16.167255 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:16.167232    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ae90dc0-c97f-4b94-be86-c434a89d100e" containerName="payload-processing"
Sep 19 04:37:16.167255 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:16.167239    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0af35b4a-3ddf-4931-bdd7-890c22863222" containerName="main"
Sep 19 04:37:16.171713 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:16.171676    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d0f416-rvkt8"
Sep 19 04:37:16.175213 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:16.175187    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s6xbq\""
Sep 19 04:37:16.175213 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:16.175188    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 19 04:37:16.178030 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:16.178003    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d0f416-rvkt8"]
Sep 19 04:37:16.213566 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:16.213526    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/afc690f8-17b4-4fab-a493-a64368e112b0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-d0f416-rvkt8\" (UID: \"afc690f8-17b4-4fab-a493-a64368e112b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d0f416-rvkt8"
Sep 19 04:37:16.315103 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:16.315073    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/afc690f8-17b4-4fab-a493-a64368e112b0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-d0f416-rvkt8\" (UID: \"afc690f8-17b4-4fab-a493-a64368e112b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d0f416-rvkt8"
Sep 19 04:37:16.315772 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:16.315751    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/afc690f8-17b4-4fab-a493-a64368e112b0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-d0f416-rvkt8\" (UID: \"afc690f8-17b4-4fab-a493-a64368e112b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d0f416-rvkt8"
Sep 19 04:37:16.485553 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:16.485463    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d0f416-rvkt8"
Sep 19 04:37:16.764940 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:16.764914    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d0f416-rvkt8"]
Sep 19 04:37:16.768740 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:37:16.767418    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafc690f8_17b4_4fab_a493_a64368e112b0.slice/crio-4683d861268494f86a3a0577b8054331879182f0de42746a0e25b83d225bf44d WatchSource:0}: Error finding container 4683d861268494f86a3a0577b8054331879182f0de42746a0e25b83d225bf44d: Status 404 returned error can't find the container with id 4683d861268494f86a3a0577b8054331879182f0de42746a0e25b83d225bf44d
Sep 19 04:37:17.723415 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:17.723379    2579 generic.go:356] "Generic (PLEG): container finished" podID="afc690f8-17b4-4fab-a493-a64368e112b0" containerID="4ec11e2f9a889ec7994f9298796d6f7a16923221124f5669aaf6eedcca50ef65" exitCode=0
Sep 19 04:37:17.723846 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:17.723505    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d0f416-rvkt8" event={"ID":"afc690f8-17b4-4fab-a493-a64368e112b0","Type":"ContainerDied","Data":"4ec11e2f9a889ec7994f9298796d6f7a16923221124f5669aaf6eedcca50ef65"}
Sep 19 04:37:17.723846 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:17.723565    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d0f416-rvkt8" event={"ID":"afc690f8-17b4-4fab-a493-a64368e112b0","Type":"ContainerStarted","Data":"4683d861268494f86a3a0577b8054331879182f0de42746a0e25b83d225bf44d"}
Sep 19 04:37:18.871529 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:18.871503    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d0f416-rvkt8"
Sep 19 04:37:18.942447 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:18.942412    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/afc690f8-17b4-4fab-a493-a64368e112b0-maas-api-service-ca\") pod \"afc690f8-17b4-4fab-a493-a64368e112b0\" (UID: \"afc690f8-17b4-4fab-a493-a64368e112b0\") "
Sep 19 04:37:18.942821 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:18.942779    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afc690f8-17b4-4fab-a493-a64368e112b0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "afc690f8-17b4-4fab-a493-a64368e112b0" (UID: "afc690f8-17b4-4fab-a493-a64368e112b0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 04:37:19.043974 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:19.043888    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/afc690f8-17b4-4fab-a493-a64368e112b0-maas-api-service-ca\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:37:19.737153 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:19.737125    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d0f416-rvkt8"
Sep 19 04:37:19.737341 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:19.737129    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d0f416-rvkt8" event={"ID":"afc690f8-17b4-4fab-a493-a64368e112b0","Type":"ContainerDied","Data":"4683d861268494f86a3a0577b8054331879182f0de42746a0e25b83d225bf44d"}
Sep 19 04:37:19.737341 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:19.737217    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4683d861268494f86a3a0577b8054331879182f0de42746a0e25b83d225bf44d"
Sep 19 04:37:21.035388 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:21.035350    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-d0f416/simulator-w4-d0f416-kserve-7c6665bd67-kqvrt"]
Sep 19 04:37:26.909569 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:26.909535    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-d0f416-6448755f7-d7wgd"]
Sep 19 04:37:26.910028 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:26.909771    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-d0f416-6448755f7-d7wgd" podUID="13dcb6c5-a168-4fbf-8a3f-177d77dea22a" containerName="payload-processing" containerID="cri-o://956b7f471e2a7ce3e7a1d2858703f3b6063f6f17654dbd9ff7420d4f6a362d35" gracePeriod=30
Sep 19 04:37:27.115528 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:27.115491    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d0f416-rvkt8"]
Sep 19 04:37:27.123596 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:27.123543    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d0f416-rvkt8"]
Sep 19 04:37:27.179832 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:27.179808    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-d0f416-6448755f7-d7wgd"
Sep 19 04:37:27.326551 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:27.326511    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13dcb6c5-a168-4fbf-8a3f-177d77dea22a-plugins-config-volume\") pod \"13dcb6c5-a168-4fbf-8a3f-177d77dea22a\" (UID: \"13dcb6c5-a168-4fbf-8a3f-177d77dea22a\") "
Sep 19 04:37:27.326749 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:27.326634    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zc7r7\" (UniqueName: \"kubernetes.io/projected/13dcb6c5-a168-4fbf-8a3f-177d77dea22a-kube-api-access-zc7r7\") pod \"13dcb6c5-a168-4fbf-8a3f-177d77dea22a\" (UID: \"13dcb6c5-a168-4fbf-8a3f-177d77dea22a\") "
Sep 19 04:37:27.326988 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:27.326959    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13dcb6c5-a168-4fbf-8a3f-177d77dea22a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "13dcb6c5-a168-4fbf-8a3f-177d77dea22a" (UID: "13dcb6c5-a168-4fbf-8a3f-177d77dea22a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 04:37:27.328875 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:27.328848    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13dcb6c5-a168-4fbf-8a3f-177d77dea22a-kube-api-access-zc7r7" (OuterVolumeSpecName: "kube-api-access-zc7r7") pod "13dcb6c5-a168-4fbf-8a3f-177d77dea22a" (UID: "13dcb6c5-a168-4fbf-8a3f-177d77dea22a"). InnerVolumeSpecName "kube-api-access-zc7r7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 04:37:27.351570 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:27.351530    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="afc690f8-17b4-4fab-a493-a64368e112b0" path="/var/lib/kubelet/pods/afc690f8-17b4-4fab-a493-a64368e112b0/volumes"
Sep 19 04:37:27.427793 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:27.427677    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13dcb6c5-a168-4fbf-8a3f-177d77dea22a-plugins-config-volume\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:37:27.427793 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:27.427741    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zc7r7\" (UniqueName: \"kubernetes.io/projected/13dcb6c5-a168-4fbf-8a3f-177d77dea22a-kube-api-access-zc7r7\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:37:27.786945 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:27.786845    2579 generic.go:356] "Generic (PLEG): container finished" podID="13dcb6c5-a168-4fbf-8a3f-177d77dea22a" containerID="956b7f471e2a7ce3e7a1d2858703f3b6063f6f17654dbd9ff7420d4f6a362d35" exitCode=0
Sep 19 04:37:27.786945 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:27.786893    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-d0f416-6448755f7-d7wgd"
Sep 19 04:37:27.786945 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:27.786924    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-d0f416-6448755f7-d7wgd" event={"ID":"13dcb6c5-a168-4fbf-8a3f-177d77dea22a","Type":"ContainerDied","Data":"956b7f471e2a7ce3e7a1d2858703f3b6063f6f17654dbd9ff7420d4f6a362d35"}
Sep 19 04:37:27.787304 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:27.786993    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-d0f416-6448755f7-d7wgd" event={"ID":"13dcb6c5-a168-4fbf-8a3f-177d77dea22a","Type":"ContainerDied","Data":"c14e7a1836bebc07a9adbbdd34230d2c740e6e7c1684ac17794df169b8dac221"}
Sep 19 04:37:27.787304 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:27.787022    2579 scope.go:117] "RemoveContainer" containerID="956b7f471e2a7ce3e7a1d2858703f3b6063f6f17654dbd9ff7420d4f6a362d35"
Sep 19 04:37:27.798967 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:27.798946    2579 scope.go:117] "RemoveContainer" containerID="956b7f471e2a7ce3e7a1d2858703f3b6063f6f17654dbd9ff7420d4f6a362d35"
Sep 19 04:37:27.799283 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:37:27.799259    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"956b7f471e2a7ce3e7a1d2858703f3b6063f6f17654dbd9ff7420d4f6a362d35\": container with ID starting with 956b7f471e2a7ce3e7a1d2858703f3b6063f6f17654dbd9ff7420d4f6a362d35 not found: ID does not exist" containerID="956b7f471e2a7ce3e7a1d2858703f3b6063f6f17654dbd9ff7420d4f6a362d35"
Sep 19 04:37:27.799372 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:27.799286    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"956b7f471e2a7ce3e7a1d2858703f3b6063f6f17654dbd9ff7420d4f6a362d35"} err="failed to get container status \"956b7f471e2a7ce3e7a1d2858703f3b6063f6f17654dbd9ff7420d4f6a362d35\": rpc error: code = NotFound desc = could not find container \"956b7f471e2a7ce3e7a1d2858703f3b6063f6f17654dbd9ff7420d4f6a362d35\": container with ID starting with 956b7f471e2a7ce3e7a1d2858703f3b6063f6f17654dbd9ff7420d4f6a362d35 not found: ID does not exist"
Sep 19 04:37:27.808623 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:27.808598    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-d0f416-6448755f7-d7wgd"]
Sep 19 04:37:27.811371 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:27.811345    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-d0f416-6448755f7-d7wgd"]
Sep 19 04:37:29.352299 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:29.352269    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13dcb6c5-a168-4fbf-8a3f-177d77dea22a" path="/var/lib/kubelet/pods/13dcb6c5-a168-4fbf-8a3f-177d77dea22a/volumes"
Sep 19 04:37:33.827397 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:33.827320    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk"]
Sep 19 04:37:33.827794 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:33.827525    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk" podUID="6239a719-1815-4ef5-8b52-a488185c44a7" containerName="istio-proxy" containerID="cri-o://d8ddcbc8882c2d447c6ec7997a8923646042ddb37493699e66940f73c0982e2b" gracePeriod=30
Sep 19 04:37:34.737109 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:34.737067    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-bcbdad/test-model-bcbdad-kserve-6488bff88b-2tfl6"]
Sep 19 04:37:34.737487 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:34.737475    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afc690f8-17b4-4fab-a493-a64368e112b0" containerName="revoke-keys"
Sep 19 04:37:34.737532 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:34.737490    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="afc690f8-17b4-4fab-a493-a64368e112b0" containerName="revoke-keys"
Sep 19 04:37:34.737532 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:34.737508    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13dcb6c5-a168-4fbf-8a3f-177d77dea22a" containerName="payload-processing"
Sep 19 04:37:34.737532 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:34.737514    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="13dcb6c5-a168-4fbf-8a3f-177d77dea22a" containerName="payload-processing"
Sep 19 04:37:34.737628 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:34.737576    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="13dcb6c5-a168-4fbf-8a3f-177d77dea22a" containerName="payload-processing"
Sep 19 04:37:34.737628 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:34.737586    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="afc690f8-17b4-4fab-a493-a64368e112b0" containerName="revoke-keys"
Sep 19 04:37:34.742044 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:34.742022    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-bcbdad/test-model-bcbdad-kserve-6488bff88b-2tfl6"
Sep 19 04:37:34.767436 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:34.767409    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-bcbdad\"/\"test-model-bcbdad-kserve-self-signed-certs\""
Sep 19 04:37:34.774584 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:34.774441    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-bcbdad\"/\"kube-root-ca.crt\""
Sep 19 04:37:34.774584 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:34.774447    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-bcbdad\"/\"openshift-service-ca.crt\""
Sep 19 04:37:34.774584 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:34.774447    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-bcbdad\"/\"default-dockercfg-k86q6\""
Sep 19 04:37:34.778317 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:34.778296    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-bcbdad/test-model-bcbdad-kserve-6488bff88b-2tfl6"]
Sep 19 04:37:34.901280 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:34.901242    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04fdd4d9-a25e-49ed-b803-9a3eb2c68745-home\") pod \"test-model-bcbdad-kserve-6488bff88b-2tfl6\" (UID: \"04fdd4d9-a25e-49ed-b803-9a3eb2c68745\") " pod="ai-tenant-e2e-inf-a-bcbdad/test-model-bcbdad-kserve-6488bff88b-2tfl6"
Sep 19 04:37:34.901280 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:34.901278    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04fdd4d9-a25e-49ed-b803-9a3eb2c68745-tmp-dir\") pod \"test-model-bcbdad-kserve-6488bff88b-2tfl6\" (UID: \"04fdd4d9-a25e-49ed-b803-9a3eb2c68745\") " pod="ai-tenant-e2e-inf-a-bcbdad/test-model-bcbdad-kserve-6488bff88b-2tfl6"
Sep 19 04:37:34.901667 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:34.901308    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04fdd4d9-a25e-49ed-b803-9a3eb2c68745-model-cache\") pod \"test-model-bcbdad-kserve-6488bff88b-2tfl6\" (UID: \"04fdd4d9-a25e-49ed-b803-9a3eb2c68745\") " pod="ai-tenant-e2e-inf-a-bcbdad/test-model-bcbdad-kserve-6488bff88b-2tfl6"
Sep 19 04:37:34.901667 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:34.901325    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rh7ds\" (UniqueName: \"kubernetes.io/projected/04fdd4d9-a25e-49ed-b803-9a3eb2c68745-kube-api-access-rh7ds\") pod \"test-model-bcbdad-kserve-6488bff88b-2tfl6\" (UID: \"04fdd4d9-a25e-49ed-b803-9a3eb2c68745\") " pod="ai-tenant-e2e-inf-a-bcbdad/test-model-bcbdad-kserve-6488bff88b-2tfl6"
Sep 19 04:37:34.901667 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:34.901391    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04fdd4d9-a25e-49ed-b803-9a3eb2c68745-dshm\") pod \"test-model-bcbdad-kserve-6488bff88b-2tfl6\" (UID: \"04fdd4d9-a25e-49ed-b803-9a3eb2c68745\") " pod="ai-tenant-e2e-inf-a-bcbdad/test-model-bcbdad-kserve-6488bff88b-2tfl6"
Sep 19 04:37:34.901667 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:34.901450    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04fdd4d9-a25e-49ed-b803-9a3eb2c68745-tls-certs\") pod \"test-model-bcbdad-kserve-6488bff88b-2tfl6\" (UID: \"04fdd4d9-a25e-49ed-b803-9a3eb2c68745\") " pod="ai-tenant-e2e-inf-a-bcbdad/test-model-bcbdad-kserve-6488bff88b-2tfl6"
Sep 19 04:37:35.002088 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:35.002004    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04fdd4d9-a25e-49ed-b803-9a3eb2c68745-home\") pod \"test-model-bcbdad-kserve-6488bff88b-2tfl6\" (UID: \"04fdd4d9-a25e-49ed-b803-9a3eb2c68745\") " pod="ai-tenant-e2e-inf-a-bcbdad/test-model-bcbdad-kserve-6488bff88b-2tfl6"
Sep 19 04:37:35.002088 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:35.002038    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04fdd4d9-a25e-49ed-b803-9a3eb2c68745-tmp-dir\") pod \"test-model-bcbdad-kserve-6488bff88b-2tfl6\" (UID: \"04fdd4d9-a25e-49ed-b803-9a3eb2c68745\") " pod="ai-tenant-e2e-inf-a-bcbdad/test-model-bcbdad-kserve-6488bff88b-2tfl6"
Sep 19 04:37:35.002313 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:35.002182    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04fdd4d9-a25e-49ed-b803-9a3eb2c68745-model-cache\") pod \"test-model-bcbdad-kserve-6488bff88b-2tfl6\" (UID: \"04fdd4d9-a25e-49ed-b803-9a3eb2c68745\") " pod="ai-tenant-e2e-inf-a-bcbdad/test-model-bcbdad-kserve-6488bff88b-2tfl6"
Sep 19 04:37:35.002313 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:35.002222    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rh7ds\" (UniqueName: \"kubernetes.io/projected/04fdd4d9-a25e-49ed-b803-9a3eb2c68745-kube-api-access-rh7ds\") pod \"test-model-bcbdad-kserve-6488bff88b-2tfl6\" (UID: \"04fdd4d9-a25e-49ed-b803-9a3eb2c68745\") " pod="ai-tenant-e2e-inf-a-bcbdad/test-model-bcbdad-kserve-6488bff88b-2tfl6"
Sep 19 04:37:35.002313 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:35.002272    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04fdd4d9-a25e-49ed-b803-9a3eb2c68745-dshm\") pod \"test-model-bcbdad-kserve-6488bff88b-2tfl6\" (UID: \"04fdd4d9-a25e-49ed-b803-9a3eb2c68745\") " pod="ai-tenant-e2e-inf-a-bcbdad/test-model-bcbdad-kserve-6488bff88b-2tfl6"
Sep 19 04:37:35.002486 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:35.002354    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04fdd4d9-a25e-49ed-b803-9a3eb2c68745-home\") pod \"test-model-bcbdad-kserve-6488bff88b-2tfl6\" (UID: \"04fdd4d9-a25e-49ed-b803-9a3eb2c68745\") " pod="ai-tenant-e2e-inf-a-bcbdad/test-model-bcbdad-kserve-6488bff88b-2tfl6"
Sep 19 04:37:35.002486 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:35.002382    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04fdd4d9-a25e-49ed-b803-9a3eb2c68745-tls-certs\") pod \"test-model-bcbdad-kserve-6488bff88b-2tfl6\" (UID: \"04fdd4d9-a25e-49ed-b803-9a3eb2c68745\") " pod="ai-tenant-e2e-inf-a-bcbdad/test-model-bcbdad-kserve-6488bff88b-2tfl6"
Sep 19 04:37:35.002486 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:35.002401    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04fdd4d9-a25e-49ed-b803-9a3eb2c68745-tmp-dir\") pod \"test-model-bcbdad-kserve-6488bff88b-2tfl6\" (UID: \"04fdd4d9-a25e-49ed-b803-9a3eb2c68745\") " pod="ai-tenant-e2e-inf-a-bcbdad/test-model-bcbdad-kserve-6488bff88b-2tfl6"
Sep 19 04:37:35.002486 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:35.002453    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04fdd4d9-a25e-49ed-b803-9a3eb2c68745-model-cache\") pod \"test-model-bcbdad-kserve-6488bff88b-2tfl6\" (UID: \"04fdd4d9-a25e-49ed-b803-9a3eb2c68745\") " pod="ai-tenant-e2e-inf-a-bcbdad/test-model-bcbdad-kserve-6488bff88b-2tfl6"
Sep 19 04:37:35.004631 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:35.004606    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04fdd4d9-a25e-49ed-b803-9a3eb2c68745-dshm\") pod \"test-model-bcbdad-kserve-6488bff88b-2tfl6\" (UID: \"04fdd4d9-a25e-49ed-b803-9a3eb2c68745\") " pod="ai-tenant-e2e-inf-a-bcbdad/test-model-bcbdad-kserve-6488bff88b-2tfl6"
Sep 19 04:37:35.005040 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:35.005020    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04fdd4d9-a25e-49ed-b803-9a3eb2c68745-tls-certs\") pod \"test-model-bcbdad-kserve-6488bff88b-2tfl6\" (UID: \"04fdd4d9-a25e-49ed-b803-9a3eb2c68745\") " pod="ai-tenant-e2e-inf-a-bcbdad/test-model-bcbdad-kserve-6488bff88b-2tfl6"
Sep 19 04:37:35.019915 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:35.019895    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rh7ds\" (UniqueName: \"kubernetes.io/projected/04fdd4d9-a25e-49ed-b803-9a3eb2c68745-kube-api-access-rh7ds\") pod \"test-model-bcbdad-kserve-6488bff88b-2tfl6\" (UID: \"04fdd4d9-a25e-49ed-b803-9a3eb2c68745\") " pod="ai-tenant-e2e-inf-a-bcbdad/test-model-bcbdad-kserve-6488bff88b-2tfl6"
Sep 19 04:37:35.050971 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:35.050938    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-bcbdad/test-model-bcbdad-kserve-6488bff88b-2tfl6"
Sep 19 04:37:35.334313 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:35.332574    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-bcbdad/test-model-bcbdad-kserve-6488bff88b-2tfl6"]
Sep 19 04:37:35.840970 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:35.840931    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-bcbdad/test-model-bcbdad-kserve-6488bff88b-2tfl6" event={"ID":"04fdd4d9-a25e-49ed-b803-9a3eb2c68745","Type":"ContainerStarted","Data":"89f62edbb05072b4e9725fde1bc356a239f7ddca18b19719ff468bc527215869"}
Sep 19 04:37:35.841165 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:35.840981    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-bcbdad/test-model-bcbdad-kserve-6488bff88b-2tfl6" event={"ID":"04fdd4d9-a25e-49ed-b803-9a3eb2c68745","Type":"ContainerStarted","Data":"43ca6635f3c79dfedf038975465ce0f536190577efc293d0f7f3c647e0bcc095"}
Sep 19 04:37:35.861026 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:35.860971    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-bcbdad/test-model-bcbdad-kserve-6488bff88b-2tfl6" podStartSLOduration=1.8609551469999999 podStartE2EDuration="1.860955147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 04:37:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:37:35.860218886 +0000 UTC m=+1435.043180034" watchObservedRunningTime="2026-09-19 04:37:35.860955147 +0000 UTC m=+1435.043916295"
Sep 19 04:37:36.051237 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:36.051201    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-bcbdad/test-model-bcbdad-kserve-6488bff88b-2tfl6"
Sep 19 04:37:36.055369 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:36.055337    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-d0f416/simulator-w4-d0f416-kserve-7c6665bd67-kqvrt" podUID="3ca22b40-223c-472d-879a-0194160787f0" containerName="main" containerID="cri-o://32c8f8982b2a3c3f58a7051aedb84b6b991164f992d5f832b9dac038203d045c" gracePeriod=45
Sep 19 04:37:36.193647 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:36.193607    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d0f416/simulator-w4-d0f416-kserve-7c6665bd67-kqvrt" podUID="3ca22b40-223c-472d-879a-0194160787f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": dial tcp 10.134.0.59:8000: connect: connection refused"
Sep 19 04:37:36.313067 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:36.313046    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d0f416/simulator-w4-d0f416-kserve-7c6665bd67-kqvrt"
Sep 19 04:37:36.337368 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:36.337336    2579 patch_prober.go:28] interesting pod/e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 19 04:37:36.337508 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:36.337390    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk" podUID="6239a719-1815-4ef5-8b52-a488185c44a7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 04:37:36.416401 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:36.416352    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ntk78\" (UniqueName: \"kubernetes.io/projected/3ca22b40-223c-472d-879a-0194160787f0-kube-api-access-ntk78\") pod \"3ca22b40-223c-472d-879a-0194160787f0\" (UID: \"3ca22b40-223c-472d-879a-0194160787f0\") "
Sep 19 04:37:36.416401 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:36.416408    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ca22b40-223c-472d-879a-0194160787f0-model-cache\") pod \"3ca22b40-223c-472d-879a-0194160787f0\" (UID: \"3ca22b40-223c-472d-879a-0194160787f0\") "
Sep 19 04:37:36.416616 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:36.416440    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ca22b40-223c-472d-879a-0194160787f0-tls-certs\") pod \"3ca22b40-223c-472d-879a-0194160787f0\" (UID: \"3ca22b40-223c-472d-879a-0194160787f0\") "
Sep 19 04:37:36.416616 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:36.416467    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ca22b40-223c-472d-879a-0194160787f0-home\") pod \"3ca22b40-223c-472d-879a-0194160787f0\" (UID: \"3ca22b40-223c-472d-879a-0194160787f0\") "
Sep 19 04:37:36.416616 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:36.416495    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ca22b40-223c-472d-879a-0194160787f0-dshm\") pod \"3ca22b40-223c-472d-879a-0194160787f0\" (UID: \"3ca22b40-223c-472d-879a-0194160787f0\") "
Sep 19 04:37:36.416616 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:36.416573    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ca22b40-223c-472d-879a-0194160787f0-tmp-dir\") pod \"3ca22b40-223c-472d-879a-0194160787f0\" (UID: \"3ca22b40-223c-472d-879a-0194160787f0\") "
Sep 19 04:37:36.416892 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:36.416729    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ca22b40-223c-472d-879a-0194160787f0-home" (OuterVolumeSpecName: "home") pod "3ca22b40-223c-472d-879a-0194160787f0" (UID: "3ca22b40-223c-472d-879a-0194160787f0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:37:36.416892 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:36.416762    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ca22b40-223c-472d-879a-0194160787f0-model-cache" (OuterVolumeSpecName: "model-cache") pod "3ca22b40-223c-472d-879a-0194160787f0" (UID: "3ca22b40-223c-472d-879a-0194160787f0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:37:36.417006 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:36.416958    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ca22b40-223c-472d-879a-0194160787f0-model-cache\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:37:36.417006 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:36.416973    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ca22b40-223c-472d-879a-0194160787f0-home\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:37:36.417006 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:36.416968    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ca22b40-223c-472d-879a-0194160787f0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3ca22b40-223c-472d-879a-0194160787f0" (UID: "3ca22b40-223c-472d-879a-0194160787f0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:37:36.418983 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:36.418929    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ca22b40-223c-472d-879a-0194160787f0-dshm" (OuterVolumeSpecName: "dshm") pod "3ca22b40-223c-472d-879a-0194160787f0" (UID: "3ca22b40-223c-472d-879a-0194160787f0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:37:36.419229 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:36.419203    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ca22b40-223c-472d-879a-0194160787f0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3ca22b40-223c-472d-879a-0194160787f0" (UID: "3ca22b40-223c-472d-879a-0194160787f0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 04:37:36.419229 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:36.419216    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ca22b40-223c-472d-879a-0194160787f0-kube-api-access-ntk78" (OuterVolumeSpecName: "kube-api-access-ntk78") pod "3ca22b40-223c-472d-879a-0194160787f0" (UID: "3ca22b40-223c-472d-879a-0194160787f0"). InnerVolumeSpecName "kube-api-access-ntk78". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 04:37:36.518088 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:36.518055    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntk78\" (UniqueName: \"kubernetes.io/projected/3ca22b40-223c-472d-879a-0194160787f0-kube-api-access-ntk78\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:37:36.518088 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:36.518088    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ca22b40-223c-472d-879a-0194160787f0-tls-certs\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:37:36.518305 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:36.518103    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ca22b40-223c-472d-879a-0194160787f0-dshm\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:37:36.518305 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:36.518116    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ca22b40-223c-472d-879a-0194160787f0-tmp-dir\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:37:36.848677 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:36.848587    2579 generic.go:356] "Generic (PLEG): container finished" podID="3ca22b40-223c-472d-879a-0194160787f0" containerID="32c8f8982b2a3c3f58a7051aedb84b6b991164f992d5f832b9dac038203d045c" exitCode=0
Sep 19 04:37:36.848677 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:36.848635    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d0f416/simulator-w4-d0f416-kserve-7c6665bd67-kqvrt"
Sep 19 04:37:36.848872 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:36.848675    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d0f416/simulator-w4-d0f416-kserve-7c6665bd67-kqvrt" event={"ID":"3ca22b40-223c-472d-879a-0194160787f0","Type":"ContainerDied","Data":"32c8f8982b2a3c3f58a7051aedb84b6b991164f992d5f832b9dac038203d045c"}
Sep 19 04:37:36.848872 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:36.848747    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d0f416/simulator-w4-d0f416-kserve-7c6665bd67-kqvrt" event={"ID":"3ca22b40-223c-472d-879a-0194160787f0","Type":"ContainerDied","Data":"ba5b7a9afdc026543bf4a545fccf2218c2d556c76d4160c821afac6ebedde0e7"}
Sep 19 04:37:36.848872 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:36.848764    2579 scope.go:117] "RemoveContainer" containerID="32c8f8982b2a3c3f58a7051aedb84b6b991164f992d5f832b9dac038203d045c"
Sep 19 04:37:36.858716 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:36.858684    2579 scope.go:117] "RemoveContainer" containerID="32c8f8982b2a3c3f58a7051aedb84b6b991164f992d5f832b9dac038203d045c"
Sep 19 04:37:36.858995 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:37:36.858973    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32c8f8982b2a3c3f58a7051aedb84b6b991164f992d5f832b9dac038203d045c\": container with ID starting with 32c8f8982b2a3c3f58a7051aedb84b6b991164f992d5f832b9dac038203d045c not found: ID does not exist" containerID="32c8f8982b2a3c3f58a7051aedb84b6b991164f992d5f832b9dac038203d045c"
Sep 19 04:37:36.859074 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:36.859007    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32c8f8982b2a3c3f58a7051aedb84b6b991164f992d5f832b9dac038203d045c"} err="failed to get container status \"32c8f8982b2a3c3f58a7051aedb84b6b991164f992d5f832b9dac038203d045c\": rpc error: code = NotFound desc = could not find container \"32c8f8982b2a3c3f58a7051aedb84b6b991164f992d5f832b9dac038203d045c\": container with ID starting with 32c8f8982b2a3c3f58a7051aedb84b6b991164f992d5f832b9dac038203d045c not found: ID does not exist"
Sep 19 04:37:36.872903 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:36.872875    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-d0f416/simulator-w4-d0f416-kserve-7c6665bd67-kqvrt"]
Sep 19 04:37:36.878582 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:36.878559    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-d0f416/simulator-w4-d0f416-kserve-7c6665bd67-kqvrt"]
Sep 19 04:37:37.351378 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:37.351345    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ca22b40-223c-472d-879a-0194160787f0" path="/var/lib/kubelet/pods/3ca22b40-223c-472d-879a-0194160787f0/volumes"
Sep 19 04:37:39.082241 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:39.082212    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk"
Sep 19 04:37:39.246841 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:39.246811    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6239a719-1815-4ef5-8b52-a488185c44a7-credential-socket\") pod \"6239a719-1815-4ef5-8b52-a488185c44a7\" (UID: \"6239a719-1815-4ef5-8b52-a488185c44a7\") "
Sep 19 04:37:39.246841 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:39.246847    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6239a719-1815-4ef5-8b52-a488185c44a7-istio-podinfo\") pod \"6239a719-1815-4ef5-8b52-a488185c44a7\" (UID: \"6239a719-1815-4ef5-8b52-a488185c44a7\") "
Sep 19 04:37:39.247040 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:39.246885    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6239a719-1815-4ef5-8b52-a488185c44a7-istio-token\") pod \"6239a719-1815-4ef5-8b52-a488185c44a7\" (UID: \"6239a719-1815-4ef5-8b52-a488185c44a7\") "
Sep 19 04:37:39.247040 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:39.246953    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6239a719-1815-4ef5-8b52-a488185c44a7-istio-data\") pod \"6239a719-1815-4ef5-8b52-a488185c44a7\" (UID: \"6239a719-1815-4ef5-8b52-a488185c44a7\") "
Sep 19 04:37:39.247040 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:39.247007    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6239a719-1815-4ef5-8b52-a488185c44a7-istiod-ca-cert\") pod \"6239a719-1815-4ef5-8b52-a488185c44a7\" (UID: \"6239a719-1815-4ef5-8b52-a488185c44a7\") "
Sep 19 04:37:39.247040 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:39.247034    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6239a719-1815-4ef5-8b52-a488185c44a7-istio-envoy\") pod \"6239a719-1815-4ef5-8b52-a488185c44a7\" (UID: \"6239a719-1815-4ef5-8b52-a488185c44a7\") "
Sep 19 04:37:39.247230 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:39.247076    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6239a719-1815-4ef5-8b52-a488185c44a7-workload-certs\") pod \"6239a719-1815-4ef5-8b52-a488185c44a7\" (UID: \"6239a719-1815-4ef5-8b52-a488185c44a7\") "
Sep 19 04:37:39.247230 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:39.247118    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6239a719-1815-4ef5-8b52-a488185c44a7-workload-socket\") pod \"6239a719-1815-4ef5-8b52-a488185c44a7\" (UID: \"6239a719-1815-4ef5-8b52-a488185c44a7\") "
Sep 19 04:37:39.247230 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:39.247133    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6239a719-1815-4ef5-8b52-a488185c44a7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6239a719-1815-4ef5-8b52-a488185c44a7" (UID: "6239a719-1815-4ef5-8b52-a488185c44a7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:37:39.247230 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:39.247191    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zf6bp\" (UniqueName: \"kubernetes.io/projected/6239a719-1815-4ef5-8b52-a488185c44a7-kube-api-access-zf6bp\") pod \"6239a719-1815-4ef5-8b52-a488185c44a7\" (UID: \"6239a719-1815-4ef5-8b52-a488185c44a7\") "
Sep 19 04:37:39.247513 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:39.247339    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6239a719-1815-4ef5-8b52-a488185c44a7-istio-data" (OuterVolumeSpecName: "istio-data") pod "6239a719-1815-4ef5-8b52-a488185c44a7" (UID: "6239a719-1815-4ef5-8b52-a488185c44a7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:37:39.247513 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:39.247444    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6239a719-1815-4ef5-8b52-a488185c44a7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6239a719-1815-4ef5-8b52-a488185c44a7" (UID: "6239a719-1815-4ef5-8b52-a488185c44a7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 04:37:39.247612 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:39.247347    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6239a719-1815-4ef5-8b52-a488185c44a7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6239a719-1815-4ef5-8b52-a488185c44a7" (UID: "6239a719-1815-4ef5-8b52-a488185c44a7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:37:39.247612 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:39.247541    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6239a719-1815-4ef5-8b52-a488185c44a7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6239a719-1815-4ef5-8b52-a488185c44a7" (UID: "6239a719-1815-4ef5-8b52-a488185c44a7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:37:39.247872 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:39.247853    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6239a719-1815-4ef5-8b52-a488185c44a7-istio-data\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:37:39.247928 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:39.247879    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6239a719-1815-4ef5-8b52-a488185c44a7-istiod-ca-cert\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:37:39.247928 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:39.247892    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6239a719-1815-4ef5-8b52-a488185c44a7-workload-certs\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:37:39.247928 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:39.247906    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6239a719-1815-4ef5-8b52-a488185c44a7-workload-socket\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:37:39.247928 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:39.247919    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6239a719-1815-4ef5-8b52-a488185c44a7-credential-socket\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:37:39.249247 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:39.249229    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6239a719-1815-4ef5-8b52-a488185c44a7-istio-token" (OuterVolumeSpecName: "istio-token") pod "6239a719-1815-4ef5-8b52-a488185c44a7" (UID: "6239a719-1815-4ef5-8b52-a488185c44a7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 04:37:39.249522 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:39.249498    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6239a719-1815-4ef5-8b52-a488185c44a7-kube-api-access-zf6bp" (OuterVolumeSpecName: "kube-api-access-zf6bp") pod "6239a719-1815-4ef5-8b52-a488185c44a7" (UID: "6239a719-1815-4ef5-8b52-a488185c44a7"). InnerVolumeSpecName "kube-api-access-zf6bp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 04:37:39.249770 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:39.249750    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6239a719-1815-4ef5-8b52-a488185c44a7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6239a719-1815-4ef5-8b52-a488185c44a7" (UID: "6239a719-1815-4ef5-8b52-a488185c44a7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 19 04:37:39.249893 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:39.249875    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6239a719-1815-4ef5-8b52-a488185c44a7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6239a719-1815-4ef5-8b52-a488185c44a7" (UID: "6239a719-1815-4ef5-8b52-a488185c44a7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:37:39.348790 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:39.348761    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zf6bp\" (UniqueName: \"kubernetes.io/projected/6239a719-1815-4ef5-8b52-a488185c44a7-kube-api-access-zf6bp\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:37:39.348790 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:39.348786    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6239a719-1815-4ef5-8b52-a488185c44a7-istio-podinfo\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:37:39.348790 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:39.348795    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6239a719-1815-4ef5-8b52-a488185c44a7-istio-token\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:37:39.349008 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:39.348803    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6239a719-1815-4ef5-8b52-a488185c44a7-istio-envoy\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:37:39.871648 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:39.871617    2579 generic.go:356] "Generic (PLEG): container finished" podID="6239a719-1815-4ef5-8b52-a488185c44a7" containerID="d8ddcbc8882c2d447c6ec7997a8923646042ddb37493699e66940f73c0982e2b" exitCode=0
Sep 19 04:37:39.871880 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:39.871666    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk"
Sep 19 04:37:39.871880 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:39.871733    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk" event={"ID":"6239a719-1815-4ef5-8b52-a488185c44a7","Type":"ContainerDied","Data":"d8ddcbc8882c2d447c6ec7997a8923646042ddb37493699e66940f73c0982e2b"}
Sep 19 04:37:39.871880 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:39.871800    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk" event={"ID":"6239a719-1815-4ef5-8b52-a488185c44a7","Type":"ContainerDied","Data":"7163390a983df51ec232e9daac7d3634a5e5b6ed7065ed9cf91df63b1312c442"}
Sep 19 04:37:39.871880 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:39.871834    2579 scope.go:117] "RemoveContainer" containerID="d8ddcbc8882c2d447c6ec7997a8923646042ddb37493699e66940f73c0982e2b"
Sep 19 04:37:39.881255 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:39.881238    2579 scope.go:117] "RemoveContainer" containerID="d57b524321eff87e26ffa888663d27890113781fdb13fea27b1883045f22b8e9"
Sep 19 04:37:39.891672 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:39.891647    2579 scope.go:117] "RemoveContainer" containerID="d8ddcbc8882c2d447c6ec7997a8923646042ddb37493699e66940f73c0982e2b"
Sep 19 04:37:39.891865 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:39.891844    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk"]
Sep 19 04:37:39.892113 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:37:39.892082    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8ddcbc8882c2d447c6ec7997a8923646042ddb37493699e66940f73c0982e2b\": container with ID starting with d8ddcbc8882c2d447c6ec7997a8923646042ddb37493699e66940f73c0982e2b not found: ID does not exist" containerID="d8ddcbc8882c2d447c6ec7997a8923646042ddb37493699e66940f73c0982e2b"
Sep 19 04:37:39.892220 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:39.892124    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8ddcbc8882c2d447c6ec7997a8923646042ddb37493699e66940f73c0982e2b"} err="failed to get container status \"d8ddcbc8882c2d447c6ec7997a8923646042ddb37493699e66940f73c0982e2b\": rpc error: code = NotFound desc = could not find container \"d8ddcbc8882c2d447c6ec7997a8923646042ddb37493699e66940f73c0982e2b\": container with ID starting with d8ddcbc8882c2d447c6ec7997a8923646042ddb37493699e66940f73c0982e2b not found: ID does not exist"
Sep 19 04:37:39.892220 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:39.892151    2579 scope.go:117] "RemoveContainer" containerID="d57b524321eff87e26ffa888663d27890113781fdb13fea27b1883045f22b8e9"
Sep 19 04:37:39.892649 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:37:39.892574    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d57b524321eff87e26ffa888663d27890113781fdb13fea27b1883045f22b8e9\": container with ID starting with d57b524321eff87e26ffa888663d27890113781fdb13fea27b1883045f22b8e9 not found: ID does not exist" containerID="d57b524321eff87e26ffa888663d27890113781fdb13fea27b1883045f22b8e9"
Sep 19 04:37:39.892649 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:39.892613    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d57b524321eff87e26ffa888663d27890113781fdb13fea27b1883045f22b8e9"} err="failed to get container status \"d57b524321eff87e26ffa888663d27890113781fdb13fea27b1883045f22b8e9\": rpc error: code = NotFound desc = could not find container \"d57b524321eff87e26ffa888663d27890113781fdb13fea27b1883045f22b8e9\": container with ID starting with d57b524321eff87e26ffa888663d27890113781fdb13fea27b1883045f22b8e9 not found: ID does not exist"
Sep 19 04:37:39.895480 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:39.895454    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-d0f416-openshift-default-6d48cfb746-7bgbk"]
Sep 19 04:37:41.351539 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:41.351510    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6239a719-1815-4ef5-8b52-a488185c44a7" path="/var/lib/kubelet/pods/6239a719-1815-4ef5-8b52-a488185c44a7/volumes"
Sep 19 04:37:45.051894 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:45.051856    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-bcbdad/test-model-bcbdad-kserve-6488bff88b-2tfl6"
Sep 19 04:37:45.064958 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:45.064932    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-bcbdad/test-model-bcbdad-kserve-6488bff88b-2tfl6"
Sep 19 04:37:45.924711 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:45.924663    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-bcbdad/test-model-bcbdad-kserve-6488bff88b-2tfl6"
Sep 19 04:37:50.331166 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:50.331129    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg"]
Sep 19 04:37:50.331537 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:50.331523    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ca22b40-223c-472d-879a-0194160787f0" containerName="main"
Sep 19 04:37:50.331581 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:50.331538    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ca22b40-223c-472d-879a-0194160787f0" containerName="main"
Sep 19 04:37:50.331581 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:50.331556    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6239a719-1815-4ef5-8b52-a488185c44a7" containerName="istio-proxy"
Sep 19 04:37:50.331581 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:50.331564    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6239a719-1815-4ef5-8b52-a488185c44a7" containerName="istio-proxy"
Sep 19 04:37:50.331673 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:50.331625    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6239a719-1815-4ef5-8b52-a488185c44a7" containerName="istio-proxy"
Sep 19 04:37:50.331673 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:50.331634    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6239a719-1815-4ef5-8b52-a488185c44a7" containerName="istio-proxy"
Sep 19 04:37:50.331673 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:50.331642    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ca22b40-223c-472d-879a-0194160787f0" containerName="main"
Sep 19 04:37:50.336423 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:50.336407    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg"
Sep 19 04:37:50.341899 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:50.341868    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-a70050-openshift-default-dockercfg-m667w\""
Sep 19 04:37:50.351095 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:50.351074    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg"]
Sep 19 04:37:50.456335 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:50.456303    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03b3cd51-7a8f-4708-bd12-e9043fc4004d-istiod-ca-cert\") pod \"e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg\" (UID: \"03b3cd51-7a8f-4708-bd12-e9043fc4004d\") " pod="openshift-ingress/e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg"
Sep 19 04:37:50.456335 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:50.456337    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03b3cd51-7a8f-4708-bd12-e9043fc4004d-istio-envoy\") pod \"e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg\" (UID: \"03b3cd51-7a8f-4708-bd12-e9043fc4004d\") " pod="openshift-ingress/e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg"
Sep 19 04:37:50.456559 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:50.456387    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03b3cd51-7a8f-4708-bd12-e9043fc4004d-credential-socket\") pod \"e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg\" (UID: \"03b3cd51-7a8f-4708-bd12-e9043fc4004d\") " pod="openshift-ingress/e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg"
Sep 19 04:37:50.456559 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:50.456503    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03b3cd51-7a8f-4708-bd12-e9043fc4004d-istio-podinfo\") pod \"e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg\" (UID: \"03b3cd51-7a8f-4708-bd12-e9043fc4004d\") " pod="openshift-ingress/e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg"
Sep 19 04:37:50.456559 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:50.456525    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03b3cd51-7a8f-4708-bd12-e9043fc4004d-istio-data\") pod \"e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg\" (UID: \"03b3cd51-7a8f-4708-bd12-e9043fc4004d\") " pod="openshift-ingress/e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg"
Sep 19 04:37:50.456685 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:50.456572    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03b3cd51-7a8f-4708-bd12-e9043fc4004d-istio-token\") pod \"e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg\" (UID: \"03b3cd51-7a8f-4708-bd12-e9043fc4004d\") " pod="openshift-ingress/e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg"
Sep 19 04:37:50.456685 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:50.456609    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhq6g\" (UniqueName: \"kubernetes.io/projected/03b3cd51-7a8f-4708-bd12-e9043fc4004d-kube-api-access-rhq6g\") pod \"e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg\" (UID: \"03b3cd51-7a8f-4708-bd12-e9043fc4004d\") " pod="openshift-ingress/e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg"
Sep 19 04:37:50.456685 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:50.456630    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03b3cd51-7a8f-4708-bd12-e9043fc4004d-workload-socket\") pod \"e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg\" (UID: \"03b3cd51-7a8f-4708-bd12-e9043fc4004d\") " pod="openshift-ingress/e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg"
Sep 19 04:37:50.456685 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:50.456656    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03b3cd51-7a8f-4708-bd12-e9043fc4004d-workload-certs\") pod \"e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg\" (UID: \"03b3cd51-7a8f-4708-bd12-e9043fc4004d\") " pod="openshift-ingress/e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg"
Sep 19 04:37:50.557865 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:50.557832    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03b3cd51-7a8f-4708-bd12-e9043fc4004d-credential-socket\") pod \"e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg\" (UID: \"03b3cd51-7a8f-4708-bd12-e9043fc4004d\") " pod="openshift-ingress/e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg"
Sep 19 04:37:50.558063 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:50.557892    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03b3cd51-7a8f-4708-bd12-e9043fc4004d-istio-podinfo\") pod \"e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg\" (UID: \"03b3cd51-7a8f-4708-bd12-e9043fc4004d\") " pod="openshift-ingress/e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg"
Sep 19 04:37:50.558063 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:50.557912    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03b3cd51-7a8f-4708-bd12-e9043fc4004d-istio-data\") pod \"e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg\" (UID: \"03b3cd51-7a8f-4708-bd12-e9043fc4004d\") " pod="openshift-ingress/e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg"
Sep 19 04:37:50.558063 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:50.557936    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03b3cd51-7a8f-4708-bd12-e9043fc4004d-istio-token\") pod \"e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg\" (UID: \"03b3cd51-7a8f-4708-bd12-e9043fc4004d\") " pod="openshift-ingress/e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg"
Sep 19 04:37:50.558063 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:50.557952    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhq6g\" (UniqueName: \"kubernetes.io/projected/03b3cd51-7a8f-4708-bd12-e9043fc4004d-kube-api-access-rhq6g\") pod \"e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg\" (UID: \"03b3cd51-7a8f-4708-bd12-e9043fc4004d\") " pod="openshift-ingress/e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg"
Sep 19 04:37:50.558063 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:50.557972    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03b3cd51-7a8f-4708-bd12-e9043fc4004d-workload-socket\") pod \"e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg\" (UID: \"03b3cd51-7a8f-4708-bd12-e9043fc4004d\") " pod="openshift-ingress/e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg"
Sep 19 04:37:50.558063 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:50.557991    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03b3cd51-7a8f-4708-bd12-e9043fc4004d-workload-certs\") pod \"e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg\" (UID: \"03b3cd51-7a8f-4708-bd12-e9043fc4004d\") " pod="openshift-ingress/e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg"
Sep 19 04:37:50.558063 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:50.558032    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03b3cd51-7a8f-4708-bd12-e9043fc4004d-istiod-ca-cert\") pod \"e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg\" (UID: \"03b3cd51-7a8f-4708-bd12-e9043fc4004d\") " pod="openshift-ingress/e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg"
Sep 19 04:37:50.558063 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:50.558057    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03b3cd51-7a8f-4708-bd12-e9043fc4004d-istio-envoy\") pod \"e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg\" (UID: \"03b3cd51-7a8f-4708-bd12-e9043fc4004d\") " pod="openshift-ingress/e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg"
Sep 19 04:37:50.558501 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:50.558239    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03b3cd51-7a8f-4708-bd12-e9043fc4004d-credential-socket\") pod \"e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg\" (UID: \"03b3cd51-7a8f-4708-bd12-e9043fc4004d\") " pod="openshift-ingress/e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg"
Sep 19 04:37:50.558501 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:50.558301    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03b3cd51-7a8f-4708-bd12-e9043fc4004d-istio-data\") pod \"e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg\" (UID: \"03b3cd51-7a8f-4708-bd12-e9043fc4004d\") " pod="openshift-ingress/e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg"
Sep 19 04:37:50.558501 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:50.558444    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03b3cd51-7a8f-4708-bd12-e9043fc4004d-workload-certs\") pod \"e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg\" (UID: \"03b3cd51-7a8f-4708-bd12-e9043fc4004d\") " pod="openshift-ingress/e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg"
Sep 19 04:37:50.558501 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:50.558485    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03b3cd51-7a8f-4708-bd12-e9043fc4004d-workload-socket\") pod \"e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg\" (UID: \"03b3cd51-7a8f-4708-bd12-e9043fc4004d\") " pod="openshift-ingress/e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg"
Sep 19 04:37:50.558944 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:50.558923    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03b3cd51-7a8f-4708-bd12-e9043fc4004d-istiod-ca-cert\") pod \"e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg\" (UID: \"03b3cd51-7a8f-4708-bd12-e9043fc4004d\") " pod="openshift-ingress/e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg"
Sep 19 04:37:50.560484 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:50.560465    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03b3cd51-7a8f-4708-bd12-e9043fc4004d-istio-envoy\") pod \"e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg\" (UID: \"03b3cd51-7a8f-4708-bd12-e9043fc4004d\") " pod="openshift-ingress/e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg"
Sep 19 04:37:50.560601 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:50.560585    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03b3cd51-7a8f-4708-bd12-e9043fc4004d-istio-podinfo\") pod \"e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg\" (UID: \"03b3cd51-7a8f-4708-bd12-e9043fc4004d\") " pod="openshift-ingress/e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg"
Sep 19 04:37:50.567094 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:50.567072    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03b3cd51-7a8f-4708-bd12-e9043fc4004d-istio-token\") pod \"e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg\" (UID: \"03b3cd51-7a8f-4708-bd12-e9043fc4004d\") " pod="openshift-ingress/e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg"
Sep 19 04:37:50.567162 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:50.567078    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhq6g\" (UniqueName: \"kubernetes.io/projected/03b3cd51-7a8f-4708-bd12-e9043fc4004d-kube-api-access-rhq6g\") pod \"e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg\" (UID: \"03b3cd51-7a8f-4708-bd12-e9043fc4004d\") " pod="openshift-ingress/e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg"
Sep 19 04:37:50.646876 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:50.646789    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg"
Sep 19 04:37:50.936778 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:50.936752    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg"]
Sep 19 04:37:50.939480 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:37:50.939456    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03b3cd51_7a8f_4708_bd12_e9043fc4004d.slice/crio-b329b955c9f8beacab2221685fe07935e44e2936b5c1311a0637198d29fd9f92 WatchSource:0}: Error finding container b329b955c9f8beacab2221685fe07935e44e2936b5c1311a0637198d29fd9f92: Status 404 returned error can't find the container with id b329b955c9f8beacab2221685fe07935e44e2936b5c1311a0637198d29fd9f92
Sep 19 04:37:50.942162 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:50.942126    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 04:37:50.942268 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:50.942191    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 04:37:50.942268 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:50.942219    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 04:37:51.949710 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:51.949669    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg" event={"ID":"03b3cd51-7a8f-4708-bd12-e9043fc4004d","Type":"ContainerStarted","Data":"545d28b6ce34e06d26a419952295fc347940da866eae6210455f31f20777bc82"}
Sep 19 04:37:51.950102 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:51.949743    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg" event={"ID":"03b3cd51-7a8f-4708-bd12-e9043fc4004d","Type":"ContainerStarted","Data":"b329b955c9f8beacab2221685fe07935e44e2936b5c1311a0637198d29fd9f92"}
Sep 19 04:37:51.977231 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:51.977174    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg" podStartSLOduration=1.9771584070000001 podStartE2EDuration="1.977158407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 04:37:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:37:51.976325416 +0000 UTC m=+1451.159286564" watchObservedRunningTime="2026-09-19 04:37:51.977158407 +0000 UTC m=+1451.160119555"
Sep 19 04:37:52.647266 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:52.647235    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg"
Sep 19 04:37:52.652195 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:52.652170    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg"
Sep 19 04:37:52.955946 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:52.955917    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg"
Sep 19 04:37:52.957070 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:37:52.957049    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg"
Sep 19 04:38:20.127457 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:38:20.127427    2579 generic.go:356] "Generic (PLEG): container finished" podID="03b3cd51-7a8f-4708-bd12-e9043fc4004d" containerID="545d28b6ce34e06d26a419952295fc347940da866eae6210455f31f20777bc82" exitCode=0
Sep 19 04:38:20.127956 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:38:20.127506    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg" event={"ID":"03b3cd51-7a8f-4708-bd12-e9043fc4004d","Type":"ContainerDied","Data":"545d28b6ce34e06d26a419952295fc347940da866eae6210455f31f20777bc82"}
Sep 19 04:38:20.127956 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:38:20.127947    2579 scope.go:117] "RemoveContainer" containerID="545d28b6ce34e06d26a419952295fc347940da866eae6210455f31f20777bc82"
Sep 19 04:38:20.129230 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:38:20.129196    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 04:38:20.129296 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:38:20.129276    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 04:38:20.129332 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:38:20.129306    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 04:38:21.135262 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:38:21.135227    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg" event={"ID":"03b3cd51-7a8f-4708-bd12-e9043fc4004d","Type":"ContainerStarted","Data":"d3563f1821ab8010a85191efc339ed2c1550e2d9dba19ad06b1276fac8fd207e"}
Sep 19 04:38:21.647197 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:38:21.647167    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg"
Sep 19 04:38:21.648230 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:38:21.648204    2579 patch_prober.go:28] interesting pod/e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Sep 19 04:38:21.648334 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:38:21.648270    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg" podUID="03b3cd51-7a8f-4708-bd12-e9043fc4004d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Sep 19 04:38:22.647430 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:38:22.647398    2579 patch_prober.go:28] interesting pod/e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Sep 19 04:38:22.647888 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:38:22.647455    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg" podUID="03b3cd51-7a8f-4708-bd12-e9043fc4004d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Sep 19 04:38:22.956279 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:38:22.956240    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg"
Sep 19 04:38:23.647994 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:38:23.647961    2579 patch_prober.go:28] interesting pod/e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Sep 19 04:38:23.648389 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:38:23.648033    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg" podUID="03b3cd51-7a8f-4708-bd12-e9043fc4004d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Sep 19 04:38:24.647094 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:38:24.647059    2579 patch_prober.go:28] interesting pod/e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Sep 19 04:38:24.647270 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:38:24.647128    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg" podUID="03b3cd51-7a8f-4708-bd12-e9043fc4004d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Sep 19 04:38:25.647169 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:38:25.647128    2579 patch_prober.go:28] interesting pod/e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Sep 19 04:38:25.647561 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:38:25.647194    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg" podUID="03b3cd51-7a8f-4708-bd12-e9043fc4004d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Sep 19 04:38:26.647773 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:38:26.647735    2579 patch_prober.go:28] interesting pod/e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Sep 19 04:38:26.648169 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:38:26.647790    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg" podUID="03b3cd51-7a8f-4708-bd12-e9043fc4004d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Sep 19 04:38:27.651088 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:38:27.651058    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg"
Sep 19 04:38:27.652074 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:38:27.652055    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg"
Sep 19 04:38:36.130619 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:38:36.130584    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0a4ccac4-67c67fdd57-7w6k9"]
Sep 19 04:38:36.131096 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:38:36.131035    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6239a719-1815-4ef5-8b52-a488185c44a7" containerName="istio-proxy"
Sep 19 04:38:36.131096 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:38:36.131047    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6239a719-1815-4ef5-8b52-a488185c44a7" containerName="istio-proxy"
Sep 19 04:38:36.135319 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:38:36.135302    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-0a4ccac4-67c67fdd57-7w6k9"
Sep 19 04:38:36.138362 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:38:36.138340    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-0a4ccac4\""
Sep 19 04:38:36.149370 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:38:36.149342    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0a4ccac4-67c67fdd57-7w6k9"]
Sep 19 04:38:36.230020 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:38:36.229982    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/334589ce-ee69-4b3c-b669-8f7fb92bfca6-plugins-config-volume\") pod \"payload-processing-e2e-mt-0a4ccac4-67c67fdd57-7w6k9\" (UID: \"334589ce-ee69-4b3c-b669-8f7fb92bfca6\") " pod="openshift-ingress/payload-processing-e2e-mt-0a4ccac4-67c67fdd57-7w6k9"
Sep 19 04:38:36.230222 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:38:36.230047    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm6n6\" (UniqueName: \"kubernetes.io/projected/334589ce-ee69-4b3c-b669-8f7fb92bfca6-kube-api-access-nm6n6\") pod \"payload-processing-e2e-mt-0a4ccac4-67c67fdd57-7w6k9\" (UID: \"334589ce-ee69-4b3c-b669-8f7fb92bfca6\") " pod="openshift-ingress/payload-processing-e2e-mt-0a4ccac4-67c67fdd57-7w6k9"
Sep 19 04:38:36.331188 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:38:36.331152    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/334589ce-ee69-4b3c-b669-8f7fb92bfca6-plugins-config-volume\") pod \"payload-processing-e2e-mt-0a4ccac4-67c67fdd57-7w6k9\" (UID: \"334589ce-ee69-4b3c-b669-8f7fb92bfca6\") " pod="openshift-ingress/payload-processing-e2e-mt-0a4ccac4-67c67fdd57-7w6k9"
Sep 19 04:38:36.331382 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:38:36.331206    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm6n6\" (UniqueName: \"kubernetes.io/projected/334589ce-ee69-4b3c-b669-8f7fb92bfca6-kube-api-access-nm6n6\") pod \"payload-processing-e2e-mt-0a4ccac4-67c67fdd57-7w6k9\" (UID: \"334589ce-ee69-4b3c-b669-8f7fb92bfca6\") " pod="openshift-ingress/payload-processing-e2e-mt-0a4ccac4-67c67fdd57-7w6k9"
Sep 19 04:38:36.331914 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:38:36.331894    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/334589ce-ee69-4b3c-b669-8f7fb92bfca6-plugins-config-volume\") pod \"payload-processing-e2e-mt-0a4ccac4-67c67fdd57-7w6k9\" (UID: \"334589ce-ee69-4b3c-b669-8f7fb92bfca6\") " pod="openshift-ingress/payload-processing-e2e-mt-0a4ccac4-67c67fdd57-7w6k9"
Sep 19 04:38:36.345041 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:38:36.345011    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm6n6\" (UniqueName: \"kubernetes.io/projected/334589ce-ee69-4b3c-b669-8f7fb92bfca6-kube-api-access-nm6n6\") pod \"payload-processing-e2e-mt-0a4ccac4-67c67fdd57-7w6k9\" (UID: \"334589ce-ee69-4b3c-b669-8f7fb92bfca6\") " pod="openshift-ingress/payload-processing-e2e-mt-0a4ccac4-67c67fdd57-7w6k9"
Sep 19 04:38:36.445044 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:38:36.445008    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-0a4ccac4-67c67fdd57-7w6k9"
Sep 19 04:38:36.734347 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:38:36.734314    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0a4ccac4-67c67fdd57-7w6k9"]
Sep 19 04:38:36.735331 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:38:36.735307    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod334589ce_ee69_4b3c_b669_8f7fb92bfca6.slice/crio-09a76f4b70c8570a09955e7d7647e6240648dee11606de37b7d0c7480647280e WatchSource:0}: Error finding container 09a76f4b70c8570a09955e7d7647e6240648dee11606de37b7d0c7480647280e: Status 404 returned error can't find the container with id 09a76f4b70c8570a09955e7d7647e6240648dee11606de37b7d0c7480647280e
Sep 19 04:38:37.236269 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:38:37.236231    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-0a4ccac4-67c67fdd57-7w6k9" event={"ID":"334589ce-ee69-4b3c-b669-8f7fb92bfca6","Type":"ContainerStarted","Data":"ee9f9f3a2bba3cda678b6554e627aebe5df26953b535e4406c40af6c30d7e9bd"}
Sep 19 04:38:37.236646 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:38:37.236408    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-0a4ccac4-67c67fdd57-7w6k9"
Sep 19 04:38:37.236646 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:38:37.236455    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-0a4ccac4-67c67fdd57-7w6k9" event={"ID":"334589ce-ee69-4b3c-b669-8f7fb92bfca6","Type":"ContainerStarted","Data":"09a76f4b70c8570a09955e7d7647e6240648dee11606de37b7d0c7480647280e"}
Sep 19 04:38:48.244780 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:38:48.244749    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-0a4ccac4-67c67fdd57-7w6k9"
Sep 19 04:38:48.266113 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:38:48.266057    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-0a4ccac4-67c67fdd57-7w6k9" podStartSLOduration=12.266037827 podStartE2EDuration="12.266037827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 04:38:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:38:37.260227495 +0000 UTC m=+1496.443188644" watchObservedRunningTime="2026-09-19 04:38:48.266037827 +0000 UTC m=+1507.448998974"
Sep 19 04:39:08.711310 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:08.711230    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0a4ccac4-j7s9b"]
Sep 19 04:39:08.714909 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:08.714892    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0a4ccac4-j7s9b"
Sep 19 04:39:08.717864 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:08.717843    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s6xbq\""
Sep 19 04:39:08.719169 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:08.719148    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 19 04:39:08.725757 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:08.725736    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0a4ccac4-j7s9b"]
Sep 19 04:39:08.749574 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:08.749538    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b8ac75d-03b6-4799-9b6a-6efdb00bf9be-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-0a4ccac4-j7s9b\" (UID: \"4b8ac75d-03b6-4799-9b6a-6efdb00bf9be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0a4ccac4-j7s9b"
Sep 19 04:39:08.850626 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:08.850576    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b8ac75d-03b6-4799-9b6a-6efdb00bf9be-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-0a4ccac4-j7s9b\" (UID: \"4b8ac75d-03b6-4799-9b6a-6efdb00bf9be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0a4ccac4-j7s9b"
Sep 19 04:39:08.851232 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:08.851209    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b8ac75d-03b6-4799-9b6a-6efdb00bf9be-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-0a4ccac4-j7s9b\" (UID: \"4b8ac75d-03b6-4799-9b6a-6efdb00bf9be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0a4ccac4-j7s9b"
Sep 19 04:39:08.892465 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:08.892427    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-87e3e6/premium-w0-87e3e6-kserve-f48dd7f99-dsx42"]
Sep 19 04:39:08.895981 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:08.895962    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-87e3e6/premium-w0-87e3e6-kserve-f48dd7f99-dsx42"
Sep 19 04:39:08.900393 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:08.900370    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-87e3e6\"/\"kube-root-ca.crt\""
Sep 19 04:39:08.900829 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:08.900812    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-87e3e6\"/\"premium-w0-87e3e6-kserve-self-signed-certs\""
Sep 19 04:39:08.901381 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:08.901360    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-87e3e6\"/\"default-dockercfg-5pqd7\""
Sep 19 04:39:08.901489 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:08.901385    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-87e3e6\"/\"openshift-service-ca.crt\""
Sep 19 04:39:08.909958 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:08.909932    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-87e3e6/premium-w0-87e3e6-kserve-f48dd7f99-dsx42"]
Sep 19 04:39:08.951955 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:08.951922    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ca3da4a-b847-47c4-a1cf-32e7f8643cea-home\") pod \"premium-w0-87e3e6-kserve-f48dd7f99-dsx42\" (UID: \"4ca3da4a-b847-47c4-a1cf-32e7f8643cea\") " pod="e2e-models-e2e-worker-w0-87e3e6/premium-w0-87e3e6-kserve-f48dd7f99-dsx42"
Sep 19 04:39:08.951955 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:08.951957    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ca3da4a-b847-47c4-a1cf-32e7f8643cea-model-cache\") pod \"premium-w0-87e3e6-kserve-f48dd7f99-dsx42\" (UID: \"4ca3da4a-b847-47c4-a1cf-32e7f8643cea\") " pod="e2e-models-e2e-worker-w0-87e3e6/premium-w0-87e3e6-kserve-f48dd7f99-dsx42"
Sep 19 04:39:08.952220 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:08.951976    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ca3da4a-b847-47c4-a1cf-32e7f8643cea-tmp-dir\") pod \"premium-w0-87e3e6-kserve-f48dd7f99-dsx42\" (UID: \"4ca3da4a-b847-47c4-a1cf-32e7f8643cea\") " pod="e2e-models-e2e-worker-w0-87e3e6/premium-w0-87e3e6-kserve-f48dd7f99-dsx42"
Sep 19 04:39:08.952220 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:08.952043    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ca3da4a-b847-47c4-a1cf-32e7f8643cea-dshm\") pod \"premium-w0-87e3e6-kserve-f48dd7f99-dsx42\" (UID: \"4ca3da4a-b847-47c4-a1cf-32e7f8643cea\") " pod="e2e-models-e2e-worker-w0-87e3e6/premium-w0-87e3e6-kserve-f48dd7f99-dsx42"
Sep 19 04:39:08.952220 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:08.952139    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ca3da4a-b847-47c4-a1cf-32e7f8643cea-tls-certs\") pod \"premium-w0-87e3e6-kserve-f48dd7f99-dsx42\" (UID: \"4ca3da4a-b847-47c4-a1cf-32e7f8643cea\") " pod="e2e-models-e2e-worker-w0-87e3e6/premium-w0-87e3e6-kserve-f48dd7f99-dsx42"
Sep 19 04:39:08.952358 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:08.952230    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8586w\" (UniqueName: \"kubernetes.io/projected/4ca3da4a-b847-47c4-a1cf-32e7f8643cea-kube-api-access-8586w\") pod \"premium-w0-87e3e6-kserve-f48dd7f99-dsx42\" (UID: \"4ca3da4a-b847-47c4-a1cf-32e7f8643cea\") " pod="e2e-models-e2e-worker-w0-87e3e6/premium-w0-87e3e6-kserve-f48dd7f99-dsx42"
Sep 19 04:39:09.025916 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:09.025836    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0a4ccac4-j7s9b"
Sep 19 04:39:09.053496 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:09.053466    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ca3da4a-b847-47c4-a1cf-32e7f8643cea-home\") pod \"premium-w0-87e3e6-kserve-f48dd7f99-dsx42\" (UID: \"4ca3da4a-b847-47c4-a1cf-32e7f8643cea\") " pod="e2e-models-e2e-worker-w0-87e3e6/premium-w0-87e3e6-kserve-f48dd7f99-dsx42"
Sep 19 04:39:09.053639 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:09.053505    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ca3da4a-b847-47c4-a1cf-32e7f8643cea-model-cache\") pod \"premium-w0-87e3e6-kserve-f48dd7f99-dsx42\" (UID: \"4ca3da4a-b847-47c4-a1cf-32e7f8643cea\") " pod="e2e-models-e2e-worker-w0-87e3e6/premium-w0-87e3e6-kserve-f48dd7f99-dsx42"
Sep 19 04:39:09.053639 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:09.053533    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ca3da4a-b847-47c4-a1cf-32e7f8643cea-tmp-dir\") pod \"premium-w0-87e3e6-kserve-f48dd7f99-dsx42\" (UID: \"4ca3da4a-b847-47c4-a1cf-32e7f8643cea\") " pod="e2e-models-e2e-worker-w0-87e3e6/premium-w0-87e3e6-kserve-f48dd7f99-dsx42"
Sep 19 04:39:09.053639 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:09.053579    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ca3da4a-b847-47c4-a1cf-32e7f8643cea-dshm\") pod \"premium-w0-87e3e6-kserve-f48dd7f99-dsx42\" (UID: \"4ca3da4a-b847-47c4-a1cf-32e7f8643cea\") " pod="e2e-models-e2e-worker-w0-87e3e6/premium-w0-87e3e6-kserve-f48dd7f99-dsx42"
Sep 19 04:39:09.053639 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:09.053637    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ca3da4a-b847-47c4-a1cf-32e7f8643cea-tls-certs\") pod \"premium-w0-87e3e6-kserve-f48dd7f99-dsx42\" (UID: \"4ca3da4a-b847-47c4-a1cf-32e7f8643cea\") " pod="e2e-models-e2e-worker-w0-87e3e6/premium-w0-87e3e6-kserve-f48dd7f99-dsx42"
Sep 19 04:39:09.053906 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:09.053739    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8586w\" (UniqueName: \"kubernetes.io/projected/4ca3da4a-b847-47c4-a1cf-32e7f8643cea-kube-api-access-8586w\") pod \"premium-w0-87e3e6-kserve-f48dd7f99-dsx42\" (UID: \"4ca3da4a-b847-47c4-a1cf-32e7f8643cea\") " pod="e2e-models-e2e-worker-w0-87e3e6/premium-w0-87e3e6-kserve-f48dd7f99-dsx42"
Sep 19 04:39:09.053906 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:09.053862    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ca3da4a-b847-47c4-a1cf-32e7f8643cea-home\") pod \"premium-w0-87e3e6-kserve-f48dd7f99-dsx42\" (UID: \"4ca3da4a-b847-47c4-a1cf-32e7f8643cea\") " pod="e2e-models-e2e-worker-w0-87e3e6/premium-w0-87e3e6-kserve-f48dd7f99-dsx42"
Sep 19 04:39:09.054012 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:09.053925    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ca3da4a-b847-47c4-a1cf-32e7f8643cea-model-cache\") pod \"premium-w0-87e3e6-kserve-f48dd7f99-dsx42\" (UID: \"4ca3da4a-b847-47c4-a1cf-32e7f8643cea\") " pod="e2e-models-e2e-worker-w0-87e3e6/premium-w0-87e3e6-kserve-f48dd7f99-dsx42"
Sep 19 04:39:09.054319 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:09.054297    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ca3da4a-b847-47c4-a1cf-32e7f8643cea-tmp-dir\") pod \"premium-w0-87e3e6-kserve-f48dd7f99-dsx42\" (UID: \"4ca3da4a-b847-47c4-a1cf-32e7f8643cea\") " pod="e2e-models-e2e-worker-w0-87e3e6/premium-w0-87e3e6-kserve-f48dd7f99-dsx42"
Sep 19 04:39:09.055994 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:09.055973    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ca3da4a-b847-47c4-a1cf-32e7f8643cea-dshm\") pod \"premium-w0-87e3e6-kserve-f48dd7f99-dsx42\" (UID: \"4ca3da4a-b847-47c4-a1cf-32e7f8643cea\") " pod="e2e-models-e2e-worker-w0-87e3e6/premium-w0-87e3e6-kserve-f48dd7f99-dsx42"
Sep 19 04:39:09.056331 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:09.056313    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ca3da4a-b847-47c4-a1cf-32e7f8643cea-tls-certs\") pod \"premium-w0-87e3e6-kserve-f48dd7f99-dsx42\" (UID: \"4ca3da4a-b847-47c4-a1cf-32e7f8643cea\") " pod="e2e-models-e2e-worker-w0-87e3e6/premium-w0-87e3e6-kserve-f48dd7f99-dsx42"
Sep 19 04:39:09.063092 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:09.063059    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8586w\" (UniqueName: \"kubernetes.io/projected/4ca3da4a-b847-47c4-a1cf-32e7f8643cea-kube-api-access-8586w\") pod \"premium-w0-87e3e6-kserve-f48dd7f99-dsx42\" (UID: \"4ca3da4a-b847-47c4-a1cf-32e7f8643cea\") " pod="e2e-models-e2e-worker-w0-87e3e6/premium-w0-87e3e6-kserve-f48dd7f99-dsx42"
Sep 19 04:39:09.205760 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:09.205689    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-87e3e6/premium-w0-87e3e6-kserve-f48dd7f99-dsx42"
Sep 19 04:39:09.308443 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:09.308282    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0a4ccac4-j7s9b"]
Sep 19 04:39:09.313122 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:39:09.313003    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b8ac75d_03b6_4799_9b6a_6efdb00bf9be.slice/crio-1f96cf502a5599e4979fbde6e614213d4891dfc65599d74621bb3971ca16ebac WatchSource:0}: Error finding container 1f96cf502a5599e4979fbde6e614213d4891dfc65599d74621bb3971ca16ebac: Status 404 returned error can't find the container with id 1f96cf502a5599e4979fbde6e614213d4891dfc65599d74621bb3971ca16ebac
Sep 19 04:39:09.429867 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:09.429827    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0a4ccac4-j7s9b" event={"ID":"4b8ac75d-03b6-4799-9b6a-6efdb00bf9be","Type":"ContainerStarted","Data":"1f96cf502a5599e4979fbde6e614213d4891dfc65599d74621bb3971ca16ebac"}
Sep 19 04:39:09.491887 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:09.491861    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-87e3e6/premium-w0-87e3e6-kserve-f48dd7f99-dsx42"]
Sep 19 04:39:09.493806 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:39:09.493782    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4ca3da4a_b847_47c4_a1cf_32e7f8643cea.slice/crio-c2e152812ac45758991e496ca431075a1675f6bf9931eb83edfb1579b76069ca WatchSource:0}: Error finding container c2e152812ac45758991e496ca431075a1675f6bf9931eb83edfb1579b76069ca: Status 404 returned error can't find the container with id c2e152812ac45758991e496ca431075a1675f6bf9931eb83edfb1579b76069ca
Sep 19 04:39:10.437394 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:10.437359    2579 generic.go:356] "Generic (PLEG): container finished" podID="4b8ac75d-03b6-4799-9b6a-6efdb00bf9be" containerID="264ebccdcb0cf39fe6d8b74044b8be54cdd1bb76b2239ded09a5441371cb11d1" exitCode=0
Sep 19 04:39:10.437882 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:10.437448    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0a4ccac4-j7s9b" event={"ID":"4b8ac75d-03b6-4799-9b6a-6efdb00bf9be","Type":"ContainerDied","Data":"264ebccdcb0cf39fe6d8b74044b8be54cdd1bb76b2239ded09a5441371cb11d1"}
Sep 19 04:39:10.439080 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:10.439057    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-87e3e6/premium-w0-87e3e6-kserve-f48dd7f99-dsx42" event={"ID":"4ca3da4a-b847-47c4-a1cf-32e7f8643cea","Type":"ContainerStarted","Data":"d781ac527d13a23ba126249cdc8bc8cb6540d4e528706e7ae8188ccfe9840c65"}
Sep 19 04:39:10.439194 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:10.439087    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-87e3e6/premium-w0-87e3e6-kserve-f48dd7f99-dsx42" event={"ID":"4ca3da4a-b847-47c4-a1cf-32e7f8643cea","Type":"ContainerStarted","Data":"c2e152812ac45758991e496ca431075a1675f6bf9931eb83edfb1579b76069ca"}
Sep 19 04:39:10.478565 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:10.478520    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-87e3e6/premium-w0-87e3e6-kserve-f48dd7f99-dsx42" podStartSLOduration=2.478505106 podStartE2EDuration="2.478505106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 04:39:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:39:10.477092532 +0000 UTC m=+1529.660053679" watchObservedRunningTime="2026-09-19 04:39:10.478505106 +0000 UTC m=+1529.661466254"
Sep 19 04:39:11.205924 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:11.205890    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-87e3e6/premium-w0-87e3e6-kserve-f48dd7f99-dsx42"
Sep 19 04:39:11.591291 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:11.591267    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0a4ccac4-j7s9b"
Sep 19 04:39:11.679933 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:11.679881    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b8ac75d-03b6-4799-9b6a-6efdb00bf9be-maas-api-service-ca\") pod \"4b8ac75d-03b6-4799-9b6a-6efdb00bf9be\" (UID: \"4b8ac75d-03b6-4799-9b6a-6efdb00bf9be\") "
Sep 19 04:39:11.680244 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:11.680218    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b8ac75d-03b6-4799-9b6a-6efdb00bf9be-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4b8ac75d-03b6-4799-9b6a-6efdb00bf9be" (UID: "4b8ac75d-03b6-4799-9b6a-6efdb00bf9be"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 04:39:11.780999 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:11.780888    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b8ac75d-03b6-4799-9b6a-6efdb00bf9be-maas-api-service-ca\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:39:12.457278 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:12.457244    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0a4ccac4-j7s9b" event={"ID":"4b8ac75d-03b6-4799-9b6a-6efdb00bf9be","Type":"ContainerDied","Data":"1f96cf502a5599e4979fbde6e614213d4891dfc65599d74621bb3971ca16ebac"}
Sep 19 04:39:12.457278 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:12.457272    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0a4ccac4-j7s9b"
Sep 19 04:39:12.457278 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:12.457296    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f96cf502a5599e4979fbde6e614213d4891dfc65599d74621bb3971ca16ebac"
Sep 19 04:39:19.206612 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:19.206579    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-87e3e6/premium-w0-87e3e6-kserve-f48dd7f99-dsx42"
Sep 19 04:39:19.219919 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:19.219888    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-87e3e6/premium-w0-87e3e6-kserve-f48dd7f99-dsx42"
Sep 19 04:39:19.517471 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:19.517389    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-87e3e6/premium-w0-87e3e6-kserve-f48dd7f99-dsx42"
Sep 19 04:39:19.907959 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:19.907931    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0a4ccac4-67c67fdd57-7w6k9"]
Sep 19 04:39:19.908196 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:19.908173    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-0a4ccac4-67c67fdd57-7w6k9" podUID="334589ce-ee69-4b3c-b669-8f7fb92bfca6" containerName="payload-processing" containerID="cri-o://ee9f9f3a2bba3cda678b6554e627aebe5df26953b535e4406c40af6c30d7e9bd" gracePeriod=30
Sep 19 04:39:20.186114 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:20.186080    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-0a4ccac4-67c67fdd57-7w6k9"
Sep 19 04:39:20.257658 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:20.257625    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0a4ccac4-j7s9b"]
Sep 19 04:39:20.265113 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:20.265085    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0a4ccac4-j7s9b"]
Sep 19 04:39:20.270417 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:20.270397    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/334589ce-ee69-4b3c-b669-8f7fb92bfca6-plugins-config-volume\") pod \"334589ce-ee69-4b3c-b669-8f7fb92bfca6\" (UID: \"334589ce-ee69-4b3c-b669-8f7fb92bfca6\") "
Sep 19 04:39:20.270508 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:20.270495    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nm6n6\" (UniqueName: \"kubernetes.io/projected/334589ce-ee69-4b3c-b669-8f7fb92bfca6-kube-api-access-nm6n6\") pod \"334589ce-ee69-4b3c-b669-8f7fb92bfca6\" (UID: \"334589ce-ee69-4b3c-b669-8f7fb92bfca6\") "
Sep 19 04:39:20.270766 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:20.270743    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/334589ce-ee69-4b3c-b669-8f7fb92bfca6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "334589ce-ee69-4b3c-b669-8f7fb92bfca6" (UID: "334589ce-ee69-4b3c-b669-8f7fb92bfca6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 04:39:20.272714 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:20.272675    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/334589ce-ee69-4b3c-b669-8f7fb92bfca6-kube-api-access-nm6n6" (OuterVolumeSpecName: "kube-api-access-nm6n6") pod "334589ce-ee69-4b3c-b669-8f7fb92bfca6" (UID: "334589ce-ee69-4b3c-b669-8f7fb92bfca6"). InnerVolumeSpecName "kube-api-access-nm6n6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 04:39:20.372201 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:20.372165    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/334589ce-ee69-4b3c-b669-8f7fb92bfca6-plugins-config-volume\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:39:20.372201 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:20.372192    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nm6n6\" (UniqueName: \"kubernetes.io/projected/334589ce-ee69-4b3c-b669-8f7fb92bfca6-kube-api-access-nm6n6\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:39:20.516198 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:20.516101    2579 generic.go:356] "Generic (PLEG): container finished" podID="334589ce-ee69-4b3c-b669-8f7fb92bfca6" containerID="ee9f9f3a2bba3cda678b6554e627aebe5df26953b535e4406c40af6c30d7e9bd" exitCode=0
Sep 19 04:39:20.516198 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:20.516152    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-0a4ccac4-67c67fdd57-7w6k9"
Sep 19 04:39:20.516198 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:20.516175    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-0a4ccac4-67c67fdd57-7w6k9" event={"ID":"334589ce-ee69-4b3c-b669-8f7fb92bfca6","Type":"ContainerDied","Data":"ee9f9f3a2bba3cda678b6554e627aebe5df26953b535e4406c40af6c30d7e9bd"}
Sep 19 04:39:20.516488 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:20.516236    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-0a4ccac4-67c67fdd57-7w6k9" event={"ID":"334589ce-ee69-4b3c-b669-8f7fb92bfca6","Type":"ContainerDied","Data":"09a76f4b70c8570a09955e7d7647e6240648dee11606de37b7d0c7480647280e"}
Sep 19 04:39:20.516488 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:20.516268    2579 scope.go:117] "RemoveContainer" containerID="ee9f9f3a2bba3cda678b6554e627aebe5df26953b535e4406c40af6c30d7e9bd"
Sep 19 04:39:20.530092 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:20.530070    2579 scope.go:117] "RemoveContainer" containerID="ee9f9f3a2bba3cda678b6554e627aebe5df26953b535e4406c40af6c30d7e9bd"
Sep 19 04:39:20.530415 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:39:20.530395    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee9f9f3a2bba3cda678b6554e627aebe5df26953b535e4406c40af6c30d7e9bd\": container with ID starting with ee9f9f3a2bba3cda678b6554e627aebe5df26953b535e4406c40af6c30d7e9bd not found: ID does not exist" containerID="ee9f9f3a2bba3cda678b6554e627aebe5df26953b535e4406c40af6c30d7e9bd"
Sep 19 04:39:20.530514 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:20.530420    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee9f9f3a2bba3cda678b6554e627aebe5df26953b535e4406c40af6c30d7e9bd"} err="failed to get container status \"ee9f9f3a2bba3cda678b6554e627aebe5df26953b535e4406c40af6c30d7e9bd\": rpc error: code = NotFound desc = could not find container \"ee9f9f3a2bba3cda678b6554e627aebe5df26953b535e4406c40af6c30d7e9bd\": container with ID starting with ee9f9f3a2bba3cda678b6554e627aebe5df26953b535e4406c40af6c30d7e9bd not found: ID does not exist"
Sep 19 04:39:20.544268 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:20.544240    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0a4ccac4-67c67fdd57-7w6k9"]
Sep 19 04:39:20.550657 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:20.550628    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0a4ccac4-67c67fdd57-7w6k9"]
Sep 19 04:39:21.352225 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:21.352188    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="334589ce-ee69-4b3c-b669-8f7fb92bfca6" path="/var/lib/kubelet/pods/334589ce-ee69-4b3c-b669-8f7fb92bfca6/volumes"
Sep 19 04:39:21.352606 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:21.352546    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b8ac75d-03b6-4799-9b6a-6efdb00bf9be" path="/var/lib/kubelet/pods/4b8ac75d-03b6-4799-9b6a-6efdb00bf9be/volumes"
Sep 19 04:39:52.398619 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:52.398584    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c1a7b6-746cd84ccd-kwzs2"]
Sep 19 04:39:52.399224 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:52.399084    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b8ac75d-03b6-4799-9b6a-6efdb00bf9be" containerName="revoke-keys"
Sep 19 04:39:52.399224 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:52.399100    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b8ac75d-03b6-4799-9b6a-6efdb00bf9be" containerName="revoke-keys"
Sep 19 04:39:52.399224 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:52.399113    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="334589ce-ee69-4b3c-b669-8f7fb92bfca6" containerName="payload-processing"
Sep 19 04:39:52.399224 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:52.399118    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="334589ce-ee69-4b3c-b669-8f7fb92bfca6" containerName="payload-processing"
Sep 19 04:39:52.399224 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:52.399184    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="334589ce-ee69-4b3c-b669-8f7fb92bfca6" containerName="payload-processing"
Sep 19 04:39:52.399224 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:52.399195    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b8ac75d-03b6-4799-9b6a-6efdb00bf9be" containerName="revoke-keys"
Sep 19 04:39:52.403789 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:52.403771    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c1a7b6-746cd84ccd-kwzs2"
Sep 19 04:39:52.406462 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:52.406438    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-c1a7b6-dockercfg-snxs6\""
Sep 19 04:39:52.407286 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:52.407264    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-c1a7b6\""
Sep 19 04:39:52.419906 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:52.419880    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c1a7b6-746cd84ccd-kwzs2"]
Sep 19 04:39:52.501138 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:52.501100    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b1c7a21-3050-4d74-8754-9c960a4dee9b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-c1a7b6-746cd84ccd-kwzs2\" (UID: \"9b1c7a21-3050-4d74-8754-9c960a4dee9b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c1a7b6-746cd84ccd-kwzs2"
Sep 19 04:39:52.501492 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:52.501470    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbdxs\" (UniqueName: \"kubernetes.io/projected/9b1c7a21-3050-4d74-8754-9c960a4dee9b-kube-api-access-fbdxs\") pod \"payload-pre-processing-e2e-shared-a-w2-c1a7b6-746cd84ccd-kwzs2\" (UID: \"9b1c7a21-3050-4d74-8754-9c960a4dee9b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c1a7b6-746cd84ccd-kwzs2"
Sep 19 04:39:52.603037 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:52.603001    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b1c7a21-3050-4d74-8754-9c960a4dee9b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-c1a7b6-746cd84ccd-kwzs2\" (UID: \"9b1c7a21-3050-4d74-8754-9c960a4dee9b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c1a7b6-746cd84ccd-kwzs2"
Sep 19 04:39:52.603203 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:52.603077    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbdxs\" (UniqueName: \"kubernetes.io/projected/9b1c7a21-3050-4d74-8754-9c960a4dee9b-kube-api-access-fbdxs\") pod \"payload-pre-processing-e2e-shared-a-w2-c1a7b6-746cd84ccd-kwzs2\" (UID: \"9b1c7a21-3050-4d74-8754-9c960a4dee9b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c1a7b6-746cd84ccd-kwzs2"
Sep 19 04:39:52.603846 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:52.603825    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b1c7a21-3050-4d74-8754-9c960a4dee9b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-c1a7b6-746cd84ccd-kwzs2\" (UID: \"9b1c7a21-3050-4d74-8754-9c960a4dee9b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c1a7b6-746cd84ccd-kwzs2"
Sep 19 04:39:52.612896 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:52.612870    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbdxs\" (UniqueName: \"kubernetes.io/projected/9b1c7a21-3050-4d74-8754-9c960a4dee9b-kube-api-access-fbdxs\") pod \"payload-pre-processing-e2e-shared-a-w2-c1a7b6-746cd84ccd-kwzs2\" (UID: \"9b1c7a21-3050-4d74-8754-9c960a4dee9b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c1a7b6-746cd84ccd-kwzs2"
Sep 19 04:39:52.714890 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:52.714848    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c1a7b6-746cd84ccd-kwzs2"
Sep 19 04:39:53.005930 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:53.004771    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c1a7b6-746cd84ccd-kwzs2"]
Sep 19 04:39:53.733354 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:53.733311    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c1a7b6-746cd84ccd-kwzs2" event={"ID":"9b1c7a21-3050-4d74-8754-9c960a4dee9b","Type":"ContainerStarted","Data":"44e7f419bb5b14ac7c9fa976fdc7f116ace19255c079bfc0ee9bd59823ff7e75"}
Sep 19 04:39:53.733841 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:53.733479    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c1a7b6-746cd84ccd-kwzs2"
Sep 19 04:39:53.733841 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:53.733526    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c1a7b6-746cd84ccd-kwzs2" event={"ID":"9b1c7a21-3050-4d74-8754-9c960a4dee9b","Type":"ContainerStarted","Data":"3ce0b1901e8853a2724b430a25e50a91a0c41944e7721c90e27b0fc08a6eff85"}
Sep 19 04:39:53.754333 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:39:53.754281    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c1a7b6-746cd84ccd-kwzs2" podStartSLOduration=1.754266597 podStartE2EDuration="1.754266597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 04:39:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:39:53.751475413 +0000 UTC m=+1572.934436560" watchObservedRunningTime="2026-09-19 04:39:53.754266597 +0000 UTC m=+1572.937227745"
Sep 19 04:40:04.742572 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:04.742533    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c1a7b6-746cd84ccd-kwzs2"
Sep 19 04:40:05.996611 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:05.996574    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a70050-z92xw"]
Sep 19 04:40:06.001743 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:06.001685    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a70050-z92xw"
Sep 19 04:40:06.011736 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:06.011316    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 19 04:40:06.011736 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:06.011316    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s6xbq\""
Sep 19 04:40:06.020328 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:06.020290    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a70050-z92xw"]
Sep 19 04:40:06.139746 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:06.139714    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3f137d9-4b2c-4436-8a41-5b5ea3e5ed5e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-a70050-z92xw\" (UID: \"d3f137d9-4b2c-4436-8a41-5b5ea3e5ed5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a70050-z92xw"
Sep 19 04:40:06.240802 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:06.240765    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3f137d9-4b2c-4436-8a41-5b5ea3e5ed5e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-a70050-z92xw\" (UID: \"d3f137d9-4b2c-4436-8a41-5b5ea3e5ed5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a70050-z92xw"
Sep 19 04:40:06.241418 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:06.241394    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3f137d9-4b2c-4436-8a41-5b5ea3e5ed5e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-a70050-z92xw\" (UID: \"d3f137d9-4b2c-4436-8a41-5b5ea3e5ed5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a70050-z92xw"
Sep 19 04:40:06.315679 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:06.315591    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a70050-z92xw"
Sep 19 04:40:06.612205 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:06.612182    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a70050-z92xw"]
Sep 19 04:40:06.614798 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:40:06.614763    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3f137d9_4b2c_4436_8a41_5b5ea3e5ed5e.slice/crio-fd1732fcd3280db72dd9e186631b003757c9c2e61bff27df8dd95289f334f3dd WatchSource:0}: Error finding container fd1732fcd3280db72dd9e186631b003757c9c2e61bff27df8dd95289f334f3dd: Status 404 returned error can't find the container with id fd1732fcd3280db72dd9e186631b003757c9c2e61bff27df8dd95289f334f3dd
Sep 19 04:40:06.819585 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:06.819553    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a70050-z92xw" event={"ID":"d3f137d9-4b2c-4436-8a41-5b5ea3e5ed5e","Type":"ContainerStarted","Data":"fd1732fcd3280db72dd9e186631b003757c9c2e61bff27df8dd95289f334f3dd"}
Sep 19 04:40:07.826637 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:07.826597    2579 generic.go:356] "Generic (PLEG): container finished" podID="d3f137d9-4b2c-4436-8a41-5b5ea3e5ed5e" containerID="ec3aeeba1b7cb8917eb8777b0e9255ff2a190eb1befbc7d9a81a7336fb5afe6a" exitCode=0
Sep 19 04:40:07.827076 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:07.826708    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a70050-z92xw" event={"ID":"d3f137d9-4b2c-4436-8a41-5b5ea3e5ed5e","Type":"ContainerDied","Data":"ec3aeeba1b7cb8917eb8777b0e9255ff2a190eb1befbc7d9a81a7336fb5afe6a"}
Sep 19 04:40:08.977661 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:08.977640    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a70050-z92xw"
Sep 19 04:40:09.068797 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:09.068755    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3f137d9-4b2c-4436-8a41-5b5ea3e5ed5e-maas-api-service-ca\") pod \"d3f137d9-4b2c-4436-8a41-5b5ea3e5ed5e\" (UID: \"d3f137d9-4b2c-4436-8a41-5b5ea3e5ed5e\") "
Sep 19 04:40:09.069134 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:09.069111    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3f137d9-4b2c-4436-8a41-5b5ea3e5ed5e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d3f137d9-4b2c-4436-8a41-5b5ea3e5ed5e" (UID: "d3f137d9-4b2c-4436-8a41-5b5ea3e5ed5e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 04:40:09.169743 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:09.169687    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3f137d9-4b2c-4436-8a41-5b5ea3e5ed5e-maas-api-service-ca\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:40:09.842043 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:09.842012    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a70050-z92xw"
Sep 19 04:40:09.842217 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:09.842045    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a70050-z92xw" event={"ID":"d3f137d9-4b2c-4436-8a41-5b5ea3e5ed5e","Type":"ContainerDied","Data":"fd1732fcd3280db72dd9e186631b003757c9c2e61bff27df8dd95289f334f3dd"}
Sep 19 04:40:09.842217 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:09.842093    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd1732fcd3280db72dd9e186631b003757c9c2e61bff27df8dd95289f334f3dd"
Sep 19 04:40:17.764107 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:17.764076    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-eac6c5-openshift-default-5bc95d8f8b-lc62q"]
Sep 19 04:40:17.764547 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:17.764532    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3f137d9-4b2c-4436-8a41-5b5ea3e5ed5e" containerName="revoke-keys"
Sep 19 04:40:17.764596 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:17.764551    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3f137d9-4b2c-4436-8a41-5b5ea3e5ed5e" containerName="revoke-keys"
Sep 19 04:40:17.764665 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:17.764655    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3f137d9-4b2c-4436-8a41-5b5ea3e5ed5e" containerName="revoke-keys"
Sep 19 04:40:17.768022 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:17.767993    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-eac6c5-openshift-default-5bc95d8f8b-lc62q"
Sep 19 04:40:17.771872 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:17.771850    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-eac6c5-openshift-default-dockercfg-scw8h\""
Sep 19 04:40:17.788901 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:17.788853    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-eac6c5-openshift-default-5bc95d8f8b-lc62q"]
Sep 19 04:40:17.959159 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:17.959126    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0c01ced-87c7-4491-aa9c-2eabd3789794-istio-token\") pod \"e2e-shared-b-w2-eac6c5-openshift-default-5bc95d8f8b-lc62q\" (UID: \"a0c01ced-87c7-4491-aa9c-2eabd3789794\") " pod="openshift-ingress/e2e-shared-b-w2-eac6c5-openshift-default-5bc95d8f8b-lc62q"
Sep 19 04:40:17.959159 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:17.959159    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0c01ced-87c7-4491-aa9c-2eabd3789794-istio-data\") pod \"e2e-shared-b-w2-eac6c5-openshift-default-5bc95d8f8b-lc62q\" (UID: \"a0c01ced-87c7-4491-aa9c-2eabd3789794\") " pod="openshift-ingress/e2e-shared-b-w2-eac6c5-openshift-default-5bc95d8f8b-lc62q"
Sep 19 04:40:17.959405 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:17.959184    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0c01ced-87c7-4491-aa9c-2eabd3789794-workload-certs\") pod \"e2e-shared-b-w2-eac6c5-openshift-default-5bc95d8f8b-lc62q\" (UID: \"a0c01ced-87c7-4491-aa9c-2eabd3789794\") " pod="openshift-ingress/e2e-shared-b-w2-eac6c5-openshift-default-5bc95d8f8b-lc62q"
Sep 19 04:40:17.959405 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:17.959299    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0c01ced-87c7-4491-aa9c-2eabd3789794-istiod-ca-cert\") pod \"e2e-shared-b-w2-eac6c5-openshift-default-5bc95d8f8b-lc62q\" (UID: \"a0c01ced-87c7-4491-aa9c-2eabd3789794\") " pod="openshift-ingress/e2e-shared-b-w2-eac6c5-openshift-default-5bc95d8f8b-lc62q"
Sep 19 04:40:17.959405 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:17.959348    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0c01ced-87c7-4491-aa9c-2eabd3789794-istio-envoy\") pod \"e2e-shared-b-w2-eac6c5-openshift-default-5bc95d8f8b-lc62q\" (UID: \"a0c01ced-87c7-4491-aa9c-2eabd3789794\") " pod="openshift-ingress/e2e-shared-b-w2-eac6c5-openshift-default-5bc95d8f8b-lc62q"
Sep 19 04:40:17.959567 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:17.959409    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0c01ced-87c7-4491-aa9c-2eabd3789794-credential-socket\") pod \"e2e-shared-b-w2-eac6c5-openshift-default-5bc95d8f8b-lc62q\" (UID: \"a0c01ced-87c7-4491-aa9c-2eabd3789794\") " pod="openshift-ingress/e2e-shared-b-w2-eac6c5-openshift-default-5bc95d8f8b-lc62q"
Sep 19 04:40:17.959567 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:17.959526    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0c01ced-87c7-4491-aa9c-2eabd3789794-istio-podinfo\") pod \"e2e-shared-b-w2-eac6c5-openshift-default-5bc95d8f8b-lc62q\" (UID: \"a0c01ced-87c7-4491-aa9c-2eabd3789794\") " pod="openshift-ingress/e2e-shared-b-w2-eac6c5-openshift-default-5bc95d8f8b-lc62q"
Sep 19 04:40:17.959567 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:17.959554    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0c01ced-87c7-4491-aa9c-2eabd3789794-workload-socket\") pod \"e2e-shared-b-w2-eac6c5-openshift-default-5bc95d8f8b-lc62q\" (UID: \"a0c01ced-87c7-4491-aa9c-2eabd3789794\") " pod="openshift-ingress/e2e-shared-b-w2-eac6c5-openshift-default-5bc95d8f8b-lc62q"
Sep 19 04:40:17.959740 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:17.959585    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6ncd\" (UniqueName: \"kubernetes.io/projected/a0c01ced-87c7-4491-aa9c-2eabd3789794-kube-api-access-c6ncd\") pod \"e2e-shared-b-w2-eac6c5-openshift-default-5bc95d8f8b-lc62q\" (UID: \"a0c01ced-87c7-4491-aa9c-2eabd3789794\") " pod="openshift-ingress/e2e-shared-b-w2-eac6c5-openshift-default-5bc95d8f8b-lc62q"
Sep 19 04:40:18.061000 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:18.060907    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0c01ced-87c7-4491-aa9c-2eabd3789794-istio-envoy\") pod \"e2e-shared-b-w2-eac6c5-openshift-default-5bc95d8f8b-lc62q\" (UID: \"a0c01ced-87c7-4491-aa9c-2eabd3789794\") " pod="openshift-ingress/e2e-shared-b-w2-eac6c5-openshift-default-5bc95d8f8b-lc62q"
Sep 19 04:40:18.061000 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:18.060945    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0c01ced-87c7-4491-aa9c-2eabd3789794-credential-socket\") pod \"e2e-shared-b-w2-eac6c5-openshift-default-5bc95d8f8b-lc62q\" (UID: \"a0c01ced-87c7-4491-aa9c-2eabd3789794\") " pod="openshift-ingress/e2e-shared-b-w2-eac6c5-openshift-default-5bc95d8f8b-lc62q"
Sep 19 04:40:18.061239 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:18.061009    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0c01ced-87c7-4491-aa9c-2eabd3789794-istio-podinfo\") pod \"e2e-shared-b-w2-eac6c5-openshift-default-5bc95d8f8b-lc62q\" (UID: \"a0c01ced-87c7-4491-aa9c-2eabd3789794\") " pod="openshift-ingress/e2e-shared-b-w2-eac6c5-openshift-default-5bc95d8f8b-lc62q"
Sep 19 04:40:18.061239 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:18.061026    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0c01ced-87c7-4491-aa9c-2eabd3789794-workload-socket\") pod \"e2e-shared-b-w2-eac6c5-openshift-default-5bc95d8f8b-lc62q\" (UID: \"a0c01ced-87c7-4491-aa9c-2eabd3789794\") " pod="openshift-ingress/e2e-shared-b-w2-eac6c5-openshift-default-5bc95d8f8b-lc62q"
Sep 19 04:40:18.061239 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:18.061046    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6ncd\" (UniqueName: \"kubernetes.io/projected/a0c01ced-87c7-4491-aa9c-2eabd3789794-kube-api-access-c6ncd\") pod \"e2e-shared-b-w2-eac6c5-openshift-default-5bc95d8f8b-lc62q\" (UID: \"a0c01ced-87c7-4491-aa9c-2eabd3789794\") " pod="openshift-ingress/e2e-shared-b-w2-eac6c5-openshift-default-5bc95d8f8b-lc62q"
Sep 19 04:40:18.061239 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:18.061088    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0c01ced-87c7-4491-aa9c-2eabd3789794-istio-token\") pod \"e2e-shared-b-w2-eac6c5-openshift-default-5bc95d8f8b-lc62q\" (UID: \"a0c01ced-87c7-4491-aa9c-2eabd3789794\") " pod="openshift-ingress/e2e-shared-b-w2-eac6c5-openshift-default-5bc95d8f8b-lc62q"
Sep 19 04:40:18.061239 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:18.061112    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0c01ced-87c7-4491-aa9c-2eabd3789794-istio-data\") pod \"e2e-shared-b-w2-eac6c5-openshift-default-5bc95d8f8b-lc62q\" (UID: \"a0c01ced-87c7-4491-aa9c-2eabd3789794\") " pod="openshift-ingress/e2e-shared-b-w2-eac6c5-openshift-default-5bc95d8f8b-lc62q"
Sep 19 04:40:18.061239 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:18.061144    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0c01ced-87c7-4491-aa9c-2eabd3789794-workload-certs\") pod \"e2e-shared-b-w2-eac6c5-openshift-default-5bc95d8f8b-lc62q\" (UID: \"a0c01ced-87c7-4491-aa9c-2eabd3789794\") " pod="openshift-ingress/e2e-shared-b-w2-eac6c5-openshift-default-5bc95d8f8b-lc62q"
Sep 19 04:40:18.061239 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:18.061188    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0c01ced-87c7-4491-aa9c-2eabd3789794-istiod-ca-cert\") pod \"e2e-shared-b-w2-eac6c5-openshift-default-5bc95d8f8b-lc62q\" (UID: \"a0c01ced-87c7-4491-aa9c-2eabd3789794\") " pod="openshift-ingress/e2e-shared-b-w2-eac6c5-openshift-default-5bc95d8f8b-lc62q"
Sep 19 04:40:18.061665 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:18.061433    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0c01ced-87c7-4491-aa9c-2eabd3789794-credential-socket\") pod \"e2e-shared-b-w2-eac6c5-openshift-default-5bc95d8f8b-lc62q\" (UID: \"a0c01ced-87c7-4491-aa9c-2eabd3789794\") " pod="openshift-ingress/e2e-shared-b-w2-eac6c5-openshift-default-5bc95d8f8b-lc62q"
Sep 19 04:40:18.061665 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:18.061597    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0c01ced-87c7-4491-aa9c-2eabd3789794-istio-data\") pod \"e2e-shared-b-w2-eac6c5-openshift-default-5bc95d8f8b-lc62q\" (UID: \"a0c01ced-87c7-4491-aa9c-2eabd3789794\") " pod="openshift-ingress/e2e-shared-b-w2-eac6c5-openshift-default-5bc95d8f8b-lc62q"
Sep 19 04:40:18.061665 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:18.061630    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0c01ced-87c7-4491-aa9c-2eabd3789794-workload-certs\") pod \"e2e-shared-b-w2-eac6c5-openshift-default-5bc95d8f8b-lc62q\" (UID: \"a0c01ced-87c7-4491-aa9c-2eabd3789794\") " pod="openshift-ingress/e2e-shared-b-w2-eac6c5-openshift-default-5bc95d8f8b-lc62q"
Sep 19 04:40:18.061846 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:18.061679    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0c01ced-87c7-4491-aa9c-2eabd3789794-workload-socket\") pod \"e2e-shared-b-w2-eac6c5-openshift-default-5bc95d8f8b-lc62q\" (UID: \"a0c01ced-87c7-4491-aa9c-2eabd3789794\") " pod="openshift-ingress/e2e-shared-b-w2-eac6c5-openshift-default-5bc95d8f8b-lc62q"
Sep 19 04:40:18.061900 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:18.061873    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0c01ced-87c7-4491-aa9c-2eabd3789794-istiod-ca-cert\") pod \"e2e-shared-b-w2-eac6c5-openshift-default-5bc95d8f8b-lc62q\" (UID: \"a0c01ced-87c7-4491-aa9c-2eabd3789794\") " pod="openshift-ingress/e2e-shared-b-w2-eac6c5-openshift-default-5bc95d8f8b-lc62q"
Sep 19 04:40:18.064106 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:18.064082    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0c01ced-87c7-4491-aa9c-2eabd3789794-istio-envoy\") pod \"e2e-shared-b-w2-eac6c5-openshift-default-5bc95d8f8b-lc62q\" (UID: \"a0c01ced-87c7-4491-aa9c-2eabd3789794\") " pod="openshift-ingress/e2e-shared-b-w2-eac6c5-openshift-default-5bc95d8f8b-lc62q"
Sep 19 04:40:18.064330 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:18.064310    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0c01ced-87c7-4491-aa9c-2eabd3789794-istio-podinfo\") pod \"e2e-shared-b-w2-eac6c5-openshift-default-5bc95d8f8b-lc62q\" (UID: \"a0c01ced-87c7-4491-aa9c-2eabd3789794\") " pod="openshift-ingress/e2e-shared-b-w2-eac6c5-openshift-default-5bc95d8f8b-lc62q"
Sep 19 04:40:18.072803 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:18.072782    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0c01ced-87c7-4491-aa9c-2eabd3789794-istio-token\") pod \"e2e-shared-b-w2-eac6c5-openshift-default-5bc95d8f8b-lc62q\" (UID: \"a0c01ced-87c7-4491-aa9c-2eabd3789794\") " pod="openshift-ingress/e2e-shared-b-w2-eac6c5-openshift-default-5bc95d8f8b-lc62q"
Sep 19 04:40:18.072989 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:18.072971    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6ncd\" (UniqueName: \"kubernetes.io/projected/a0c01ced-87c7-4491-aa9c-2eabd3789794-kube-api-access-c6ncd\") pod \"e2e-shared-b-w2-eac6c5-openshift-default-5bc95d8f8b-lc62q\" (UID: \"a0c01ced-87c7-4491-aa9c-2eabd3789794\") " pod="openshift-ingress/e2e-shared-b-w2-eac6c5-openshift-default-5bc95d8f8b-lc62q"
Sep 19 04:40:18.077811 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:18.077791    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-eac6c5-openshift-default-5bc95d8f8b-lc62q"
Sep 19 04:40:18.365574 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:18.365551    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-eac6c5-openshift-default-5bc95d8f8b-lc62q"]
Sep 19 04:40:18.366595 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:40:18.366569    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0c01ced_87c7_4491_aa9c_2eabd3789794.slice/crio-c5678405880d45e2bf36e5da99c8feac9abb3444b8e9d571e9081ac5ee0bd850 WatchSource:0}: Error finding container c5678405880d45e2bf36e5da99c8feac9abb3444b8e9d571e9081ac5ee0bd850: Status 404 returned error can't find the container with id c5678405880d45e2bf36e5da99c8feac9abb3444b8e9d571e9081ac5ee0bd850
Sep 19 04:40:18.369364 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:18.369325    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 04:40:18.369449 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:18.369395    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 04:40:18.369449 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:18.369432    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 04:40:18.905683 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:18.905643    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-eac6c5-openshift-default-5bc95d8f8b-lc62q" event={"ID":"a0c01ced-87c7-4491-aa9c-2eabd3789794","Type":"ContainerStarted","Data":"7a90f1a74af62738e0bb8312e425d3ae5d4868c1e49b4c76bb2ad12f24097a49"}
Sep 19 04:40:18.906159 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:18.905721    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-eac6c5-openshift-default-5bc95d8f8b-lc62q" event={"ID":"a0c01ced-87c7-4491-aa9c-2eabd3789794","Type":"ContainerStarted","Data":"c5678405880d45e2bf36e5da99c8feac9abb3444b8e9d571e9081ac5ee0bd850"}
Sep 19 04:40:18.942291 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:18.942239    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-eac6c5-openshift-default-5bc95d8f8b-lc62q" podStartSLOduration=1.942222795 podStartE2EDuration="1.942222795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 04:40:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:40:18.942201148 +0000 UTC m=+1598.125162296" watchObservedRunningTime="2026-09-19 04:40:18.942222795 +0000 UTC m=+1598.125183942"
Sep 19 04:40:19.078453 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:19.078424    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-eac6c5-openshift-default-5bc95d8f8b-lc62q"
Sep 19 04:40:19.083724 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:19.083681    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-eac6c5-openshift-default-5bc95d8f8b-lc62q"
Sep 19 04:40:19.625465 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:19.625007    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a70050-z92xw"]
Sep 19 04:40:19.635960 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:19.635935    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a70050-z92xw"]
Sep 19 04:40:19.912993 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:19.912966    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-eac6c5-openshift-default-5bc95d8f8b-lc62q"
Sep 19 04:40:19.914233 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:19.914198    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-eac6c5-openshift-default-5bc95d8f8b-lc62q"
Sep 19 04:40:21.352576 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:21.352547    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3f137d9-4b2c-4436-8a41-5b5ea3e5ed5e" path="/var/lib/kubelet/pods/d3f137d9-4b2c-4436-8a41-5b5ea3e5ed5e/volumes"
Sep 19 04:40:28.130547 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:28.130515    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg"]
Sep 19 04:40:28.131018 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:28.130746    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg" podUID="03b3cd51-7a8f-4708-bd12-e9043fc4004d" containerName="istio-proxy" containerID="cri-o://d3563f1821ab8010a85191efc339ed2c1550e2d9dba19ad06b1276fac8fd207e" gracePeriod=30
Sep 19 04:40:33.392538 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:33.392466    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg"
Sep 19 04:40:33.513623 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:33.513586    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03b3cd51-7a8f-4708-bd12-e9043fc4004d-workload-socket\") pod \"03b3cd51-7a8f-4708-bd12-e9043fc4004d\" (UID: \"03b3cd51-7a8f-4708-bd12-e9043fc4004d\") "
Sep 19 04:40:33.513810 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:33.513643    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03b3cd51-7a8f-4708-bd12-e9043fc4004d-credential-socket\") pod \"03b3cd51-7a8f-4708-bd12-e9043fc4004d\" (UID: \"03b3cd51-7a8f-4708-bd12-e9043fc4004d\") "
Sep 19 04:40:33.513810 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:33.513689    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03b3cd51-7a8f-4708-bd12-e9043fc4004d-workload-certs\") pod \"03b3cd51-7a8f-4708-bd12-e9043fc4004d\" (UID: \"03b3cd51-7a8f-4708-bd12-e9043fc4004d\") "
Sep 19 04:40:33.513810 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:33.513777    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03b3cd51-7a8f-4708-bd12-e9043fc4004d-istiod-ca-cert\") pod \"03b3cd51-7a8f-4708-bd12-e9043fc4004d\" (UID: \"03b3cd51-7a8f-4708-bd12-e9043fc4004d\") "
Sep 19 04:40:33.513810 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:33.513807    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03b3cd51-7a8f-4708-bd12-e9043fc4004d-istio-podinfo\") pod \"03b3cd51-7a8f-4708-bd12-e9043fc4004d\" (UID: \"03b3cd51-7a8f-4708-bd12-e9043fc4004d\") "
Sep 19 04:40:33.514023 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:33.513884    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03b3cd51-7a8f-4708-bd12-e9043fc4004d-istio-token\") pod \"03b3cd51-7a8f-4708-bd12-e9043fc4004d\" (UID: \"03b3cd51-7a8f-4708-bd12-e9043fc4004d\") "
Sep 19 04:40:33.514023 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:33.513920    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rhq6g\" (UniqueName: \"kubernetes.io/projected/03b3cd51-7a8f-4708-bd12-e9043fc4004d-kube-api-access-rhq6g\") pod \"03b3cd51-7a8f-4708-bd12-e9043fc4004d\" (UID: \"03b3cd51-7a8f-4708-bd12-e9043fc4004d\") "
Sep 19 04:40:33.514023 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:33.513917    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03b3cd51-7a8f-4708-bd12-e9043fc4004d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "03b3cd51-7a8f-4708-bd12-e9043fc4004d" (UID: "03b3cd51-7a8f-4708-bd12-e9043fc4004d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:40:33.514023 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:33.513947    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03b3cd51-7a8f-4708-bd12-e9043fc4004d-istio-envoy\") pod \"03b3cd51-7a8f-4708-bd12-e9043fc4004d\" (UID: \"03b3cd51-7a8f-4708-bd12-e9043fc4004d\") "
Sep 19 04:40:33.514023 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:33.513992    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03b3cd51-7a8f-4708-bd12-e9043fc4004d-istio-data\") pod \"03b3cd51-7a8f-4708-bd12-e9043fc4004d\" (UID: \"03b3cd51-7a8f-4708-bd12-e9043fc4004d\") "
Sep 19 04:40:33.514240 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:33.514068    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03b3cd51-7a8f-4708-bd12-e9043fc4004d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "03b3cd51-7a8f-4708-bd12-e9043fc4004d" (UID: "03b3cd51-7a8f-4708-bd12-e9043fc4004d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:40:33.514240 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:33.514172    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03b3cd51-7a8f-4708-bd12-e9043fc4004d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "03b3cd51-7a8f-4708-bd12-e9043fc4004d" (UID: "03b3cd51-7a8f-4708-bd12-e9043fc4004d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 04:40:33.514319 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:33.514247    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03b3cd51-7a8f-4708-bd12-e9043fc4004d-istio-data" (OuterVolumeSpecName: "istio-data") pod "03b3cd51-7a8f-4708-bd12-e9043fc4004d" (UID: "03b3cd51-7a8f-4708-bd12-e9043fc4004d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:40:33.514474 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:33.514453    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03b3cd51-7a8f-4708-bd12-e9043fc4004d-istio-data\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:40:33.514557 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:33.514479    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03b3cd51-7a8f-4708-bd12-e9043fc4004d-workload-socket\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:40:33.514557 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:33.514493    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03b3cd51-7a8f-4708-bd12-e9043fc4004d-workload-certs\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:40:33.514557 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:33.514507    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03b3cd51-7a8f-4708-bd12-e9043fc4004d-istiod-ca-cert\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:40:33.514557 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:33.514520    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03b3cd51-7a8f-4708-bd12-e9043fc4004d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "03b3cd51-7a8f-4708-bd12-e9043fc4004d" (UID: "03b3cd51-7a8f-4708-bd12-e9043fc4004d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:40:33.516306 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:33.516277    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03b3cd51-7a8f-4708-bd12-e9043fc4004d-istio-token" (OuterVolumeSpecName: "istio-token") pod "03b3cd51-7a8f-4708-bd12-e9043fc4004d" (UID: "03b3cd51-7a8f-4708-bd12-e9043fc4004d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 04:40:33.516416 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:33.516363    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03b3cd51-7a8f-4708-bd12-e9043fc4004d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "03b3cd51-7a8f-4708-bd12-e9043fc4004d" (UID: "03b3cd51-7a8f-4708-bd12-e9043fc4004d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:40:33.516500 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:33.516483    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03b3cd51-7a8f-4708-bd12-e9043fc4004d-kube-api-access-rhq6g" (OuterVolumeSpecName: "kube-api-access-rhq6g") pod "03b3cd51-7a8f-4708-bd12-e9043fc4004d" (UID: "03b3cd51-7a8f-4708-bd12-e9043fc4004d"). InnerVolumeSpecName "kube-api-access-rhq6g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 04:40:33.516772 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:33.516753    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/03b3cd51-7a8f-4708-bd12-e9043fc4004d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "03b3cd51-7a8f-4708-bd12-e9043fc4004d" (UID: "03b3cd51-7a8f-4708-bd12-e9043fc4004d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 19 04:40:33.615056 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:33.615022    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03b3cd51-7a8f-4708-bd12-e9043fc4004d-istio-podinfo\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:40:33.615056 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:33.615052    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03b3cd51-7a8f-4708-bd12-e9043fc4004d-istio-token\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:40:33.615056 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:33.615063    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhq6g\" (UniqueName: \"kubernetes.io/projected/03b3cd51-7a8f-4708-bd12-e9043fc4004d-kube-api-access-rhq6g\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:40:33.615281 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:33.615071    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03b3cd51-7a8f-4708-bd12-e9043fc4004d-istio-envoy\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:40:33.615281 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:33.615082    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03b3cd51-7a8f-4708-bd12-e9043fc4004d-credential-socket\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:40:34.003520 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:34.003482    2579 generic.go:356] "Generic (PLEG): container finished" podID="03b3cd51-7a8f-4708-bd12-e9043fc4004d" containerID="d3563f1821ab8010a85191efc339ed2c1550e2d9dba19ad06b1276fac8fd207e" exitCode=0
Sep 19 04:40:34.003520 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:34.003518    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg" event={"ID":"03b3cd51-7a8f-4708-bd12-e9043fc4004d","Type":"ContainerDied","Data":"d3563f1821ab8010a85191efc339ed2c1550e2d9dba19ad06b1276fac8fd207e"}
Sep 19 04:40:34.003809 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:34.003533    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg"
Sep 19 04:40:34.003809 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:34.003574    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg" event={"ID":"03b3cd51-7a8f-4708-bd12-e9043fc4004d","Type":"ContainerDied","Data":"b329b955c9f8beacab2221685fe07935e44e2936b5c1311a0637198d29fd9f92"}
Sep 19 04:40:34.003809 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:34.003610    2579 scope.go:117] "RemoveContainer" containerID="d3563f1821ab8010a85191efc339ed2c1550e2d9dba19ad06b1276fac8fd207e"
Sep 19 04:40:34.014169 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:34.014154    2579 scope.go:117] "RemoveContainer" containerID="545d28b6ce34e06d26a419952295fc347940da866eae6210455f31f20777bc82"
Sep 19 04:40:34.023208 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:34.023191    2579 scope.go:117] "RemoveContainer" containerID="d3563f1821ab8010a85191efc339ed2c1550e2d9dba19ad06b1276fac8fd207e"
Sep 19 04:40:34.023480 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:40:34.023461    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3563f1821ab8010a85191efc339ed2c1550e2d9dba19ad06b1276fac8fd207e\": container with ID starting with d3563f1821ab8010a85191efc339ed2c1550e2d9dba19ad06b1276fac8fd207e not found: ID does not exist" containerID="d3563f1821ab8010a85191efc339ed2c1550e2d9dba19ad06b1276fac8fd207e"
Sep 19 04:40:34.023541 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:34.023489    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3563f1821ab8010a85191efc339ed2c1550e2d9dba19ad06b1276fac8fd207e"} err="failed to get container status \"d3563f1821ab8010a85191efc339ed2c1550e2d9dba19ad06b1276fac8fd207e\": rpc error: code = NotFound desc = could not find container \"d3563f1821ab8010a85191efc339ed2c1550e2d9dba19ad06b1276fac8fd207e\": container with ID starting with d3563f1821ab8010a85191efc339ed2c1550e2d9dba19ad06b1276fac8fd207e not found: ID does not exist"
Sep 19 04:40:34.023541 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:34.023506    2579 scope.go:117] "RemoveContainer" containerID="545d28b6ce34e06d26a419952295fc347940da866eae6210455f31f20777bc82"
Sep 19 04:40:34.023779 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:40:34.023761    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"545d28b6ce34e06d26a419952295fc347940da866eae6210455f31f20777bc82\": container with ID starting with 545d28b6ce34e06d26a419952295fc347940da866eae6210455f31f20777bc82 not found: ID does not exist" containerID="545d28b6ce34e06d26a419952295fc347940da866eae6210455f31f20777bc82"
Sep 19 04:40:34.023829 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:34.023785    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"545d28b6ce34e06d26a419952295fc347940da866eae6210455f31f20777bc82"} err="failed to get container status \"545d28b6ce34e06d26a419952295fc347940da866eae6210455f31f20777bc82\": rpc error: code = NotFound desc = could not find container \"545d28b6ce34e06d26a419952295fc347940da866eae6210455f31f20777bc82\": container with ID starting with 545d28b6ce34e06d26a419952295fc347940da866eae6210455f31f20777bc82 not found: ID does not exist"
Sep 19 04:40:34.028859 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:34.028838    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg"]
Sep 19 04:40:34.038408 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:34.038385    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-a70050-openshift-default-566bdbfbd8-zzfgg"]
Sep 19 04:40:35.351889 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:35.351858    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03b3cd51-7a8f-4708-bd12-e9043fc4004d" path="/var/lib/kubelet/pods/03b3cd51-7a8f-4708-bd12-e9043fc4004d/volumes"
Sep 19 04:40:41.490491 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:41.490455    2579 scope.go:117] "RemoveContainer" containerID="8fb84c42cd3afb7dfaca1eab1468ff6df01d5c5707db9eda2bf176694a5cb98d"
Sep 19 04:40:49.556313 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:49.556280    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-87e3e6/distinct-w0-87e3e6-kserve-54c675b78b-vsvns"]
Sep 19 04:40:49.556807 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:49.556790    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03b3cd51-7a8f-4708-bd12-e9043fc4004d" containerName="istio-proxy"
Sep 19 04:40:49.556857 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:49.556810    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="03b3cd51-7a8f-4708-bd12-e9043fc4004d" containerName="istio-proxy"
Sep 19 04:40:49.556900 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:49.556891    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="03b3cd51-7a8f-4708-bd12-e9043fc4004d" containerName="istio-proxy"
Sep 19 04:40:49.561225 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:49.561207    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-87e3e6/distinct-w0-87e3e6-kserve-54c675b78b-vsvns"
Sep 19 04:40:49.565140 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:49.565115    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-87e3e6\"/\"distinct-w0-87e3e6-kserve-self-signed-certs\""
Sep 19 04:40:49.571999 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:49.571975    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-87e3e6/distinct-w0-87e3e6-kserve-54c675b78b-vsvns"]
Sep 19 04:40:49.678458 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:49.678427    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91cd381a-3058-4df7-954e-984be634029e-tls-certs\") pod \"distinct-w0-87e3e6-kserve-54c675b78b-vsvns\" (UID: \"91cd381a-3058-4df7-954e-984be634029e\") " pod="e2e-models-e2e-worker-w0-87e3e6/distinct-w0-87e3e6-kserve-54c675b78b-vsvns"
Sep 19 04:40:49.678608 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:49.678479    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91cd381a-3058-4df7-954e-984be634029e-home\") pod \"distinct-w0-87e3e6-kserve-54c675b78b-vsvns\" (UID: \"91cd381a-3058-4df7-954e-984be634029e\") " pod="e2e-models-e2e-worker-w0-87e3e6/distinct-w0-87e3e6-kserve-54c675b78b-vsvns"
Sep 19 04:40:49.678608 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:49.678535    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91cd381a-3058-4df7-954e-984be634029e-dshm\") pod \"distinct-w0-87e3e6-kserve-54c675b78b-vsvns\" (UID: \"91cd381a-3058-4df7-954e-984be634029e\") " pod="e2e-models-e2e-worker-w0-87e3e6/distinct-w0-87e3e6-kserve-54c675b78b-vsvns"
Sep 19 04:40:49.678608 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:49.678567    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r822\" (UniqueName: \"kubernetes.io/projected/91cd381a-3058-4df7-954e-984be634029e-kube-api-access-8r822\") pod \"distinct-w0-87e3e6-kserve-54c675b78b-vsvns\" (UID: \"91cd381a-3058-4df7-954e-984be634029e\") " pod="e2e-models-e2e-worker-w0-87e3e6/distinct-w0-87e3e6-kserve-54c675b78b-vsvns"
Sep 19 04:40:49.678608 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:49.678587    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91cd381a-3058-4df7-954e-984be634029e-model-cache\") pod \"distinct-w0-87e3e6-kserve-54c675b78b-vsvns\" (UID: \"91cd381a-3058-4df7-954e-984be634029e\") " pod="e2e-models-e2e-worker-w0-87e3e6/distinct-w0-87e3e6-kserve-54c675b78b-vsvns"
Sep 19 04:40:49.678819 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:49.678622    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91cd381a-3058-4df7-954e-984be634029e-tmp-dir\") pod \"distinct-w0-87e3e6-kserve-54c675b78b-vsvns\" (UID: \"91cd381a-3058-4df7-954e-984be634029e\") " pod="e2e-models-e2e-worker-w0-87e3e6/distinct-w0-87e3e6-kserve-54c675b78b-vsvns"
Sep 19 04:40:49.779194 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:49.779155    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91cd381a-3058-4df7-954e-984be634029e-home\") pod \"distinct-w0-87e3e6-kserve-54c675b78b-vsvns\" (UID: \"91cd381a-3058-4df7-954e-984be634029e\") " pod="e2e-models-e2e-worker-w0-87e3e6/distinct-w0-87e3e6-kserve-54c675b78b-vsvns"
Sep 19 04:40:49.779194 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:49.779198    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91cd381a-3058-4df7-954e-984be634029e-dshm\") pod \"distinct-w0-87e3e6-kserve-54c675b78b-vsvns\" (UID: \"91cd381a-3058-4df7-954e-984be634029e\") " pod="e2e-models-e2e-worker-w0-87e3e6/distinct-w0-87e3e6-kserve-54c675b78b-vsvns"
Sep 19 04:40:49.779480 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:49.779220    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r822\" (UniqueName: \"kubernetes.io/projected/91cd381a-3058-4df7-954e-984be634029e-kube-api-access-8r822\") pod \"distinct-w0-87e3e6-kserve-54c675b78b-vsvns\" (UID: \"91cd381a-3058-4df7-954e-984be634029e\") " pod="e2e-models-e2e-worker-w0-87e3e6/distinct-w0-87e3e6-kserve-54c675b78b-vsvns"
Sep 19 04:40:49.779480 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:49.779237    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91cd381a-3058-4df7-954e-984be634029e-model-cache\") pod \"distinct-w0-87e3e6-kserve-54c675b78b-vsvns\" (UID: \"91cd381a-3058-4df7-954e-984be634029e\") " pod="e2e-models-e2e-worker-w0-87e3e6/distinct-w0-87e3e6-kserve-54c675b78b-vsvns"
Sep 19 04:40:49.779480 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:49.779364    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91cd381a-3058-4df7-954e-984be634029e-tmp-dir\") pod \"distinct-w0-87e3e6-kserve-54c675b78b-vsvns\" (UID: \"91cd381a-3058-4df7-954e-984be634029e\") " pod="e2e-models-e2e-worker-w0-87e3e6/distinct-w0-87e3e6-kserve-54c675b78b-vsvns"
Sep 19 04:40:49.779639 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:49.779509    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91cd381a-3058-4df7-954e-984be634029e-tls-certs\") pod \"distinct-w0-87e3e6-kserve-54c675b78b-vsvns\" (UID: \"91cd381a-3058-4df7-954e-984be634029e\") " pod="e2e-models-e2e-worker-w0-87e3e6/distinct-w0-87e3e6-kserve-54c675b78b-vsvns"
Sep 19 04:40:49.779639 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:49.779618    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91cd381a-3058-4df7-954e-984be634029e-home\") pod \"distinct-w0-87e3e6-kserve-54c675b78b-vsvns\" (UID: \"91cd381a-3058-4df7-954e-984be634029e\") " pod="e2e-models-e2e-worker-w0-87e3e6/distinct-w0-87e3e6-kserve-54c675b78b-vsvns"
Sep 19 04:40:49.779639 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:49.779630    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91cd381a-3058-4df7-954e-984be634029e-model-cache\") pod \"distinct-w0-87e3e6-kserve-54c675b78b-vsvns\" (UID: \"91cd381a-3058-4df7-954e-984be634029e\") " pod="e2e-models-e2e-worker-w0-87e3e6/distinct-w0-87e3e6-kserve-54c675b78b-vsvns"
Sep 19 04:40:49.779802 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:49.779681    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91cd381a-3058-4df7-954e-984be634029e-tmp-dir\") pod \"distinct-w0-87e3e6-kserve-54c675b78b-vsvns\" (UID: \"91cd381a-3058-4df7-954e-984be634029e\") " pod="e2e-models-e2e-worker-w0-87e3e6/distinct-w0-87e3e6-kserve-54c675b78b-vsvns"
Sep 19 04:40:49.786729 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:49.781687    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91cd381a-3058-4df7-954e-984be634029e-dshm\") pod \"distinct-w0-87e3e6-kserve-54c675b78b-vsvns\" (UID: \"91cd381a-3058-4df7-954e-984be634029e\") " pod="e2e-models-e2e-worker-w0-87e3e6/distinct-w0-87e3e6-kserve-54c675b78b-vsvns"
Sep 19 04:40:49.786729 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:49.786226    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91cd381a-3058-4df7-954e-984be634029e-tls-certs\") pod \"distinct-w0-87e3e6-kserve-54c675b78b-vsvns\" (UID: \"91cd381a-3058-4df7-954e-984be634029e\") " pod="e2e-models-e2e-worker-w0-87e3e6/distinct-w0-87e3e6-kserve-54c675b78b-vsvns"
Sep 19 04:40:49.795625 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:49.795589    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r822\" (UniqueName: \"kubernetes.io/projected/91cd381a-3058-4df7-954e-984be634029e-kube-api-access-8r822\") pod \"distinct-w0-87e3e6-kserve-54c675b78b-vsvns\" (UID: \"91cd381a-3058-4df7-954e-984be634029e\") " pod="e2e-models-e2e-worker-w0-87e3e6/distinct-w0-87e3e6-kserve-54c675b78b-vsvns"
Sep 19 04:40:49.872036 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:49.871940    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-87e3e6/distinct-w0-87e3e6-kserve-54c675b78b-vsvns"
Sep 19 04:40:50.158411 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:50.158382    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-87e3e6/distinct-w0-87e3e6-kserve-54c675b78b-vsvns"]
Sep 19 04:40:50.160616 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:40:50.160591    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod91cd381a_3058_4df7_954e_984be634029e.slice/crio-cd73e1cfd97471f1db15a4acb93d579a7a99ef164f8cbf2e1b092e6a373e3adb WatchSource:0}: Error finding container cd73e1cfd97471f1db15a4acb93d579a7a99ef164f8cbf2e1b092e6a373e3adb: Status 404 returned error can't find the container with id cd73e1cfd97471f1db15a4acb93d579a7a99ef164f8cbf2e1b092e6a373e3adb
Sep 19 04:40:51.116047 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:51.115992    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-87e3e6/distinct-w0-87e3e6-kserve-54c675b78b-vsvns" event={"ID":"91cd381a-3058-4df7-954e-984be634029e","Type":"ContainerStarted","Data":"e266d067b1572459b72323e6eea2f6a7aff885511f2239ef9768b5d6cb995c3f"}
Sep 19 04:40:51.116047 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:51.116048    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-87e3e6/distinct-w0-87e3e6-kserve-54c675b78b-vsvns" event={"ID":"91cd381a-3058-4df7-954e-984be634029e","Type":"ContainerStarted","Data":"cd73e1cfd97471f1db15a4acb93d579a7a99ef164f8cbf2e1b092e6a373e3adb"}
Sep 19 04:40:51.152624 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:51.152567    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-87e3e6/distinct-w0-87e3e6-kserve-54c675b78b-vsvns" podStartSLOduration=2.152551315 podStartE2EDuration="2.152551315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 04:40:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:40:51.149215817 +0000 UTC m=+1630.332176965" watchObservedRunningTime="2026-09-19 04:40:51.152551315 +0000 UTC m=+1630.335512463"
Sep 19 04:40:51.872796 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:51.872759    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-87e3e6/distinct-w0-87e3e6-kserve-54c675b78b-vsvns"
Sep 19 04:40:59.872833 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:59.872792    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-87e3e6/distinct-w0-87e3e6-kserve-54c675b78b-vsvns"
Sep 19 04:40:59.887046 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:40:59.887017    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-87e3e6/distinct-w0-87e3e6-kserve-54c675b78b-vsvns"
Sep 19 04:41:00.183254 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:00.183220    2579 generic.go:356] "Generic (PLEG): container finished" podID="a0c01ced-87c7-4491-aa9c-2eabd3789794" containerID="7a90f1a74af62738e0bb8312e425d3ae5d4868c1e49b4c76bb2ad12f24097a49" exitCode=0
Sep 19 04:41:00.183475 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:00.183299    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-eac6c5-openshift-default-5bc95d8f8b-lc62q" event={"ID":"a0c01ced-87c7-4491-aa9c-2eabd3789794","Type":"ContainerDied","Data":"7a90f1a74af62738e0bb8312e425d3ae5d4868c1e49b4c76bb2ad12f24097a49"}
Sep 19 04:41:00.183947 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:00.183918    2579 scope.go:117] "RemoveContainer" containerID="7a90f1a74af62738e0bb8312e425d3ae5d4868c1e49b4c76bb2ad12f24097a49"
Sep 19 04:41:00.185156 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:00.185126    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 04:41:00.185266 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:00.185206    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 04:41:00.185336 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:00.185272    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 04:41:00.202488 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:00.202468    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-87e3e6/distinct-w0-87e3e6-kserve-54c675b78b-vsvns"
Sep 19 04:41:01.195737 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:01.195688    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-eac6c5-openshift-default-5bc95d8f8b-lc62q" event={"ID":"a0c01ced-87c7-4491-aa9c-2eabd3789794","Type":"ContainerStarted","Data":"7cba260d9bfc04edb10a51cf1c512bb9605110e83a882a0910dde07f414e6860"}
Sep 19 04:41:01.208361 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:01.208323    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w"]
Sep 19 04:41:01.208855 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:01.208835    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03b3cd51-7a8f-4708-bd12-e9043fc4004d" containerName="istio-proxy"
Sep 19 04:41:01.208855 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:01.208856    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="03b3cd51-7a8f-4708-bd12-e9043fc4004d" containerName="istio-proxy"
Sep 19 04:41:01.209020 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:01.208979    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="03b3cd51-7a8f-4708-bd12-e9043fc4004d" containerName="istio-proxy"
Sep 19 04:41:01.211015 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:01.210994    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w"
Sep 19 04:41:01.214979 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:01.214962    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-a0ed681d-openshift-default-dockercfg-s4m7s\""
Sep 19 04:41:01.235830 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:01.235798    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w"]
Sep 19 04:41:01.302312 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:01.302274    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gzct\" (UniqueName: \"kubernetes.io/projected/b427907c-0354-4e21-930b-e3ef32dff3b9-kube-api-access-4gzct\") pod \"e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w\" (UID: \"b427907c-0354-4e21-930b-e3ef32dff3b9\") " pod="openshift-ingress/e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w"
Sep 19 04:41:01.302312 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:01.302324    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b427907c-0354-4e21-930b-e3ef32dff3b9-credential-socket\") pod \"e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w\" (UID: \"b427907c-0354-4e21-930b-e3ef32dff3b9\") " pod="openshift-ingress/e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w"
Sep 19 04:41:01.302591 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:01.302353    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b427907c-0354-4e21-930b-e3ef32dff3b9-istiod-ca-cert\") pod \"e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w\" (UID: \"b427907c-0354-4e21-930b-e3ef32dff3b9\") " pod="openshift-ingress/e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w"
Sep 19 04:41:01.302591 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:01.302410    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b427907c-0354-4e21-930b-e3ef32dff3b9-istio-podinfo\") pod \"e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w\" (UID: \"b427907c-0354-4e21-930b-e3ef32dff3b9\") " pod="openshift-ingress/e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w"
Sep 19 04:41:01.302591 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:01.302454    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b427907c-0354-4e21-930b-e3ef32dff3b9-istio-envoy\") pod \"e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w\" (UID: \"b427907c-0354-4e21-930b-e3ef32dff3b9\") " pod="openshift-ingress/e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w"
Sep 19 04:41:01.302591 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:01.302506    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b427907c-0354-4e21-930b-e3ef32dff3b9-istio-data\") pod \"e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w\" (UID: \"b427907c-0354-4e21-930b-e3ef32dff3b9\") " pod="openshift-ingress/e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w"
Sep 19 04:41:01.302591 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:01.302533    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b427907c-0354-4e21-930b-e3ef32dff3b9-workload-certs\") pod \"e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w\" (UID: \"b427907c-0354-4e21-930b-e3ef32dff3b9\") " pod="openshift-ingress/e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w"
Sep 19 04:41:01.302591 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:01.302576    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b427907c-0354-4e21-930b-e3ef32dff3b9-istio-token\") pod \"e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w\" (UID: \"b427907c-0354-4e21-930b-e3ef32dff3b9\") " pod="openshift-ingress/e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w"
Sep 19 04:41:01.302976 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:01.302656    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b427907c-0354-4e21-930b-e3ef32dff3b9-workload-socket\") pod \"e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w\" (UID: \"b427907c-0354-4e21-930b-e3ef32dff3b9\") " pod="openshift-ingress/e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w"
Sep 19 04:41:01.403918 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:01.403864    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b427907c-0354-4e21-930b-e3ef32dff3b9-workload-socket\") pod \"e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w\" (UID: \"b427907c-0354-4e21-930b-e3ef32dff3b9\") " pod="openshift-ingress/e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w"
Sep 19 04:41:01.404130 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:01.403979    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gzct\" (UniqueName: \"kubernetes.io/projected/b427907c-0354-4e21-930b-e3ef32dff3b9-kube-api-access-4gzct\") pod \"e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w\" (UID: \"b427907c-0354-4e21-930b-e3ef32dff3b9\") " pod="openshift-ingress/e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w"
Sep 19 04:41:01.404130 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:01.404011    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b427907c-0354-4e21-930b-e3ef32dff3b9-credential-socket\") pod \"e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w\" (UID: \"b427907c-0354-4e21-930b-e3ef32dff3b9\") " pod="openshift-ingress/e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w"
Sep 19 04:41:01.404130 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:01.404035    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b427907c-0354-4e21-930b-e3ef32dff3b9-istiod-ca-cert\") pod \"e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w\" (UID: \"b427907c-0354-4e21-930b-e3ef32dff3b9\") " pod="openshift-ingress/e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w"
Sep 19 04:41:01.404130 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:01.404084    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b427907c-0354-4e21-930b-e3ef32dff3b9-istio-podinfo\") pod \"e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w\" (UID: \"b427907c-0354-4e21-930b-e3ef32dff3b9\") " pod="openshift-ingress/e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w"
Sep 19 04:41:01.404372 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:01.404133    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b427907c-0354-4e21-930b-e3ef32dff3b9-istio-envoy\") pod \"e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w\" (UID: \"b427907c-0354-4e21-930b-e3ef32dff3b9\") " pod="openshift-ingress/e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w"
Sep 19 04:41:01.404372 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:01.404192    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b427907c-0354-4e21-930b-e3ef32dff3b9-istio-data\") pod \"e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w\" (UID: \"b427907c-0354-4e21-930b-e3ef32dff3b9\") " pod="openshift-ingress/e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w"
Sep 19 04:41:01.404372 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:01.404233    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b427907c-0354-4e21-930b-e3ef32dff3b9-workload-certs\") pod \"e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w\" (UID: \"b427907c-0354-4e21-930b-e3ef32dff3b9\") " pod="openshift-ingress/e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w"
Sep 19 04:41:01.404372 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:01.404302    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b427907c-0354-4e21-930b-e3ef32dff3b9-istio-token\") pod \"e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w\" (UID: \"b427907c-0354-4e21-930b-e3ef32dff3b9\") " pod="openshift-ingress/e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w"
Sep 19 04:41:01.404372 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:01.404301    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b427907c-0354-4e21-930b-e3ef32dff3b9-workload-socket\") pod \"e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w\" (UID: \"b427907c-0354-4e21-930b-e3ef32dff3b9\") " pod="openshift-ingress/e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w"
Sep 19 04:41:01.404631 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:01.404372    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b427907c-0354-4e21-930b-e3ef32dff3b9-credential-socket\") pod \"e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w\" (UID: \"b427907c-0354-4e21-930b-e3ef32dff3b9\") " pod="openshift-ingress/e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w"
Sep 19 04:41:01.404772 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:01.404743    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b427907c-0354-4e21-930b-e3ef32dff3b9-workload-certs\") pod \"e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w\" (UID: \"b427907c-0354-4e21-930b-e3ef32dff3b9\") " pod="openshift-ingress/e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w"
Sep 19 04:41:01.404930 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:01.404835    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b427907c-0354-4e21-930b-e3ef32dff3b9-istio-data\") pod \"e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w\" (UID: \"b427907c-0354-4e21-930b-e3ef32dff3b9\") " pod="openshift-ingress/e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w"
Sep 19 04:41:01.405123 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:01.405102    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b427907c-0354-4e21-930b-e3ef32dff3b9-istiod-ca-cert\") pod \"e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w\" (UID: \"b427907c-0354-4e21-930b-e3ef32dff3b9\") " pod="openshift-ingress/e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w"
Sep 19 04:41:01.406822 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:01.406806    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b427907c-0354-4e21-930b-e3ef32dff3b9-istio-podinfo\") pod \"e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w\" (UID: \"b427907c-0354-4e21-930b-e3ef32dff3b9\") " pod="openshift-ingress/e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w"
Sep 19 04:41:01.406985 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:01.406971    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b427907c-0354-4e21-930b-e3ef32dff3b9-istio-envoy\") pod \"e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w\" (UID: \"b427907c-0354-4e21-930b-e3ef32dff3b9\") " pod="openshift-ingress/e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w"
Sep 19 04:41:01.417511 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:01.417483    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gzct\" (UniqueName: \"kubernetes.io/projected/b427907c-0354-4e21-930b-e3ef32dff3b9-kube-api-access-4gzct\") pod \"e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w\" (UID: \"b427907c-0354-4e21-930b-e3ef32dff3b9\") " pod="openshift-ingress/e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w"
Sep 19 04:41:01.417662 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:01.417642    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b427907c-0354-4e21-930b-e3ef32dff3b9-istio-token\") pod \"e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w\" (UID: \"b427907c-0354-4e21-930b-e3ef32dff3b9\") " pod="openshift-ingress/e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w"
Sep 19 04:41:01.525838 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:01.525737    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w"
Sep 19 04:41:01.849483 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:41:01.849451    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb427907c_0354_4e21_930b_e3ef32dff3b9.slice/crio-1fa5588e00e60ee84eb1196d2dfe89142b4ff424ddff7cda6aa0590e63258f58 WatchSource:0}: Error finding container 1fa5588e00e60ee84eb1196d2dfe89142b4ff424ddff7cda6aa0590e63258f58: Status 404 returned error can't find the container with id 1fa5588e00e60ee84eb1196d2dfe89142b4ff424ddff7cda6aa0590e63258f58
Sep 19 04:41:01.850993 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:01.850969    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w"]
Sep 19 04:41:01.851965 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:01.851930    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 04:41:01.852017 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:01.852007    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 04:41:01.852052 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:01.852042    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 04:41:02.078341 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:02.078307    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-eac6c5-openshift-default-5bc95d8f8b-lc62q"
Sep 19 04:41:02.079721 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:02.079682    2579 patch_prober.go:28] interesting pod/e2e-shared-b-w2-eac6c5-openshift-default-5bc95d8f8b-lc62q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Sep 19 04:41:02.079809 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:02.079758    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-eac6c5-openshift-default-5bc95d8f8b-lc62q" podUID="a0c01ced-87c7-4491-aa9c-2eabd3789794" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Sep 19 04:41:02.209615 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:02.209571    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w" event={"ID":"b427907c-0354-4e21-930b-e3ef32dff3b9","Type":"ContainerStarted","Data":"0b7e5a62b6f1af77d9049f75023c1413507f84d42fa1a3ebefe4a2cb15d9e75e"}
Sep 19 04:41:02.210116 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:02.209636    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w" event={"ID":"b427907c-0354-4e21-930b-e3ef32dff3b9","Type":"ContainerStarted","Data":"1fa5588e00e60ee84eb1196d2dfe89142b4ff424ddff7cda6aa0590e63258f58"}
Sep 19 04:41:02.231182 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:02.231129    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w" podStartSLOduration=1.231108346 podStartE2EDuration="1.231108346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 04:41:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:41:02.229885403 +0000 UTC m=+1641.412846552" watchObservedRunningTime="2026-09-19 04:41:02.231108346 +0000 UTC m=+1641.414069493"
Sep 19 04:41:02.526704 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:02.526602    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w"
Sep 19 04:41:02.532091 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:02.532062    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w"
Sep 19 04:41:03.078878 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:03.078845    2579 patch_prober.go:28] interesting pod/e2e-shared-b-w2-eac6c5-openshift-default-5bc95d8f8b-lc62q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Sep 19 04:41:03.079069 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:03.078900    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-eac6c5-openshift-default-5bc95d8f8b-lc62q" podUID="a0c01ced-87c7-4491-aa9c-2eabd3789794" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Sep 19 04:41:03.216102 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:03.216069    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w"
Sep 19 04:41:03.217098 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:03.217076    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w"
Sep 19 04:41:04.078843 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:04.078801    2579 patch_prober.go:28] interesting pod/e2e-shared-b-w2-eac6c5-openshift-default-5bc95d8f8b-lc62q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Sep 19 04:41:04.079032 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:04.078859    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-eac6c5-openshift-default-5bc95d8f8b-lc62q" podUID="a0c01ced-87c7-4491-aa9c-2eabd3789794" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Sep 19 04:41:04.913253 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:04.913211    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-eac6c5-openshift-default-5bc95d8f8b-lc62q"
Sep 19 04:41:05.079083 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:05.079049    2579 patch_prober.go:28] interesting pod/e2e-shared-b-w2-eac6c5-openshift-default-5bc95d8f8b-lc62q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Sep 19 04:41:05.079237 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:05.079111    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-eac6c5-openshift-default-5bc95d8f8b-lc62q" podUID="a0c01ced-87c7-4491-aa9c-2eabd3789794" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Sep 19 04:41:06.078275 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:06.078238    2579 patch_prober.go:28] interesting pod/e2e-shared-b-w2-eac6c5-openshift-default-5bc95d8f8b-lc62q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Sep 19 04:41:06.078643 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:06.078299    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-eac6c5-openshift-default-5bc95d8f8b-lc62q" podUID="a0c01ced-87c7-4491-aa9c-2eabd3789794" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Sep 19 04:41:07.108398 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:07.108362    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-eac6c5-openshift-default-5bc95d8f8b-lc62q"
Sep 19 04:41:07.110559 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:07.110539    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-eac6c5-openshift-default-5bc95d8f8b-lc62q"
Sep 19 04:41:33.217786 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:33.217674    2579 patch_prober.go:28] interesting pod/e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Sep 19 04:41:33.217786 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:33.217771    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w" podUID="b427907c-0354-4e21-930b-e3ef32dff3b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Sep 19 04:41:33.420872 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:33.420833    2579 generic.go:356] "Generic (PLEG): container finished" podID="b427907c-0354-4e21-930b-e3ef32dff3b9" containerID="0b7e5a62b6f1af77d9049f75023c1413507f84d42fa1a3ebefe4a2cb15d9e75e" exitCode=0
Sep 19 04:41:33.421043 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:33.420873    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w" event={"ID":"b427907c-0354-4e21-930b-e3ef32dff3b9","Type":"ContainerDied","Data":"0b7e5a62b6f1af77d9049f75023c1413507f84d42fa1a3ebefe4a2cb15d9e75e"}
Sep 19 04:41:33.421287 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:33.421273    2579 scope.go:117] "RemoveContainer" containerID="0b7e5a62b6f1af77d9049f75023c1413507f84d42fa1a3ebefe4a2cb15d9e75e"
Sep 19 04:41:33.422603 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:33.422576    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 04:41:33.422669 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:33.422647    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 04:41:33.422732 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:33.422679    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 04:41:34.430222 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:34.430189    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w" event={"ID":"b427907c-0354-4e21-930b-e3ef32dff3b9","Type":"ContainerStarted","Data":"5d4f57df6c58261a58ea006ef0fe764523ceaee00593a7f07ee09af4224454f7"}
Sep 19 04:41:34.525987 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:34.525952    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w"
Sep 19 04:41:34.526161 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:34.526129    2579 patch_prober.go:28] interesting pod/e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Sep 19 04:41:34.526208 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:34.526183    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w" podUID="b427907c-0354-4e21-930b-e3ef32dff3b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Sep 19 04:41:35.526378 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:35.526343    2579 patch_prober.go:28] interesting pod/e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Sep 19 04:41:35.526807 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:35.526399    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w" podUID="b427907c-0354-4e21-930b-e3ef32dff3b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Sep 19 04:41:36.526807 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:36.526772    2579 patch_prober.go:28] interesting pod/e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Sep 19 04:41:36.527218 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:36.526823    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w" podUID="b427907c-0354-4e21-930b-e3ef32dff3b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Sep 19 04:41:37.527007 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:37.526967    2579 patch_prober.go:28] interesting pod/e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Sep 19 04:41:37.527411 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:37.527021    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w" podUID="b427907c-0354-4e21-930b-e3ef32dff3b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Sep 19 04:41:38.526562 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:38.526529    2579 patch_prober.go:28] interesting pod/e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Sep 19 04:41:38.526857 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:38.526827    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w" podUID="b427907c-0354-4e21-930b-e3ef32dff3b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Sep 19 04:41:39.557007 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:39.556972    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w"
Sep 19 04:41:39.558005 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:39.557977    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w"
Sep 19 04:41:39.559379 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:39.559359    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w"
Sep 19 04:41:41.520950 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:41.520916    2579 scope.go:117] "RemoveContainer" containerID="16d4d7f362a14ee5a553a5dab2ca5c86a66df5738414a04a8bba3883af04d53a"
Sep 19 04:41:46.670575 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:41:46.670542    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-bcbdad/test-model-bcbdad-kserve-6488bff88b-2tfl6"]
Sep 19 04:42:00.394935 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:00.394891    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bcbdad-smwbz"]
Sep 19 04:42:00.398730 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:00.398688    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bcbdad-smwbz"
Sep 19 04:42:00.401371 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:00.401350    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s6xbq\""
Sep 19 04:42:00.403556 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:00.403533    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 19 04:42:00.409635 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:00.409610    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bcbdad-smwbz"]
Sep 19 04:42:00.552088 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:00.552059    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ab7c5a7-4566-4e16-a8f6-cf561d5855f4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-bcbdad-smwbz\" (UID: \"6ab7c5a7-4566-4e16-a8f6-cf561d5855f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bcbdad-smwbz"
Sep 19 04:42:00.652893 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:00.652864    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ab7c5a7-4566-4e16-a8f6-cf561d5855f4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-bcbdad-smwbz\" (UID: \"6ab7c5a7-4566-4e16-a8f6-cf561d5855f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bcbdad-smwbz"
Sep 19 04:42:00.653501 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:00.653482    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ab7c5a7-4566-4e16-a8f6-cf561d5855f4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-bcbdad-smwbz\" (UID: \"6ab7c5a7-4566-4e16-a8f6-cf561d5855f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bcbdad-smwbz"
Sep 19 04:42:00.711852 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:00.711819    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bcbdad-smwbz"
Sep 19 04:42:00.997088 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:00.997063    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bcbdad-smwbz"]
Sep 19 04:42:00.999903 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:42:00.999878    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ab7c5a7_4566_4e16_a8f6_cf561d5855f4.slice/crio-c7ceef65f7c97c0e692edca635bfe26439ae9d4b91ce99c6b28df4348d5f9566 WatchSource:0}: Error finding container c7ceef65f7c97c0e692edca635bfe26439ae9d4b91ce99c6b28df4348d5f9566: Status 404 returned error can't find the container with id c7ceef65f7c97c0e692edca635bfe26439ae9d4b91ce99c6b28df4348d5f9566
Sep 19 04:42:01.610733 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:01.610623    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bcbdad-smwbz" event={"ID":"6ab7c5a7-4566-4e16-a8f6-cf561d5855f4","Type":"ContainerStarted","Data":"372c0e635ec2b448bd4cbf591e9e452ab01f6d2792bd46320e424e98f734eff6"}
Sep 19 04:42:01.611089 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:01.610744    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bcbdad-smwbz" event={"ID":"6ab7c5a7-4566-4e16-a8f6-cf561d5855f4","Type":"ContainerStarted","Data":"c7ceef65f7c97c0e692edca635bfe26439ae9d4b91ce99c6b28df4348d5f9566"}
Sep 19 04:42:01.634104 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:01.634056    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bcbdad-smwbz" podStartSLOduration=1.634039732 podStartE2EDuration="1.634039732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 04:42:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:42:01.634038907 +0000 UTC m=+1700.817000054" watchObservedRunningTime="2026-09-19 04:42:01.634039732 +0000 UTC m=+1700.817000909"
Sep 19 04:42:01.691014 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:01.690955    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-bcbdad/test-model-bcbdad-kserve-6488bff88b-2tfl6" podUID="04fdd4d9-a25e-49ed-b803-9a3eb2c68745" containerName="main" containerID="cri-o://89f62edbb05072b4e9725fde1bc356a239f7ddca18b19719ff468bc527215869" gracePeriod=45
Sep 19 04:42:01.955988 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:01.955964    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-bcbdad/test-model-bcbdad-kserve-6488bff88b-2tfl6"
Sep 19 04:42:02.067027 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:02.066996    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04fdd4d9-a25e-49ed-b803-9a3eb2c68745-tls-certs\") pod \"04fdd4d9-a25e-49ed-b803-9a3eb2c68745\" (UID: \"04fdd4d9-a25e-49ed-b803-9a3eb2c68745\") "
Sep 19 04:42:02.067027 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:02.067045    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04fdd4d9-a25e-49ed-b803-9a3eb2c68745-model-cache\") pod \"04fdd4d9-a25e-49ed-b803-9a3eb2c68745\" (UID: \"04fdd4d9-a25e-49ed-b803-9a3eb2c68745\") "
Sep 19 04:42:02.067252 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:02.067091    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04fdd4d9-a25e-49ed-b803-9a3eb2c68745-dshm\") pod \"04fdd4d9-a25e-49ed-b803-9a3eb2c68745\" (UID: \"04fdd4d9-a25e-49ed-b803-9a3eb2c68745\") "
Sep 19 04:42:02.067252 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:02.067123    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04fdd4d9-a25e-49ed-b803-9a3eb2c68745-home\") pod \"04fdd4d9-a25e-49ed-b803-9a3eb2c68745\" (UID: \"04fdd4d9-a25e-49ed-b803-9a3eb2c68745\") "
Sep 19 04:42:02.067252 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:02.067185    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rh7ds\" (UniqueName: \"kubernetes.io/projected/04fdd4d9-a25e-49ed-b803-9a3eb2c68745-kube-api-access-rh7ds\") pod \"04fdd4d9-a25e-49ed-b803-9a3eb2c68745\" (UID: \"04fdd4d9-a25e-49ed-b803-9a3eb2c68745\") "
Sep 19 04:42:02.067252 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:02.067203    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04fdd4d9-a25e-49ed-b803-9a3eb2c68745-tmp-dir\") pod \"04fdd4d9-a25e-49ed-b803-9a3eb2c68745\" (UID: \"04fdd4d9-a25e-49ed-b803-9a3eb2c68745\") "
Sep 19 04:42:02.067441 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:02.067417    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04fdd4d9-a25e-49ed-b803-9a3eb2c68745-model-cache" (OuterVolumeSpecName: "model-cache") pod "04fdd4d9-a25e-49ed-b803-9a3eb2c68745" (UID: "04fdd4d9-a25e-49ed-b803-9a3eb2c68745"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:42:02.067502 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:02.067468    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04fdd4d9-a25e-49ed-b803-9a3eb2c68745-home" (OuterVolumeSpecName: "home") pod "04fdd4d9-a25e-49ed-b803-9a3eb2c68745" (UID: "04fdd4d9-a25e-49ed-b803-9a3eb2c68745"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:42:02.067562 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:02.067539    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04fdd4d9-a25e-49ed-b803-9a3eb2c68745-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "04fdd4d9-a25e-49ed-b803-9a3eb2c68745" (UID: "04fdd4d9-a25e-49ed-b803-9a3eb2c68745"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:42:02.067620 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:02.067604    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04fdd4d9-a25e-49ed-b803-9a3eb2c68745-model-cache\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:42:02.067665 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:02.067623    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04fdd4d9-a25e-49ed-b803-9a3eb2c68745-home\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:42:02.069516 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:02.069488    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04fdd4d9-a25e-49ed-b803-9a3eb2c68745-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "04fdd4d9-a25e-49ed-b803-9a3eb2c68745" (UID: "04fdd4d9-a25e-49ed-b803-9a3eb2c68745"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 04:42:02.069622 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:02.069522    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04fdd4d9-a25e-49ed-b803-9a3eb2c68745-dshm" (OuterVolumeSpecName: "dshm") pod "04fdd4d9-a25e-49ed-b803-9a3eb2c68745" (UID: "04fdd4d9-a25e-49ed-b803-9a3eb2c68745"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:42:02.069622 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:02.069608    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04fdd4d9-a25e-49ed-b803-9a3eb2c68745-kube-api-access-rh7ds" (OuterVolumeSpecName: "kube-api-access-rh7ds") pod "04fdd4d9-a25e-49ed-b803-9a3eb2c68745" (UID: "04fdd4d9-a25e-49ed-b803-9a3eb2c68745"). InnerVolumeSpecName "kube-api-access-rh7ds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 04:42:02.169368 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:02.169286    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04fdd4d9-a25e-49ed-b803-9a3eb2c68745-dshm\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:42:02.169368 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:02.169321    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rh7ds\" (UniqueName: \"kubernetes.io/projected/04fdd4d9-a25e-49ed-b803-9a3eb2c68745-kube-api-access-rh7ds\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:42:02.169368 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:02.169333    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04fdd4d9-a25e-49ed-b803-9a3eb2c68745-tmp-dir\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:42:02.169368 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:02.169348    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04fdd4d9-a25e-49ed-b803-9a3eb2c68745-tls-certs\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:42:02.619158 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:02.619126    2579 generic.go:356] "Generic (PLEG): container finished" podID="04fdd4d9-a25e-49ed-b803-9a3eb2c68745" containerID="89f62edbb05072b4e9725fde1bc356a239f7ddca18b19719ff468bc527215869" exitCode=0
Sep 19 04:42:02.619632 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:02.619171    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-bcbdad/test-model-bcbdad-kserve-6488bff88b-2tfl6"
Sep 19 04:42:02.619632 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:02.619219    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-bcbdad/test-model-bcbdad-kserve-6488bff88b-2tfl6" event={"ID":"04fdd4d9-a25e-49ed-b803-9a3eb2c68745","Type":"ContainerDied","Data":"89f62edbb05072b4e9725fde1bc356a239f7ddca18b19719ff468bc527215869"}
Sep 19 04:42:02.619632 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:02.619260    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-bcbdad/test-model-bcbdad-kserve-6488bff88b-2tfl6" event={"ID":"04fdd4d9-a25e-49ed-b803-9a3eb2c68745","Type":"ContainerDied","Data":"43ca6635f3c79dfedf038975465ce0f536190577efc293d0f7f3c647e0bcc095"}
Sep 19 04:42:02.619632 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:02.619279    2579 scope.go:117] "RemoveContainer" containerID="89f62edbb05072b4e9725fde1bc356a239f7ddca18b19719ff468bc527215869"
Sep 19 04:42:02.621235 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:02.621213    2579 generic.go:356] "Generic (PLEG): container finished" podID="6ab7c5a7-4566-4e16-a8f6-cf561d5855f4" containerID="372c0e635ec2b448bd4cbf591e9e452ab01f6d2792bd46320e424e98f734eff6" exitCode=0
Sep 19 04:42:02.621362 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:02.621346    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bcbdad-smwbz" event={"ID":"6ab7c5a7-4566-4e16-a8f6-cf561d5855f4","Type":"ContainerDied","Data":"372c0e635ec2b448bd4cbf591e9e452ab01f6d2792bd46320e424e98f734eff6"}
Sep 19 04:42:02.630157 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:02.630131    2579 scope.go:117] "RemoveContainer" containerID="89f62edbb05072b4e9725fde1bc356a239f7ddca18b19719ff468bc527215869"
Sep 19 04:42:02.630449 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:42:02.630429    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89f62edbb05072b4e9725fde1bc356a239f7ddca18b19719ff468bc527215869\": container with ID starting with 89f62edbb05072b4e9725fde1bc356a239f7ddca18b19719ff468bc527215869 not found: ID does not exist" containerID="89f62edbb05072b4e9725fde1bc356a239f7ddca18b19719ff468bc527215869"
Sep 19 04:42:02.630519 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:02.630459    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89f62edbb05072b4e9725fde1bc356a239f7ddca18b19719ff468bc527215869"} err="failed to get container status \"89f62edbb05072b4e9725fde1bc356a239f7ddca18b19719ff468bc527215869\": rpc error: code = NotFound desc = could not find container \"89f62edbb05072b4e9725fde1bc356a239f7ddca18b19719ff468bc527215869\": container with ID starting with 89f62edbb05072b4e9725fde1bc356a239f7ddca18b19719ff468bc527215869 not found: ID does not exist"
Sep 19 04:42:02.686320 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:02.686278    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-bcbdad/test-model-bcbdad-kserve-6488bff88b-2tfl6"]
Sep 19 04:42:02.696709 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:02.696656    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-bcbdad/test-model-bcbdad-kserve-6488bff88b-2tfl6"]
Sep 19 04:42:02.915349 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:02.915309    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-bcbdad/test-model-bcbdad-kserve-6488bff88b-2tfl6" podUID="04fdd4d9-a25e-49ed-b803-9a3eb2c68745" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded"
Sep 19 04:42:03.352436 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:03.352356    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04fdd4d9-a25e-49ed-b803-9a3eb2c68745" path="/var/lib/kubelet/pods/04fdd4d9-a25e-49ed-b803-9a3eb2c68745/volumes"
Sep 19 04:42:03.774928 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:03.774903    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bcbdad-smwbz"
Sep 19 04:42:03.888520 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:03.888407    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ab7c5a7-4566-4e16-a8f6-cf561d5855f4-maas-api-service-ca\") pod \"6ab7c5a7-4566-4e16-a8f6-cf561d5855f4\" (UID: \"6ab7c5a7-4566-4e16-a8f6-cf561d5855f4\") "
Sep 19 04:42:03.888907 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:03.888886    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ab7c5a7-4566-4e16-a8f6-cf561d5855f4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6ab7c5a7-4566-4e16-a8f6-cf561d5855f4" (UID: "6ab7c5a7-4566-4e16-a8f6-cf561d5855f4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 04:42:03.990226 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:03.990189    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ab7c5a7-4566-4e16-a8f6-cf561d5855f4-maas-api-service-ca\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:42:04.637274 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:04.637242    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bcbdad-smwbz" event={"ID":"6ab7c5a7-4566-4e16-a8f6-cf561d5855f4","Type":"ContainerDied","Data":"c7ceef65f7c97c0e692edca635bfe26439ae9d4b91ce99c6b28df4348d5f9566"}
Sep 19 04:42:04.637464 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:04.637278    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bcbdad-smwbz"
Sep 19 04:42:04.637464 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:04.637289    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7ceef65f7c97c0e692edca635bfe26439ae9d4b91ce99c6b28df4348d5f9566"
Sep 19 04:42:06.296554 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:06.289247    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-aefa94/noten-model-aefa94-kserve-78f5fb9c56-mqfmj"]
Sep 19 04:42:06.296554 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:06.290199    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ab7c5a7-4566-4e16-a8f6-cf561d5855f4" containerName="revoke-keys"
Sep 19 04:42:06.296554 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:06.290219    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ab7c5a7-4566-4e16-a8f6-cf561d5855f4" containerName="revoke-keys"
Sep 19 04:42:06.296554 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:06.290242    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04fdd4d9-a25e-49ed-b803-9a3eb2c68745" containerName="main"
Sep 19 04:42:06.296554 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:06.290249    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="04fdd4d9-a25e-49ed-b803-9a3eb2c68745" containerName="main"
Sep 19 04:42:06.296554 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:06.290477    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="04fdd4d9-a25e-49ed-b803-9a3eb2c68745" containerName="main"
Sep 19 04:42:06.296554 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:06.290507    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ab7c5a7-4566-4e16-a8f6-cf561d5855f4" containerName="revoke-keys"
Sep 19 04:42:06.296554 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:06.294831    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-aefa94/noten-model-aefa94-kserve-78f5fb9c56-mqfmj"
Sep 19 04:42:06.299045 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:06.299019    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-aefa94\"/\"default-dockercfg-qnfzm\""
Sep 19 04:42:06.301125 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:06.301102    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-aefa94\"/\"kube-root-ca.crt\""
Sep 19 04:42:06.301228 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:06.301115    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-aefa94\"/\"noten-model-aefa94-kserve-self-signed-certs\""
Sep 19 04:42:06.301763 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:06.301736    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-aefa94\"/\"openshift-service-ca.crt\""
Sep 19 04:42:06.306857 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:06.306834    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-aefa94/noten-model-aefa94-kserve-78f5fb9c56-mqfmj"]
Sep 19 04:42:06.415585 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:06.415553    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f06467cc-6d14-4f96-ab33-0b3cbdc14ff4-home\") pod \"noten-model-aefa94-kserve-78f5fb9c56-mqfmj\" (UID: \"f06467cc-6d14-4f96-ab33-0b3cbdc14ff4\") " pod="ai-tenant-e2e-noten-aefa94/noten-model-aefa94-kserve-78f5fb9c56-mqfmj"
Sep 19 04:42:06.415763 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:06.415599    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f06467cc-6d14-4f96-ab33-0b3cbdc14ff4-dshm\") pod \"noten-model-aefa94-kserve-78f5fb9c56-mqfmj\" (UID: \"f06467cc-6d14-4f96-ab33-0b3cbdc14ff4\") " pod="ai-tenant-e2e-noten-aefa94/noten-model-aefa94-kserve-78f5fb9c56-mqfmj"
Sep 19 04:42:06.415763 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:06.415617    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f06467cc-6d14-4f96-ab33-0b3cbdc14ff4-tls-certs\") pod \"noten-model-aefa94-kserve-78f5fb9c56-mqfmj\" (UID: \"f06467cc-6d14-4f96-ab33-0b3cbdc14ff4\") " pod="ai-tenant-e2e-noten-aefa94/noten-model-aefa94-kserve-78f5fb9c56-mqfmj"
Sep 19 04:42:06.415763 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:06.415671    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f06467cc-6d14-4f96-ab33-0b3cbdc14ff4-model-cache\") pod \"noten-model-aefa94-kserve-78f5fb9c56-mqfmj\" (UID: \"f06467cc-6d14-4f96-ab33-0b3cbdc14ff4\") " pod="ai-tenant-e2e-noten-aefa94/noten-model-aefa94-kserve-78f5fb9c56-mqfmj"
Sep 19 04:42:06.415891 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:06.415791    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdwc5\" (UniqueName: \"kubernetes.io/projected/f06467cc-6d14-4f96-ab33-0b3cbdc14ff4-kube-api-access-bdwc5\") pod \"noten-model-aefa94-kserve-78f5fb9c56-mqfmj\" (UID: \"f06467cc-6d14-4f96-ab33-0b3cbdc14ff4\") " pod="ai-tenant-e2e-noten-aefa94/noten-model-aefa94-kserve-78f5fb9c56-mqfmj"
Sep 19 04:42:06.415927 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:06.415890    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f06467cc-6d14-4f96-ab33-0b3cbdc14ff4-tmp-dir\") pod \"noten-model-aefa94-kserve-78f5fb9c56-mqfmj\" (UID: \"f06467cc-6d14-4f96-ab33-0b3cbdc14ff4\") " pod="ai-tenant-e2e-noten-aefa94/noten-model-aefa94-kserve-78f5fb9c56-mqfmj"
Sep 19 04:42:06.517675 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:06.517632    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f06467cc-6d14-4f96-ab33-0b3cbdc14ff4-home\") pod \"noten-model-aefa94-kserve-78f5fb9c56-mqfmj\" (UID: \"f06467cc-6d14-4f96-ab33-0b3cbdc14ff4\") " pod="ai-tenant-e2e-noten-aefa94/noten-model-aefa94-kserve-78f5fb9c56-mqfmj"
Sep 19 04:42:06.517883 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:06.517846    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f06467cc-6d14-4f96-ab33-0b3cbdc14ff4-dshm\") pod \"noten-model-aefa94-kserve-78f5fb9c56-mqfmj\" (UID: \"f06467cc-6d14-4f96-ab33-0b3cbdc14ff4\") " pod="ai-tenant-e2e-noten-aefa94/noten-model-aefa94-kserve-78f5fb9c56-mqfmj"
Sep 19 04:42:06.517883 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:06.517877    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f06467cc-6d14-4f96-ab33-0b3cbdc14ff4-tls-certs\") pod \"noten-model-aefa94-kserve-78f5fb9c56-mqfmj\" (UID: \"f06467cc-6d14-4f96-ab33-0b3cbdc14ff4\") " pod="ai-tenant-e2e-noten-aefa94/noten-model-aefa94-kserve-78f5fb9c56-mqfmj"
Sep 19 04:42:06.518031 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:06.517910    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f06467cc-6d14-4f96-ab33-0b3cbdc14ff4-model-cache\") pod \"noten-model-aefa94-kserve-78f5fb9c56-mqfmj\" (UID: \"f06467cc-6d14-4f96-ab33-0b3cbdc14ff4\") " pod="ai-tenant-e2e-noten-aefa94/noten-model-aefa94-kserve-78f5fb9c56-mqfmj"
Sep 19 04:42:06.518031 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:06.517982    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f06467cc-6d14-4f96-ab33-0b3cbdc14ff4-home\") pod \"noten-model-aefa94-kserve-78f5fb9c56-mqfmj\" (UID: \"f06467cc-6d14-4f96-ab33-0b3cbdc14ff4\") " pod="ai-tenant-e2e-noten-aefa94/noten-model-aefa94-kserve-78f5fb9c56-mqfmj"
Sep 19 04:42:06.518031 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:06.518016    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdwc5\" (UniqueName: \"kubernetes.io/projected/f06467cc-6d14-4f96-ab33-0b3cbdc14ff4-kube-api-access-bdwc5\") pod \"noten-model-aefa94-kserve-78f5fb9c56-mqfmj\" (UID: \"f06467cc-6d14-4f96-ab33-0b3cbdc14ff4\") " pod="ai-tenant-e2e-noten-aefa94/noten-model-aefa94-kserve-78f5fb9c56-mqfmj"
Sep 19 04:42:06.520721 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:06.518196    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f06467cc-6d14-4f96-ab33-0b3cbdc14ff4-tmp-dir\") pod \"noten-model-aefa94-kserve-78f5fb9c56-mqfmj\" (UID: \"f06467cc-6d14-4f96-ab33-0b3cbdc14ff4\") " pod="ai-tenant-e2e-noten-aefa94/noten-model-aefa94-kserve-78f5fb9c56-mqfmj"
Sep 19 04:42:06.520721 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:06.518656    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f06467cc-6d14-4f96-ab33-0b3cbdc14ff4-tmp-dir\") pod \"noten-model-aefa94-kserve-78f5fb9c56-mqfmj\" (UID: \"f06467cc-6d14-4f96-ab33-0b3cbdc14ff4\") " pod="ai-tenant-e2e-noten-aefa94/noten-model-aefa94-kserve-78f5fb9c56-mqfmj"
Sep 19 04:42:06.520721 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:06.518953    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f06467cc-6d14-4f96-ab33-0b3cbdc14ff4-model-cache\") pod \"noten-model-aefa94-kserve-78f5fb9c56-mqfmj\" (UID: \"f06467cc-6d14-4f96-ab33-0b3cbdc14ff4\") " pod="ai-tenant-e2e-noten-aefa94/noten-model-aefa94-kserve-78f5fb9c56-mqfmj"
Sep 19 04:42:06.521324 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:06.521300    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f06467cc-6d14-4f96-ab33-0b3cbdc14ff4-dshm\") pod \"noten-model-aefa94-kserve-78f5fb9c56-mqfmj\" (UID: \"f06467cc-6d14-4f96-ab33-0b3cbdc14ff4\") " pod="ai-tenant-e2e-noten-aefa94/noten-model-aefa94-kserve-78f5fb9c56-mqfmj"
Sep 19 04:42:06.521545 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:06.521520    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f06467cc-6d14-4f96-ab33-0b3cbdc14ff4-tls-certs\") pod \"noten-model-aefa94-kserve-78f5fb9c56-mqfmj\" (UID: \"f06467cc-6d14-4f96-ab33-0b3cbdc14ff4\") " pod="ai-tenant-e2e-noten-aefa94/noten-model-aefa94-kserve-78f5fb9c56-mqfmj"
Sep 19 04:42:06.536425 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:06.536398    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdwc5\" (UniqueName: \"kubernetes.io/projected/f06467cc-6d14-4f96-ab33-0b3cbdc14ff4-kube-api-access-bdwc5\") pod \"noten-model-aefa94-kserve-78f5fb9c56-mqfmj\" (UID: \"f06467cc-6d14-4f96-ab33-0b3cbdc14ff4\") " pod="ai-tenant-e2e-noten-aefa94/noten-model-aefa94-kserve-78f5fb9c56-mqfmj"
Sep 19 04:42:06.606125 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:06.606046    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-aefa94/noten-model-aefa94-kserve-78f5fb9c56-mqfmj"
Sep 19 04:42:06.891080 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:06.891048    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-aefa94/noten-model-aefa94-kserve-78f5fb9c56-mqfmj"]
Sep 19 04:42:06.893048 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:42:06.893024    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf06467cc_6d14_4f96_ab33_0b3cbdc14ff4.slice/crio-f5118b1c68995bc64c54bbfd7384ff2ad080c0ae28819791693b1706456f54b9 WatchSource:0}: Error finding container f5118b1c68995bc64c54bbfd7384ff2ad080c0ae28819791693b1706456f54b9: Status 404 returned error can't find the container with id f5118b1c68995bc64c54bbfd7384ff2ad080c0ae28819791693b1706456f54b9
Sep 19 04:42:07.659310 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:07.659272    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-aefa94/noten-model-aefa94-kserve-78f5fb9c56-mqfmj" event={"ID":"f06467cc-6d14-4f96-ab33-0b3cbdc14ff4","Type":"ContainerStarted","Data":"31679db2329617babf151efdfb7aa5af358e6150d0965d9c8cd7f7d2eaa68247"}
Sep 19 04:42:07.659798 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:07.659323    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-aefa94/noten-model-aefa94-kserve-78f5fb9c56-mqfmj" event={"ID":"f06467cc-6d14-4f96-ab33-0b3cbdc14ff4","Type":"ContainerStarted","Data":"f5118b1c68995bc64c54bbfd7384ff2ad080c0ae28819791693b1706456f54b9"}
Sep 19 04:42:07.692278 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:07.692221    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-aefa94/noten-model-aefa94-kserve-78f5fb9c56-mqfmj" podStartSLOduration=1.692203809 podStartE2EDuration="1.692203809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 04:42:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:42:07.690759224 +0000 UTC m=+1706.873720372" watchObservedRunningTime="2026-09-19 04:42:07.692203809 +0000 UTC m=+1706.875165002"
Sep 19 04:42:08.606150 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:08.606117    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-aefa94/noten-model-aefa94-kserve-78f5fb9c56-mqfmj"
Sep 19 04:42:11.415037 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:11.414996    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bcbdad-smwbz"]
Sep 19 04:42:11.420433 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:11.420402    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bcbdad-smwbz"]
Sep 19 04:42:13.352521 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:13.352491    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ab7c5a7-4566-4e16-a8f6-cf561d5855f4" path="/var/lib/kubelet/pods/6ab7c5a7-4566-4e16-a8f6-cf561d5855f4/volumes"
Sep 19 04:42:16.606787 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:16.606741    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-aefa94/noten-model-aefa94-kserve-78f5fb9c56-mqfmj"
Sep 19 04:42:16.619211 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:16.619185    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-aefa94/noten-model-aefa94-kserve-78f5fb9c56-mqfmj"
Sep 19 04:42:16.729988 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:16.729958    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-aefa94/noten-model-aefa94-kserve-78f5fb9c56-mqfmj"
Sep 19 04:42:19.209612 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:19.209565    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6b1935-s64s4"]
Sep 19 04:42:19.215050 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:19.215011    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6b1935-s64s4"
Sep 19 04:42:19.217778 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:19.217745    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 19 04:42:19.217952 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:19.217754    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s6xbq\""
Sep 19 04:42:19.224112 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:19.224083    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6b1935-s64s4"]
Sep 19 04:42:19.354759 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:19.354728    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e715774-6df7-4642-a0e8-a5519b394cea-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6b1935-s64s4\" (UID: \"7e715774-6df7-4642-a0e8-a5519b394cea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6b1935-s64s4"
Sep 19 04:42:19.455654 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:19.455612    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e715774-6df7-4642-a0e8-a5519b394cea-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6b1935-s64s4\" (UID: \"7e715774-6df7-4642-a0e8-a5519b394cea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6b1935-s64s4"
Sep 19 04:42:19.456341 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:19.456310    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e715774-6df7-4642-a0e8-a5519b394cea-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6b1935-s64s4\" (UID: \"7e715774-6df7-4642-a0e8-a5519b394cea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6b1935-s64s4"
Sep 19 04:42:19.530984 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:19.530863    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6b1935-s64s4"
Sep 19 04:42:19.829114 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:19.829016    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6b1935-s64s4"]
Sep 19 04:42:19.833313 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:42:19.833274    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e715774_6df7_4642_a0e8_a5519b394cea.slice/crio-b684dd589cf846641490d19f337951d7b5030ca98ba7810d3fba714f704091aa WatchSource:0}: Error finding container b684dd589cf846641490d19f337951d7b5030ca98ba7810d3fba714f704091aa: Status 404 returned error can't find the container with id b684dd589cf846641490d19f337951d7b5030ca98ba7810d3fba714f704091aa
Sep 19 04:42:20.747726 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:20.747670    2579 generic.go:356] "Generic (PLEG): container finished" podID="7e715774-6df7-4642-a0e8-a5519b394cea" containerID="89ffce6f354c6cebba8024128404ade1e2b591a415625a493a411f5ad22d2f22" exitCode=0
Sep 19 04:42:20.748117 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:20.747736    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6b1935-s64s4" event={"ID":"7e715774-6df7-4642-a0e8-a5519b394cea","Type":"ContainerDied","Data":"89ffce6f354c6cebba8024128404ade1e2b591a415625a493a411f5ad22d2f22"}
Sep 19 04:42:20.748117 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:20.747773    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6b1935-s64s4" event={"ID":"7e715774-6df7-4642-a0e8-a5519b394cea","Type":"ContainerStarted","Data":"b684dd589cf846641490d19f337951d7b5030ca98ba7810d3fba714f704091aa"}
Sep 19 04:42:21.904619 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:21.904595    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6b1935-s64s4"
Sep 19 04:42:21.980870 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:21.980837    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e715774-6df7-4642-a0e8-a5519b394cea-maas-api-service-ca\") pod \"7e715774-6df7-4642-a0e8-a5519b394cea\" (UID: \"7e715774-6df7-4642-a0e8-a5519b394cea\") "
Sep 19 04:42:21.981200 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:21.981177    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e715774-6df7-4642-a0e8-a5519b394cea-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7e715774-6df7-4642-a0e8-a5519b394cea" (UID: "7e715774-6df7-4642-a0e8-a5519b394cea"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 04:42:22.082604 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:22.082503    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e715774-6df7-4642-a0e8-a5519b394cea-maas-api-service-ca\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:42:22.764516 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:22.764473    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6b1935-s64s4" event={"ID":"7e715774-6df7-4642-a0e8-a5519b394cea","Type":"ContainerDied","Data":"b684dd589cf846641490d19f337951d7b5030ca98ba7810d3fba714f704091aa"}
Sep 19 04:42:22.764779 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:22.764492    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6b1935-s64s4"
Sep 19 04:42:22.764779 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:22.764560    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b684dd589cf846641490d19f337951d7b5030ca98ba7810d3fba714f704091aa"
Sep 19 04:42:29.880824 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:29.880788    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6b1935-7b7d4bc579-l6sgg"]
Sep 19 04:42:29.881326 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:29.881049    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-6b1935-7b7d4bc579-l6sgg" podUID="ec508902-bbc1-4675-85c4-535eb8ba6d99" containerName="payload-processing" containerID="cri-o://86e5a34bdf13674ee28fefc4183ec01bc9018bf06c77ec306fd688c1174468fb" gracePeriod=30
Sep 19 04:42:30.154902 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:30.154575    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6b1935-s64s4"]
Sep 19 04:42:30.157585 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:30.157548    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6b1935-s64s4"]
Sep 19 04:42:30.166994 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:30.166968    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-6b1935-7b7d4bc579-l6sgg"
Sep 19 04:42:30.273389 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:30.273355    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h58v7\" (UniqueName: \"kubernetes.io/projected/ec508902-bbc1-4675-85c4-535eb8ba6d99-kube-api-access-h58v7\") pod \"ec508902-bbc1-4675-85c4-535eb8ba6d99\" (UID: \"ec508902-bbc1-4675-85c4-535eb8ba6d99\") "
Sep 19 04:42:30.273580 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:30.273429    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec508902-bbc1-4675-85c4-535eb8ba6d99-plugins-config-volume\") pod \"ec508902-bbc1-4675-85c4-535eb8ba6d99\" (UID: \"ec508902-bbc1-4675-85c4-535eb8ba6d99\") "
Sep 19 04:42:30.273861 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:30.273836    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec508902-bbc1-4675-85c4-535eb8ba6d99-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ec508902-bbc1-4675-85c4-535eb8ba6d99" (UID: "ec508902-bbc1-4675-85c4-535eb8ba6d99"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 04:42:30.275590 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:30.275567    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec508902-bbc1-4675-85c4-535eb8ba6d99-kube-api-access-h58v7" (OuterVolumeSpecName: "kube-api-access-h58v7") pod "ec508902-bbc1-4675-85c4-535eb8ba6d99" (UID: "ec508902-bbc1-4675-85c4-535eb8ba6d99"). InnerVolumeSpecName "kube-api-access-h58v7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 04:42:30.374951 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:30.374908    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h58v7\" (UniqueName: \"kubernetes.io/projected/ec508902-bbc1-4675-85c4-535eb8ba6d99-kube-api-access-h58v7\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:42:30.374951 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:30.374944    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec508902-bbc1-4675-85c4-535eb8ba6d99-plugins-config-volume\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:42:30.820760 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:30.820728    2579 generic.go:356] "Generic (PLEG): container finished" podID="ec508902-bbc1-4675-85c4-535eb8ba6d99" containerID="86e5a34bdf13674ee28fefc4183ec01bc9018bf06c77ec306fd688c1174468fb" exitCode=0
Sep 19 04:42:30.820760 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:30.820766    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-6b1935-7b7d4bc579-l6sgg" event={"ID":"ec508902-bbc1-4675-85c4-535eb8ba6d99","Type":"ContainerDied","Data":"86e5a34bdf13674ee28fefc4183ec01bc9018bf06c77ec306fd688c1174468fb"}
Sep 19 04:42:30.821023 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:30.820784    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-6b1935-7b7d4bc579-l6sgg"
Sep 19 04:42:30.821023 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:30.820805    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-6b1935-7b7d4bc579-l6sgg" event={"ID":"ec508902-bbc1-4675-85c4-535eb8ba6d99","Type":"ContainerDied","Data":"16108035eaa11f242a59192071fffc2951f199b6784e883cf4ccd6eca96c8d77"}
Sep 19 04:42:30.821023 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:30.820827    2579 scope.go:117] "RemoveContainer" containerID="86e5a34bdf13674ee28fefc4183ec01bc9018bf06c77ec306fd688c1174468fb"
Sep 19 04:42:30.831103 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:30.831080    2579 scope.go:117] "RemoveContainer" containerID="86e5a34bdf13674ee28fefc4183ec01bc9018bf06c77ec306fd688c1174468fb"
Sep 19 04:42:30.831389 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:42:30.831367    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86e5a34bdf13674ee28fefc4183ec01bc9018bf06c77ec306fd688c1174468fb\": container with ID starting with 86e5a34bdf13674ee28fefc4183ec01bc9018bf06c77ec306fd688c1174468fb not found: ID does not exist" containerID="86e5a34bdf13674ee28fefc4183ec01bc9018bf06c77ec306fd688c1174468fb"
Sep 19 04:42:30.831432 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:30.831400    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86e5a34bdf13674ee28fefc4183ec01bc9018bf06c77ec306fd688c1174468fb"} err="failed to get container status \"86e5a34bdf13674ee28fefc4183ec01bc9018bf06c77ec306fd688c1174468fb\": rpc error: code = NotFound desc = could not find container \"86e5a34bdf13674ee28fefc4183ec01bc9018bf06c77ec306fd688c1174468fb\": container with ID starting with 86e5a34bdf13674ee28fefc4183ec01bc9018bf06c77ec306fd688c1174468fb not found: ID does not exist"
Sep 19 04:42:30.857289 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:30.857257    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6b1935-7b7d4bc579-l6sgg"]
Sep 19 04:42:30.863775 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:30.863745    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6b1935-7b7d4bc579-l6sgg"]
Sep 19 04:42:31.359089 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:31.359055    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e715774-6df7-4642-a0e8-a5519b394cea" path="/var/lib/kubelet/pods/7e715774-6df7-4642-a0e8-a5519b394cea/volumes"
Sep 19 04:42:31.359506 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:31.359491    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec508902-bbc1-4675-85c4-535eb8ba6d99" path="/var/lib/kubelet/pods/ec508902-bbc1-4675-85c4-535eb8ba6d99/volumes"
Sep 19 04:42:38.923463 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:38.923424    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-87e3e6/distinct-2-w0-87e3e6-kserve-7694d86767-6sz6p"]
Sep 19 04:42:38.925179 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:38.925145    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec508902-bbc1-4675-85c4-535eb8ba6d99" containerName="payload-processing"
Sep 19 04:42:38.925179 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:38.925176    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec508902-bbc1-4675-85c4-535eb8ba6d99" containerName="payload-processing"
Sep 19 04:42:38.925413 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:38.925193    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e715774-6df7-4642-a0e8-a5519b394cea" containerName="revoke-keys"
Sep 19 04:42:38.925413 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:38.925202    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e715774-6df7-4642-a0e8-a5519b394cea" containerName="revoke-keys"
Sep 19 04:42:38.925413 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:38.925382    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec508902-bbc1-4675-85c4-535eb8ba6d99" containerName="payload-processing"
Sep 19 04:42:38.925413 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:38.925394    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e715774-6df7-4642-a0e8-a5519b394cea" containerName="revoke-keys"
Sep 19 04:42:38.930678 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:38.930656    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-87e3e6/distinct-2-w0-87e3e6-kserve-7694d86767-6sz6p"
Sep 19 04:42:38.938302 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:38.938283    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-87e3e6\"/\"distinct-2-w0-87e3e6-kserve-self-signed-certs\""
Sep 19 04:42:38.943533 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:38.943507    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-87e3e6/distinct-2-w0-87e3e6-kserve-7694d86767-6sz6p"]
Sep 19 04:42:39.062943 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:39.062913    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98309c08-4d67-45bc-9848-ec80ec181224-model-cache\") pod \"distinct-2-w0-87e3e6-kserve-7694d86767-6sz6p\" (UID: \"98309c08-4d67-45bc-9848-ec80ec181224\") " pod="e2e-models-e2e-worker-w0-87e3e6/distinct-2-w0-87e3e6-kserve-7694d86767-6sz6p"
Sep 19 04:42:39.062943 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:39.062946    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98309c08-4d67-45bc-9848-ec80ec181224-home\") pod \"distinct-2-w0-87e3e6-kserve-7694d86767-6sz6p\" (UID: \"98309c08-4d67-45bc-9848-ec80ec181224\") " pod="e2e-models-e2e-worker-w0-87e3e6/distinct-2-w0-87e3e6-kserve-7694d86767-6sz6p"
Sep 19 04:42:39.063146 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:39.062981    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8f9k\" (UniqueName: \"kubernetes.io/projected/98309c08-4d67-45bc-9848-ec80ec181224-kube-api-access-s8f9k\") pod \"distinct-2-w0-87e3e6-kserve-7694d86767-6sz6p\" (UID: \"98309c08-4d67-45bc-9848-ec80ec181224\") " pod="e2e-models-e2e-worker-w0-87e3e6/distinct-2-w0-87e3e6-kserve-7694d86767-6sz6p"
Sep 19 04:42:39.063146 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:39.063002    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98309c08-4d67-45bc-9848-ec80ec181224-tmp-dir\") pod \"distinct-2-w0-87e3e6-kserve-7694d86767-6sz6p\" (UID: \"98309c08-4d67-45bc-9848-ec80ec181224\") " pod="e2e-models-e2e-worker-w0-87e3e6/distinct-2-w0-87e3e6-kserve-7694d86767-6sz6p"
Sep 19 04:42:39.063146 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:39.063046    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98309c08-4d67-45bc-9848-ec80ec181224-dshm\") pod \"distinct-2-w0-87e3e6-kserve-7694d86767-6sz6p\" (UID: \"98309c08-4d67-45bc-9848-ec80ec181224\") " pod="e2e-models-e2e-worker-w0-87e3e6/distinct-2-w0-87e3e6-kserve-7694d86767-6sz6p"
Sep 19 04:42:39.063146 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:39.063094    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98309c08-4d67-45bc-9848-ec80ec181224-tls-certs\") pod \"distinct-2-w0-87e3e6-kserve-7694d86767-6sz6p\" (UID: \"98309c08-4d67-45bc-9848-ec80ec181224\") " pod="e2e-models-e2e-worker-w0-87e3e6/distinct-2-w0-87e3e6-kserve-7694d86767-6sz6p"
Sep 19 04:42:39.164457 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:39.164416    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98309c08-4d67-45bc-9848-ec80ec181224-dshm\") pod \"distinct-2-w0-87e3e6-kserve-7694d86767-6sz6p\" (UID: \"98309c08-4d67-45bc-9848-ec80ec181224\") " pod="e2e-models-e2e-worker-w0-87e3e6/distinct-2-w0-87e3e6-kserve-7694d86767-6sz6p"
Sep 19 04:42:39.164650 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:39.164482    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98309c08-4d67-45bc-9848-ec80ec181224-tls-certs\") pod \"distinct-2-w0-87e3e6-kserve-7694d86767-6sz6p\" (UID: \"98309c08-4d67-45bc-9848-ec80ec181224\") " pod="e2e-models-e2e-worker-w0-87e3e6/distinct-2-w0-87e3e6-kserve-7694d86767-6sz6p"
Sep 19 04:42:39.164650 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:39.164533    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98309c08-4d67-45bc-9848-ec80ec181224-model-cache\") pod \"distinct-2-w0-87e3e6-kserve-7694d86767-6sz6p\" (UID: \"98309c08-4d67-45bc-9848-ec80ec181224\") " pod="e2e-models-e2e-worker-w0-87e3e6/distinct-2-w0-87e3e6-kserve-7694d86767-6sz6p"
Sep 19 04:42:39.164650 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:39.164551    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98309c08-4d67-45bc-9848-ec80ec181224-home\") pod \"distinct-2-w0-87e3e6-kserve-7694d86767-6sz6p\" (UID: \"98309c08-4d67-45bc-9848-ec80ec181224\") " pod="e2e-models-e2e-worker-w0-87e3e6/distinct-2-w0-87e3e6-kserve-7694d86767-6sz6p"
Sep 19 04:42:39.164851 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:39.164714    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8f9k\" (UniqueName: \"kubernetes.io/projected/98309c08-4d67-45bc-9848-ec80ec181224-kube-api-access-s8f9k\") pod \"distinct-2-w0-87e3e6-kserve-7694d86767-6sz6p\" (UID: \"98309c08-4d67-45bc-9848-ec80ec181224\") " pod="e2e-models-e2e-worker-w0-87e3e6/distinct-2-w0-87e3e6-kserve-7694d86767-6sz6p"
Sep 19 04:42:39.164851 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:39.164763    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98309c08-4d67-45bc-9848-ec80ec181224-tmp-dir\") pod \"distinct-2-w0-87e3e6-kserve-7694d86767-6sz6p\" (UID: \"98309c08-4d67-45bc-9848-ec80ec181224\") " pod="e2e-models-e2e-worker-w0-87e3e6/distinct-2-w0-87e3e6-kserve-7694d86767-6sz6p"
Sep 19 04:42:39.165025 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:39.164996    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98309c08-4d67-45bc-9848-ec80ec181224-model-cache\") pod \"distinct-2-w0-87e3e6-kserve-7694d86767-6sz6p\" (UID: \"98309c08-4d67-45bc-9848-ec80ec181224\") " pod="e2e-models-e2e-worker-w0-87e3e6/distinct-2-w0-87e3e6-kserve-7694d86767-6sz6p"
Sep 19 04:42:39.165131 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:39.165025    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98309c08-4d67-45bc-9848-ec80ec181224-home\") pod \"distinct-2-w0-87e3e6-kserve-7694d86767-6sz6p\" (UID: \"98309c08-4d67-45bc-9848-ec80ec181224\") " pod="e2e-models-e2e-worker-w0-87e3e6/distinct-2-w0-87e3e6-kserve-7694d86767-6sz6p"
Sep 19 04:42:39.165131 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:39.165095    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98309c08-4d67-45bc-9848-ec80ec181224-tmp-dir\") pod \"distinct-2-w0-87e3e6-kserve-7694d86767-6sz6p\" (UID: \"98309c08-4d67-45bc-9848-ec80ec181224\") " pod="e2e-models-e2e-worker-w0-87e3e6/distinct-2-w0-87e3e6-kserve-7694d86767-6sz6p"
Sep 19 04:42:39.166936 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:39.166917    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98309c08-4d67-45bc-9848-ec80ec181224-dshm\") pod \"distinct-2-w0-87e3e6-kserve-7694d86767-6sz6p\" (UID: \"98309c08-4d67-45bc-9848-ec80ec181224\") " pod="e2e-models-e2e-worker-w0-87e3e6/distinct-2-w0-87e3e6-kserve-7694d86767-6sz6p"
Sep 19 04:42:39.167177 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:39.167161    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98309c08-4d67-45bc-9848-ec80ec181224-tls-certs\") pod \"distinct-2-w0-87e3e6-kserve-7694d86767-6sz6p\" (UID: \"98309c08-4d67-45bc-9848-ec80ec181224\") " pod="e2e-models-e2e-worker-w0-87e3e6/distinct-2-w0-87e3e6-kserve-7694d86767-6sz6p"
Sep 19 04:42:39.175042 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:39.174980    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8f9k\" (UniqueName: \"kubernetes.io/projected/98309c08-4d67-45bc-9848-ec80ec181224-kube-api-access-s8f9k\") pod \"distinct-2-w0-87e3e6-kserve-7694d86767-6sz6p\" (UID: \"98309c08-4d67-45bc-9848-ec80ec181224\") " pod="e2e-models-e2e-worker-w0-87e3e6/distinct-2-w0-87e3e6-kserve-7694d86767-6sz6p"
Sep 19 04:42:39.240107 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:39.240061    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-87e3e6/distinct-2-w0-87e3e6-kserve-7694d86767-6sz6p"
Sep 19 04:42:39.524587 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:39.524561    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-87e3e6/distinct-2-w0-87e3e6-kserve-7694d86767-6sz6p"]
Sep 19 04:42:39.525879 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:42:39.525849    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod98309c08_4d67_45bc_9848_ec80ec181224.slice/crio-8ad8a9517274808d206933ef5fec1fd8dc0ccda51a7f3454285bf5e82a95ac59 WatchSource:0}: Error finding container 8ad8a9517274808d206933ef5fec1fd8dc0ccda51a7f3454285bf5e82a95ac59: Status 404 returned error can't find the container with id 8ad8a9517274808d206933ef5fec1fd8dc0ccda51a7f3454285bf5e82a95ac59
Sep 19 04:42:39.889139 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:39.888137    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-87e3e6/distinct-2-w0-87e3e6-kserve-7694d86767-6sz6p" event={"ID":"98309c08-4d67-45bc-9848-ec80ec181224","Type":"ContainerStarted","Data":"4afb6217682191470e34b1df907f36996b64787ff7ec2d9a4ec06d64374b440a"}
Sep 19 04:42:39.889139 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:39.888189    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-87e3e6/distinct-2-w0-87e3e6-kserve-7694d86767-6sz6p" event={"ID":"98309c08-4d67-45bc-9848-ec80ec181224","Type":"ContainerStarted","Data":"8ad8a9517274808d206933ef5fec1fd8dc0ccda51a7f3454285bf5e82a95ac59"}
Sep 19 04:42:39.920442 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:39.920393    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-87e3e6/distinct-2-w0-87e3e6-kserve-7694d86767-6sz6p" podStartSLOduration=1.920376681 podStartE2EDuration="1.920376681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 04:42:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:42:39.918049637 +0000 UTC m=+1739.101010785" watchObservedRunningTime="2026-09-19 04:42:39.920376681 +0000 UTC m=+1739.103337828"
Sep 19 04:42:40.240898 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:40.240862    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-87e3e6/distinct-2-w0-87e3e6-kserve-7694d86767-6sz6p"
Sep 19 04:42:41.545048 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:41.545021    2579 scope.go:117] "RemoveContainer" containerID="a7955c60484c221712e9ab739f7c963426a9c4e6ac94ca231d83afc9a566b84f"
Sep 19 04:42:41.555832 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:41.555812    2579 scope.go:117] "RemoveContainer" containerID="2183a12d05180f60cd0b279966f0e861750bebc9f5140d7cb691e4a22ff214e1"
Sep 19 04:42:41.565468 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:41.565442    2579 scope.go:117] "RemoveContainer" containerID="ef52a76e4019c044f63ea87f72fe83b713593d68619abcd7bdbc83f38af0facb"
Sep 19 04:42:45.249399 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:45.249368    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-aefa94/noten-model-aefa94-kserve-78f5fb9c56-mqfmj"]
Sep 19 04:42:49.240184 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:49.240141    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-87e3e6/distinct-2-w0-87e3e6-kserve-7694d86767-6sz6p"
Sep 19 04:42:49.253159 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:49.253130    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-87e3e6/distinct-2-w0-87e3e6-kserve-7694d86767-6sz6p"
Sep 19 04:42:49.966809 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:42:49.966781    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-87e3e6/distinct-2-w0-87e3e6-kserve-7694d86767-6sz6p"
Sep 19 04:43:00.269050 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:00.268992    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-aefa94/noten-model-aefa94-kserve-78f5fb9c56-mqfmj" podUID="f06467cc-6d14-4f96-ab33-0b3cbdc14ff4" containerName="main" containerID="cri-o://31679db2329617babf151efdfb7aa5af358e6150d0965d9c8cd7f7d2eaa68247" gracePeriod=45
Sep 19 04:43:00.542146 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:00.542122    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-aefa94/noten-model-aefa94-kserve-78f5fb9c56-mqfmj"
Sep 19 04:43:00.690881 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:00.690849    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f06467cc-6d14-4f96-ab33-0b3cbdc14ff4-dshm\") pod \"f06467cc-6d14-4f96-ab33-0b3cbdc14ff4\" (UID: \"f06467cc-6d14-4f96-ab33-0b3cbdc14ff4\") "
Sep 19 04:43:00.691066 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:00.690900    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bdwc5\" (UniqueName: \"kubernetes.io/projected/f06467cc-6d14-4f96-ab33-0b3cbdc14ff4-kube-api-access-bdwc5\") pod \"f06467cc-6d14-4f96-ab33-0b3cbdc14ff4\" (UID: \"f06467cc-6d14-4f96-ab33-0b3cbdc14ff4\") "
Sep 19 04:43:00.691066 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:00.690958    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f06467cc-6d14-4f96-ab33-0b3cbdc14ff4-tmp-dir\") pod \"f06467cc-6d14-4f96-ab33-0b3cbdc14ff4\" (UID: \"f06467cc-6d14-4f96-ab33-0b3cbdc14ff4\") "
Sep 19 04:43:00.691066 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:00.690992    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f06467cc-6d14-4f96-ab33-0b3cbdc14ff4-tls-certs\") pod \"f06467cc-6d14-4f96-ab33-0b3cbdc14ff4\" (UID: \"f06467cc-6d14-4f96-ab33-0b3cbdc14ff4\") "
Sep 19 04:43:00.691066 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:00.691034    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f06467cc-6d14-4f96-ab33-0b3cbdc14ff4-home\") pod \"f06467cc-6d14-4f96-ab33-0b3cbdc14ff4\" (UID: \"f06467cc-6d14-4f96-ab33-0b3cbdc14ff4\") "
Sep 19 04:43:00.691066 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:00.691051    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f06467cc-6d14-4f96-ab33-0b3cbdc14ff4-model-cache\") pod \"f06467cc-6d14-4f96-ab33-0b3cbdc14ff4\" (UID: \"f06467cc-6d14-4f96-ab33-0b3cbdc14ff4\") "
Sep 19 04:43:00.691401 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:00.691242    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f06467cc-6d14-4f96-ab33-0b3cbdc14ff4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f06467cc-6d14-4f96-ab33-0b3cbdc14ff4" (UID: "f06467cc-6d14-4f96-ab33-0b3cbdc14ff4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:43:00.691401 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:00.691355    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f06467cc-6d14-4f96-ab33-0b3cbdc14ff4-home" (OuterVolumeSpecName: "home") pod "f06467cc-6d14-4f96-ab33-0b3cbdc14ff4" (UID: "f06467cc-6d14-4f96-ab33-0b3cbdc14ff4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:43:00.691508 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:00.691440    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f06467cc-6d14-4f96-ab33-0b3cbdc14ff4-model-cache" (OuterVolumeSpecName: "model-cache") pod "f06467cc-6d14-4f96-ab33-0b3cbdc14ff4" (UID: "f06467cc-6d14-4f96-ab33-0b3cbdc14ff4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:43:00.691567 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:00.691502    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f06467cc-6d14-4f96-ab33-0b3cbdc14ff4-home\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:43:00.691567 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:00.691522    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f06467cc-6d14-4f96-ab33-0b3cbdc14ff4-tmp-dir\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:43:00.693368 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:00.693340    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f06467cc-6d14-4f96-ab33-0b3cbdc14ff4-kube-api-access-bdwc5" (OuterVolumeSpecName: "kube-api-access-bdwc5") pod "f06467cc-6d14-4f96-ab33-0b3cbdc14ff4" (UID: "f06467cc-6d14-4f96-ab33-0b3cbdc14ff4"). InnerVolumeSpecName "kube-api-access-bdwc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 04:43:00.693368 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:00.693345    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f06467cc-6d14-4f96-ab33-0b3cbdc14ff4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f06467cc-6d14-4f96-ab33-0b3cbdc14ff4" (UID: "f06467cc-6d14-4f96-ab33-0b3cbdc14ff4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 04:43:00.693512 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:00.693386    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f06467cc-6d14-4f96-ab33-0b3cbdc14ff4-dshm" (OuterVolumeSpecName: "dshm") pod "f06467cc-6d14-4f96-ab33-0b3cbdc14ff4" (UID: "f06467cc-6d14-4f96-ab33-0b3cbdc14ff4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:43:00.793023 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:00.792922    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f06467cc-6d14-4f96-ab33-0b3cbdc14ff4-dshm\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:43:00.793023 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:00.792960    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdwc5\" (UniqueName: \"kubernetes.io/projected/f06467cc-6d14-4f96-ab33-0b3cbdc14ff4-kube-api-access-bdwc5\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:43:00.793023 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:00.792975    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f06467cc-6d14-4f96-ab33-0b3cbdc14ff4-tls-certs\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:43:00.793023 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:00.792988    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f06467cc-6d14-4f96-ab33-0b3cbdc14ff4-model-cache\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:43:01.023675 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:01.023628    2579 generic.go:356] "Generic (PLEG): container finished" podID="f06467cc-6d14-4f96-ab33-0b3cbdc14ff4" containerID="31679db2329617babf151efdfb7aa5af358e6150d0965d9c8cd7f7d2eaa68247" exitCode=0
Sep 19 04:43:01.023675 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:01.023675    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-aefa94/noten-model-aefa94-kserve-78f5fb9c56-mqfmj"
Sep 19 04:43:01.023977 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:01.023735    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-aefa94/noten-model-aefa94-kserve-78f5fb9c56-mqfmj" event={"ID":"f06467cc-6d14-4f96-ab33-0b3cbdc14ff4","Type":"ContainerDied","Data":"31679db2329617babf151efdfb7aa5af358e6150d0965d9c8cd7f7d2eaa68247"}
Sep 19 04:43:01.023977 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:01.023780    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-aefa94/noten-model-aefa94-kserve-78f5fb9c56-mqfmj" event={"ID":"f06467cc-6d14-4f96-ab33-0b3cbdc14ff4","Type":"ContainerDied","Data":"f5118b1c68995bc64c54bbfd7384ff2ad080c0ae28819791693b1706456f54b9"}
Sep 19 04:43:01.023977 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:01.023798    2579 scope.go:117] "RemoveContainer" containerID="31679db2329617babf151efdfb7aa5af358e6150d0965d9c8cd7f7d2eaa68247"
Sep 19 04:43:01.033510 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:01.033495    2579 scope.go:117] "RemoveContainer" containerID="31679db2329617babf151efdfb7aa5af358e6150d0965d9c8cd7f7d2eaa68247"
Sep 19 04:43:01.033819 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:43:01.033801    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31679db2329617babf151efdfb7aa5af358e6150d0965d9c8cd7f7d2eaa68247\": container with ID starting with 31679db2329617babf151efdfb7aa5af358e6150d0965d9c8cd7f7d2eaa68247 not found: ID does not exist" containerID="31679db2329617babf151efdfb7aa5af358e6150d0965d9c8cd7f7d2eaa68247"
Sep 19 04:43:01.033879 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:01.033829    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31679db2329617babf151efdfb7aa5af358e6150d0965d9c8cd7f7d2eaa68247"} err="failed to get container status \"31679db2329617babf151efdfb7aa5af358e6150d0965d9c8cd7f7d2eaa68247\": rpc error: code = NotFound desc = could not find container \"31679db2329617babf151efdfb7aa5af358e6150d0965d9c8cd7f7d2eaa68247\": container with ID starting with 31679db2329617babf151efdfb7aa5af358e6150d0965d9c8cd7f7d2eaa68247 not found: ID does not exist"
Sep 19 04:43:01.047431 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:01.047365    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-aefa94/noten-model-aefa94-kserve-78f5fb9c56-mqfmj"]
Sep 19 04:43:01.054165 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:01.054139    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-aefa94/noten-model-aefa94-kserve-78f5fb9c56-mqfmj"]
Sep 19 04:43:01.353848 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:01.353775    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f06467cc-6d14-4f96-ab33-0b3cbdc14ff4" path="/var/lib/kubelet/pods/f06467cc-6d14-4f96-ab33-0b3cbdc14ff4/volumes"
Sep 19 04:43:24.130439 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:24.130396    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8bbcca-5c75fd676b-jc48f"]
Sep 19 04:43:24.131056 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:24.131037    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f06467cc-6d14-4f96-ab33-0b3cbdc14ff4" containerName="main"
Sep 19 04:43:24.131144 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:24.131058    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f06467cc-6d14-4f96-ab33-0b3cbdc14ff4" containerName="main"
Sep 19 04:43:24.131201 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:24.131154    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f06467cc-6d14-4f96-ab33-0b3cbdc14ff4" containerName="main"
Sep 19 04:43:24.134492 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:24.134473    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8bbcca-5c75fd676b-jc48f"
Sep 19 04:43:24.137921 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:24.137899    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-8bbcca\""
Sep 19 04:43:24.157687 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:24.157662    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8bbcca-5c75fd676b-jc48f"]
Sep 19 04:43:24.233453 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:24.233419    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa764d71-eb0c-4967-b58c-a6fe70ccc4a9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-8bbcca-5c75fd676b-jc48f\" (UID: \"aa764d71-eb0c-4967-b58c-a6fe70ccc4a9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8bbcca-5c75fd676b-jc48f"
Sep 19 04:43:24.233647 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:24.233471    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxp94\" (UniqueName: \"kubernetes.io/projected/aa764d71-eb0c-4967-b58c-a6fe70ccc4a9-kube-api-access-pxp94\") pod \"payload-pre-processing-e2e-shared-b-w3-8bbcca-5c75fd676b-jc48f\" (UID: \"aa764d71-eb0c-4967-b58c-a6fe70ccc4a9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8bbcca-5c75fd676b-jc48f"
Sep 19 04:43:24.334170 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:24.334135    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa764d71-eb0c-4967-b58c-a6fe70ccc4a9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-8bbcca-5c75fd676b-jc48f\" (UID: \"aa764d71-eb0c-4967-b58c-a6fe70ccc4a9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8bbcca-5c75fd676b-jc48f"
Sep 19 04:43:24.334362 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:24.334179    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxp94\" (UniqueName: \"kubernetes.io/projected/aa764d71-eb0c-4967-b58c-a6fe70ccc4a9-kube-api-access-pxp94\") pod \"payload-pre-processing-e2e-shared-b-w3-8bbcca-5c75fd676b-jc48f\" (UID: \"aa764d71-eb0c-4967-b58c-a6fe70ccc4a9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8bbcca-5c75fd676b-jc48f"
Sep 19 04:43:24.334931 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:24.334901    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa764d71-eb0c-4967-b58c-a6fe70ccc4a9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-8bbcca-5c75fd676b-jc48f\" (UID: \"aa764d71-eb0c-4967-b58c-a6fe70ccc4a9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8bbcca-5c75fd676b-jc48f"
Sep 19 04:43:24.344996 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:24.344969    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxp94\" (UniqueName: \"kubernetes.io/projected/aa764d71-eb0c-4967-b58c-a6fe70ccc4a9-kube-api-access-pxp94\") pod \"payload-pre-processing-e2e-shared-b-w3-8bbcca-5c75fd676b-jc48f\" (UID: \"aa764d71-eb0c-4967-b58c-a6fe70ccc4a9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8bbcca-5c75fd676b-jc48f"
Sep 19 04:43:24.443797 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:24.443758    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8bbcca-5c75fd676b-jc48f"
Sep 19 04:43:24.728229 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:24.728200    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8bbcca-5c75fd676b-jc48f"]
Sep 19 04:43:24.729449 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:43:24.729423    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa764d71_eb0c_4967_b58c_a6fe70ccc4a9.slice/crio-5ff0ddbb091e0cbcdd0bbf5812b85c54adeb29728127bd79de179344d16703a1 WatchSource:0}: Error finding container 5ff0ddbb091e0cbcdd0bbf5812b85c54adeb29728127bd79de179344d16703a1: Status 404 returned error can't find the container with id 5ff0ddbb091e0cbcdd0bbf5812b85c54adeb29728127bd79de179344d16703a1
Sep 19 04:43:25.201264 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:25.201223    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8bbcca-5c75fd676b-jc48f" event={"ID":"aa764d71-eb0c-4967-b58c-a6fe70ccc4a9","Type":"ContainerStarted","Data":"9633d2df8d8fbcc5c6cc6af31db40f887d86e2d9b48ff95103e7e1f9cb088eb4"}
Sep 19 04:43:25.201659 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:25.201284    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8bbcca-5c75fd676b-jc48f" event={"ID":"aa764d71-eb0c-4967-b58c-a6fe70ccc4a9","Type":"ContainerStarted","Data":"5ff0ddbb091e0cbcdd0bbf5812b85c54adeb29728127bd79de179344d16703a1"}
Sep 19 04:43:25.201659 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:25.201334    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8bbcca-5c75fd676b-jc48f"
Sep 19 04:43:25.243152 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:25.243102    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8bbcca-5c75fd676b-jc48f" podStartSLOduration=1.243085625 podStartE2EDuration="1.243085625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 04:43:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:43:25.242081413 +0000 UTC m=+1784.425042562" watchObservedRunningTime="2026-09-19 04:43:25.243085625 +0000 UTC m=+1784.426046773"
Sep 19 04:43:26.579977 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:26.579941    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a0ed681d-vfwqb"]
Sep 19 04:43:26.583796 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:26.583767    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a0ed681d-vfwqb"
Sep 19 04:43:26.586803 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:26.586770    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 19 04:43:26.586944 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:26.586770    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s6xbq\""
Sep 19 04:43:26.594364 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:26.594340    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a0ed681d-vfwqb"]
Sep 19 04:43:26.661451 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:26.661412    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d73e5e22-d2e4-42cd-8a29-a30d467ec3a0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-a0ed681d-vfwqb\" (UID: \"d73e5e22-d2e4-42cd-8a29-a30d467ec3a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a0ed681d-vfwqb"
Sep 19 04:43:26.762542 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:26.762509    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d73e5e22-d2e4-42cd-8a29-a30d467ec3a0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-a0ed681d-vfwqb\" (UID: \"d73e5e22-d2e4-42cd-8a29-a30d467ec3a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a0ed681d-vfwqb"
Sep 19 04:43:26.763134 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:26.763115    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d73e5e22-d2e4-42cd-8a29-a30d467ec3a0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-a0ed681d-vfwqb\" (UID: \"d73e5e22-d2e4-42cd-8a29-a30d467ec3a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a0ed681d-vfwqb"
Sep 19 04:43:26.898248 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:26.898158    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a0ed681d-vfwqb"
Sep 19 04:43:27.182579 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:27.182531    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a0ed681d-vfwqb"]
Sep 19 04:43:27.183578 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:43:27.183525    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd73e5e22_d2e4_42cd_8a29_a30d467ec3a0.slice/crio-e6a8b437c0378f5c4230f66ed6ef0cec992b26383789eaf290179cca258669e3 WatchSource:0}: Error finding container e6a8b437c0378f5c4230f66ed6ef0cec992b26383789eaf290179cca258669e3: Status 404 returned error can't find the container with id e6a8b437c0378f5c4230f66ed6ef0cec992b26383789eaf290179cca258669e3
Sep 19 04:43:27.216714 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:27.216667    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a0ed681d-vfwqb" event={"ID":"d73e5e22-d2e4-42cd-8a29-a30d467ec3a0","Type":"ContainerStarted","Data":"e6a8b437c0378f5c4230f66ed6ef0cec992b26383789eaf290179cca258669e3"}
Sep 19 04:43:28.224504 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:28.224474    2579 generic.go:356] "Generic (PLEG): container finished" podID="d73e5e22-d2e4-42cd-8a29-a30d467ec3a0" containerID="cfe7610fd69df562a3852f1b2b72bc5e48948d6f5f6038dcfc9bb8f7d7f2feb9" exitCode=0
Sep 19 04:43:28.224956 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:28.224563    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a0ed681d-vfwqb" event={"ID":"d73e5e22-d2e4-42cd-8a29-a30d467ec3a0","Type":"ContainerDied","Data":"cfe7610fd69df562a3852f1b2b72bc5e48948d6f5f6038dcfc9bb8f7d7f2feb9"}
Sep 19 04:43:29.390997 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:29.390971    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a0ed681d-vfwqb"
Sep 19 04:43:29.492161 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:29.492070    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d73e5e22-d2e4-42cd-8a29-a30d467ec3a0-maas-api-service-ca\") pod \"d73e5e22-d2e4-42cd-8a29-a30d467ec3a0\" (UID: \"d73e5e22-d2e4-42cd-8a29-a30d467ec3a0\") "
Sep 19 04:43:29.492549 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:29.492499    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d73e5e22-d2e4-42cd-8a29-a30d467ec3a0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d73e5e22-d2e4-42cd-8a29-a30d467ec3a0" (UID: "d73e5e22-d2e4-42cd-8a29-a30d467ec3a0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 04:43:29.593944 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:29.593908    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d73e5e22-d2e4-42cd-8a29-a30d467ec3a0-maas-api-service-ca\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:43:30.238534 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:30.238510    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a0ed681d-vfwqb"
Sep 19 04:43:30.238753 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:30.238498    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a0ed681d-vfwqb" event={"ID":"d73e5e22-d2e4-42cd-8a29-a30d467ec3a0","Type":"ContainerDied","Data":"e6a8b437c0378f5c4230f66ed6ef0cec992b26383789eaf290179cca258669e3"}
Sep 19 04:43:30.238753 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:30.238639    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6a8b437c0378f5c4230f66ed6ef0cec992b26383789eaf290179cca258669e3"
Sep 19 04:43:36.209376 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:36.209307    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8bbcca-5c75fd676b-jc48f"
Sep 19 04:43:41.616676 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:41.616644    2579 scope.go:117] "RemoveContainer" containerID="4ec11e2f9a889ec7994f9298796d6f7a16923221124f5669aaf6eedcca50ef65"
Sep 19 04:43:45.666629 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:45.666577    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a0ed681d-vfwqb"]
Sep 19 04:43:45.670613 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:45.670582    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a0ed681d-vfwqb"]
Sep 19 04:43:47.355976 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:47.355934    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d73e5e22-d2e4-42cd-8a29-a30d467ec3a0" path="/var/lib/kubelet/pods/d73e5e22-d2e4-42cd-8a29-a30d467ec3a0/volumes"
Sep 19 04:43:50.837858 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:50.837821    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-561bb1/auth-test-model-561bb1-kserve-747479f548-x5znp"]
Sep 19 04:43:50.838327 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:50.838256    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d73e5e22-d2e4-42cd-8a29-a30d467ec3a0" containerName="revoke-keys"
Sep 19 04:43:50.838327 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:50.838267    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d73e5e22-d2e4-42cd-8a29-a30d467ec3a0" containerName="revoke-keys"
Sep 19 04:43:50.838404 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:50.838350    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d73e5e22-d2e4-42cd-8a29-a30d467ec3a0" containerName="revoke-keys"
Sep 19 04:43:50.841552 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:50.841534    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-561bb1/auth-test-model-561bb1-kserve-747479f548-x5znp"
Sep 19 04:43:50.844570 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:50.844544    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-561bb1\"/\"auth-test-model-561bb1-kserve-self-signed-certs\""
Sep 19 04:43:50.845776 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:50.845754    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-561bb1\"/\"kube-root-ca.crt\""
Sep 19 04:43:50.845887 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:50.845846    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-561bb1\"/\"openshift-service-ca.crt\""
Sep 19 04:43:50.846968 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:50.846948    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-561bb1\"/\"default-dockercfg-w2jgl\""
Sep 19 04:43:50.853013 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:50.852988    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-561bb1/auth-test-model-561bb1-kserve-747479f548-x5znp"]
Sep 19 04:43:50.910389 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:50.910353    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e185da2-a21a-48ea-858f-e2c225df87d6-dshm\") pod \"auth-test-model-561bb1-kserve-747479f548-x5znp\" (UID: \"8e185da2-a21a-48ea-858f-e2c225df87d6\") " pod="ai-tenant-e2e-shared-a-w3-561bb1/auth-test-model-561bb1-kserve-747479f548-x5znp"
Sep 19 04:43:50.910555 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:50.910461    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mddz\" (UniqueName: \"kubernetes.io/projected/8e185da2-a21a-48ea-858f-e2c225df87d6-kube-api-access-6mddz\") pod \"auth-test-model-561bb1-kserve-747479f548-x5znp\" (UID: \"8e185da2-a21a-48ea-858f-e2c225df87d6\") " pod="ai-tenant-e2e-shared-a-w3-561bb1/auth-test-model-561bb1-kserve-747479f548-x5znp"
Sep 19 04:43:50.910555 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:50.910488    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e185da2-a21a-48ea-858f-e2c225df87d6-tmp-dir\") pod \"auth-test-model-561bb1-kserve-747479f548-x5znp\" (UID: \"8e185da2-a21a-48ea-858f-e2c225df87d6\") " pod="ai-tenant-e2e-shared-a-w3-561bb1/auth-test-model-561bb1-kserve-747479f548-x5znp"
Sep 19 04:43:50.910555 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:50.910515    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e185da2-a21a-48ea-858f-e2c225df87d6-model-cache\") pod \"auth-test-model-561bb1-kserve-747479f548-x5znp\" (UID: \"8e185da2-a21a-48ea-858f-e2c225df87d6\") " pod="ai-tenant-e2e-shared-a-w3-561bb1/auth-test-model-561bb1-kserve-747479f548-x5znp"
Sep 19 04:43:50.910555 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:50.910540    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e185da2-a21a-48ea-858f-e2c225df87d6-tls-certs\") pod \"auth-test-model-561bb1-kserve-747479f548-x5znp\" (UID: \"8e185da2-a21a-48ea-858f-e2c225df87d6\") " pod="ai-tenant-e2e-shared-a-w3-561bb1/auth-test-model-561bb1-kserve-747479f548-x5znp"
Sep 19 04:43:50.910715 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:50.910614    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e185da2-a21a-48ea-858f-e2c225df87d6-home\") pod \"auth-test-model-561bb1-kserve-747479f548-x5znp\" (UID: \"8e185da2-a21a-48ea-858f-e2c225df87d6\") " pod="ai-tenant-e2e-shared-a-w3-561bb1/auth-test-model-561bb1-kserve-747479f548-x5znp"
Sep 19 04:43:51.011558 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:51.011514    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mddz\" (UniqueName: \"kubernetes.io/projected/8e185da2-a21a-48ea-858f-e2c225df87d6-kube-api-access-6mddz\") pod \"auth-test-model-561bb1-kserve-747479f548-x5znp\" (UID: \"8e185da2-a21a-48ea-858f-e2c225df87d6\") " pod="ai-tenant-e2e-shared-a-w3-561bb1/auth-test-model-561bb1-kserve-747479f548-x5znp"
Sep 19 04:43:51.011558 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:51.011554    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e185da2-a21a-48ea-858f-e2c225df87d6-tmp-dir\") pod \"auth-test-model-561bb1-kserve-747479f548-x5znp\" (UID: \"8e185da2-a21a-48ea-858f-e2c225df87d6\") " pod="ai-tenant-e2e-shared-a-w3-561bb1/auth-test-model-561bb1-kserve-747479f548-x5znp"
Sep 19 04:43:51.011859 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:51.011578    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e185da2-a21a-48ea-858f-e2c225df87d6-model-cache\") pod \"auth-test-model-561bb1-kserve-747479f548-x5znp\" (UID: \"8e185da2-a21a-48ea-858f-e2c225df87d6\") " pod="ai-tenant-e2e-shared-a-w3-561bb1/auth-test-model-561bb1-kserve-747479f548-x5znp"
Sep 19 04:43:51.011859 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:51.011603    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e185da2-a21a-48ea-858f-e2c225df87d6-tls-certs\") pod \"auth-test-model-561bb1-kserve-747479f548-x5znp\" (UID: \"8e185da2-a21a-48ea-858f-e2c225df87d6\") " pod="ai-tenant-e2e-shared-a-w3-561bb1/auth-test-model-561bb1-kserve-747479f548-x5znp"
Sep 19 04:43:51.011859 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:51.011636    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e185da2-a21a-48ea-858f-e2c225df87d6-home\") pod \"auth-test-model-561bb1-kserve-747479f548-x5znp\" (UID: \"8e185da2-a21a-48ea-858f-e2c225df87d6\") " pod="ai-tenant-e2e-shared-a-w3-561bb1/auth-test-model-561bb1-kserve-747479f548-x5znp"
Sep 19 04:43:51.011859 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:51.011664    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e185da2-a21a-48ea-858f-e2c225df87d6-dshm\") pod \"auth-test-model-561bb1-kserve-747479f548-x5znp\" (UID: \"8e185da2-a21a-48ea-858f-e2c225df87d6\") " pod="ai-tenant-e2e-shared-a-w3-561bb1/auth-test-model-561bb1-kserve-747479f548-x5znp"
Sep 19 04:43:51.012065 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:51.012041    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e185da2-a21a-48ea-858f-e2c225df87d6-tmp-dir\") pod \"auth-test-model-561bb1-kserve-747479f548-x5znp\" (UID: \"8e185da2-a21a-48ea-858f-e2c225df87d6\") " pod="ai-tenant-e2e-shared-a-w3-561bb1/auth-test-model-561bb1-kserve-747479f548-x5znp"
Sep 19 04:43:51.012107 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:51.012063    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e185da2-a21a-48ea-858f-e2c225df87d6-model-cache\") pod \"auth-test-model-561bb1-kserve-747479f548-x5znp\" (UID: \"8e185da2-a21a-48ea-858f-e2c225df87d6\") " pod="ai-tenant-e2e-shared-a-w3-561bb1/auth-test-model-561bb1-kserve-747479f548-x5znp"
Sep 19 04:43:51.012145 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:51.012131    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e185da2-a21a-48ea-858f-e2c225df87d6-home\") pod \"auth-test-model-561bb1-kserve-747479f548-x5znp\" (UID: \"8e185da2-a21a-48ea-858f-e2c225df87d6\") " pod="ai-tenant-e2e-shared-a-w3-561bb1/auth-test-model-561bb1-kserve-747479f548-x5znp"
Sep 19 04:43:51.014179 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:51.014158    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e185da2-a21a-48ea-858f-e2c225df87d6-dshm\") pod \"auth-test-model-561bb1-kserve-747479f548-x5znp\" (UID: \"8e185da2-a21a-48ea-858f-e2c225df87d6\") " pod="ai-tenant-e2e-shared-a-w3-561bb1/auth-test-model-561bb1-kserve-747479f548-x5znp"
Sep 19 04:43:51.014424 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:51.014404    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e185da2-a21a-48ea-858f-e2c225df87d6-tls-certs\") pod \"auth-test-model-561bb1-kserve-747479f548-x5znp\" (UID: \"8e185da2-a21a-48ea-858f-e2c225df87d6\") " pod="ai-tenant-e2e-shared-a-w3-561bb1/auth-test-model-561bb1-kserve-747479f548-x5znp"
Sep 19 04:43:51.021507 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:51.021478    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mddz\" (UniqueName: \"kubernetes.io/projected/8e185da2-a21a-48ea-858f-e2c225df87d6-kube-api-access-6mddz\") pod \"auth-test-model-561bb1-kserve-747479f548-x5znp\" (UID: \"8e185da2-a21a-48ea-858f-e2c225df87d6\") " pod="ai-tenant-e2e-shared-a-w3-561bb1/auth-test-model-561bb1-kserve-747479f548-x5znp"
Sep 19 04:43:51.151158 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:51.151129    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-561bb1/auth-test-model-561bb1-kserve-747479f548-x5znp"
Sep 19 04:43:51.429247 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:51.429223    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-561bb1/auth-test-model-561bb1-kserve-747479f548-x5znp"]
Sep 19 04:43:51.430761 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:43:51.430735    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8e185da2_a21a_48ea_858f_e2c225df87d6.slice/crio-54b12f6d7b2806278bb91252a86bc89b9d1f849835d4412a0110a4efbc3f1e70 WatchSource:0}: Error finding container 54b12f6d7b2806278bb91252a86bc89b9d1f849835d4412a0110a4efbc3f1e70: Status 404 returned error can't find the container with id 54b12f6d7b2806278bb91252a86bc89b9d1f849835d4412a0110a4efbc3f1e70
Sep 19 04:43:52.395372 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:52.395332    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-561bb1/auth-test-model-561bb1-kserve-747479f548-x5znp" event={"ID":"8e185da2-a21a-48ea-858f-e2c225df87d6","Type":"ContainerStarted","Data":"14308c49e918f6062fd6b5da38953f17ff1ff0f5aa0b97b4647ed37c1c87de99"}
Sep 19 04:43:52.395772 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:52.395382    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-561bb1/auth-test-model-561bb1-kserve-747479f548-x5znp" event={"ID":"8e185da2-a21a-48ea-858f-e2c225df87d6","Type":"ContainerStarted","Data":"54b12f6d7b2806278bb91252a86bc89b9d1f849835d4412a0110a4efbc3f1e70"}
Sep 19 04:43:53.151941 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:43:53.151909    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-561bb1/auth-test-model-561bb1-kserve-747479f548-x5znp"
Sep 19 04:44:01.151514 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:44:01.151477    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-561bb1/auth-test-model-561bb1-kserve-747479f548-x5znp"
Sep 19 04:44:01.165106 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:44:01.165078    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-561bb1/auth-test-model-561bb1-kserve-747479f548-x5znp"
Sep 19 04:44:01.186941 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:44:01.186883    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-561bb1/auth-test-model-561bb1-kserve-747479f548-x5znp" podStartSLOduration=11.186868127 podStartE2EDuration="11.186868127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 04:43:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:43:52.481260026 +0000 UTC m=+1811.664221174" watchObservedRunningTime="2026-09-19 04:44:01.186868127 +0000 UTC m=+1820.369829311"
Sep 19 04:44:01.459684 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:44:01.459600    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-561bb1/auth-test-model-561bb1-kserve-747479f548-x5znp"
Sep 19 04:44:54.662814 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:44:54.662780    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w"]
Sep 19 04:44:54.663292 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:44:54.663042    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w" podUID="b427907c-0354-4e21-930b-e3ef32dff3b9" containerName="istio-proxy" containerID="cri-o://5d4f57df6c58261a58ea006ef0fe764523ceaee00593a7f07ee09af4224454f7" gracePeriod=30
Sep 19 04:44:59.824649 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:44:59.824599    2579 generic.go:356] "Generic (PLEG): container finished" podID="b427907c-0354-4e21-930b-e3ef32dff3b9" containerID="5d4f57df6c58261a58ea006ef0fe764523ceaee00593a7f07ee09af4224454f7" exitCode=0
Sep 19 04:44:59.824649 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:44:59.824619    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w" event={"ID":"b427907c-0354-4e21-930b-e3ef32dff3b9","Type":"ContainerDied","Data":"5d4f57df6c58261a58ea006ef0fe764523ceaee00593a7f07ee09af4224454f7"}
Sep 19 04:44:59.825168 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:44:59.824730    2579 scope.go:117] "RemoveContainer" containerID="0b7e5a62b6f1af77d9049f75023c1413507f84d42fa1a3ebefe4a2cb15d9e75e"
Sep 19 04:44:59.934018 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:44:59.933960    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w"
Sep 19 04:45:00.027433 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:00.027382    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b427907c-0354-4e21-930b-e3ef32dff3b9-istiod-ca-cert\") pod \"b427907c-0354-4e21-930b-e3ef32dff3b9\" (UID: \"b427907c-0354-4e21-930b-e3ef32dff3b9\") "
Sep 19 04:45:00.027651 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:00.027458    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b427907c-0354-4e21-930b-e3ef32dff3b9-workload-certs\") pod \"b427907c-0354-4e21-930b-e3ef32dff3b9\" (UID: \"b427907c-0354-4e21-930b-e3ef32dff3b9\") "
Sep 19 04:45:00.027651 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:00.027527    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b427907c-0354-4e21-930b-e3ef32dff3b9-istio-data\") pod \"b427907c-0354-4e21-930b-e3ef32dff3b9\" (UID: \"b427907c-0354-4e21-930b-e3ef32dff3b9\") "
Sep 19 04:45:00.027651 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:00.027583    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4gzct\" (UniqueName: \"kubernetes.io/projected/b427907c-0354-4e21-930b-e3ef32dff3b9-kube-api-access-4gzct\") pod \"b427907c-0354-4e21-930b-e3ef32dff3b9\" (UID: \"b427907c-0354-4e21-930b-e3ef32dff3b9\") "
Sep 19 04:45:00.027651 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:00.027631    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b427907c-0354-4e21-930b-e3ef32dff3b9-istio-token\") pod \"b427907c-0354-4e21-930b-e3ef32dff3b9\" (UID: \"b427907c-0354-4e21-930b-e3ef32dff3b9\") "
Sep 19 04:45:00.027948 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:00.027742    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b427907c-0354-4e21-930b-e3ef32dff3b9-istio-envoy\") pod \"b427907c-0354-4e21-930b-e3ef32dff3b9\" (UID: \"b427907c-0354-4e21-930b-e3ef32dff3b9\") "
Sep 19 04:45:00.027948 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:00.027791    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b427907c-0354-4e21-930b-e3ef32dff3b9-workload-socket\") pod \"b427907c-0354-4e21-930b-e3ef32dff3b9\" (UID: \"b427907c-0354-4e21-930b-e3ef32dff3b9\") "
Sep 19 04:45:00.027948 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:00.027798    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b427907c-0354-4e21-930b-e3ef32dff3b9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b427907c-0354-4e21-930b-e3ef32dff3b9" (UID: "b427907c-0354-4e21-930b-e3ef32dff3b9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:45:00.027948 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:00.027823    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b427907c-0354-4e21-930b-e3ef32dff3b9-istio-data" (OuterVolumeSpecName: "istio-data") pod "b427907c-0354-4e21-930b-e3ef32dff3b9" (UID: "b427907c-0354-4e21-930b-e3ef32dff3b9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:45:00.027948 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:00.027833    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b427907c-0354-4e21-930b-e3ef32dff3b9-istio-podinfo\") pod \"b427907c-0354-4e21-930b-e3ef32dff3b9\" (UID: \"b427907c-0354-4e21-930b-e3ef32dff3b9\") "
Sep 19 04:45:00.027948 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:00.027884    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b427907c-0354-4e21-930b-e3ef32dff3b9-credential-socket\") pod \"b427907c-0354-4e21-930b-e3ef32dff3b9\" (UID: \"b427907c-0354-4e21-930b-e3ef32dff3b9\") "
Sep 19 04:45:00.028286 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:00.028265    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b427907c-0354-4e21-930b-e3ef32dff3b9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b427907c-0354-4e21-930b-e3ef32dff3b9" (UID: "b427907c-0354-4e21-930b-e3ef32dff3b9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 04:45:00.028355 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:00.028312    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b427907c-0354-4e21-930b-e3ef32dff3b9-workload-certs\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:45:00.028355 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:00.028331    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b427907c-0354-4e21-930b-e3ef32dff3b9-istio-data\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:45:00.028887 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:00.028513    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b427907c-0354-4e21-930b-e3ef32dff3b9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b427907c-0354-4e21-930b-e3ef32dff3b9" (UID: "b427907c-0354-4e21-930b-e3ef32dff3b9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:45:00.028887 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:00.028730    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b427907c-0354-4e21-930b-e3ef32dff3b9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b427907c-0354-4e21-930b-e3ef32dff3b9" (UID: "b427907c-0354-4e21-930b-e3ef32dff3b9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:45:00.030684 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:00.030622    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b427907c-0354-4e21-930b-e3ef32dff3b9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b427907c-0354-4e21-930b-e3ef32dff3b9" (UID: "b427907c-0354-4e21-930b-e3ef32dff3b9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 19 04:45:00.030684 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:00.030632    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b427907c-0354-4e21-930b-e3ef32dff3b9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b427907c-0354-4e21-930b-e3ef32dff3b9" (UID: "b427907c-0354-4e21-930b-e3ef32dff3b9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:45:00.030915 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:00.030753    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b427907c-0354-4e21-930b-e3ef32dff3b9-istio-token" (OuterVolumeSpecName: "istio-token") pod "b427907c-0354-4e21-930b-e3ef32dff3b9" (UID: "b427907c-0354-4e21-930b-e3ef32dff3b9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 04:45:00.031232 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:00.031207    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b427907c-0354-4e21-930b-e3ef32dff3b9-kube-api-access-4gzct" (OuterVolumeSpecName: "kube-api-access-4gzct") pod "b427907c-0354-4e21-930b-e3ef32dff3b9" (UID: "b427907c-0354-4e21-930b-e3ef32dff3b9"). InnerVolumeSpecName "kube-api-access-4gzct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 04:45:00.129189 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:00.129152    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gzct\" (UniqueName: \"kubernetes.io/projected/b427907c-0354-4e21-930b-e3ef32dff3b9-kube-api-access-4gzct\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:45:00.129189 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:00.129185    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b427907c-0354-4e21-930b-e3ef32dff3b9-istio-token\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:45:00.129189 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:00.129195    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b427907c-0354-4e21-930b-e3ef32dff3b9-istio-envoy\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:45:00.129520 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:00.129203    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b427907c-0354-4e21-930b-e3ef32dff3b9-workload-socket\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:45:00.129520 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:00.129211    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b427907c-0354-4e21-930b-e3ef32dff3b9-istio-podinfo\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:45:00.129520 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:00.129219    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b427907c-0354-4e21-930b-e3ef32dff3b9-credential-socket\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:45:00.129520 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:00.129227    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b427907c-0354-4e21-930b-e3ef32dff3b9-istiod-ca-cert\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:45:00.172923 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:00.172884    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-eac6c5-29829885-pxvwr"]
Sep 19 04:45:00.173337 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:00.173323    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b427907c-0354-4e21-930b-e3ef32dff3b9" containerName="istio-proxy"
Sep 19 04:45:00.173381 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:00.173339    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b427907c-0354-4e21-930b-e3ef32dff3b9" containerName="istio-proxy"
Sep 19 04:45:00.173441 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:00.173430    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b427907c-0354-4e21-930b-e3ef32dff3b9" containerName="istio-proxy"
Sep 19 04:45:00.173480 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:00.173448    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b427907c-0354-4e21-930b-e3ef32dff3b9" containerName="istio-proxy"
Sep 19 04:45:00.176888 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:00.176872    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-eac6c5-29829885-pxvwr"
Sep 19 04:45:00.179975 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:00.179949    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s6xbq\""
Sep 19 04:45:00.181313 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:00.181290    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8bbcca-29829885-m47zz"]
Sep 19 04:45:00.181745 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:00.181727    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b427907c-0354-4e21-930b-e3ef32dff3b9" containerName="istio-proxy"
Sep 19 04:45:00.181833 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:00.181746    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b427907c-0354-4e21-930b-e3ef32dff3b9" containerName="istio-proxy"
Sep 19 04:45:00.186196 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:00.185201    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29829885-xlqqv"]
Sep 19 04:45:00.186196 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:00.185329    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8bbcca-29829885-m47zz"
Sep 19 04:45:00.189124 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:00.189101    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829885-xlqqv"
Sep 19 04:45:00.191014 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:00.190995    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-eac6c5-29829885-pxvwr"
Sep 19 04:45:00.201463 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:00.201442    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8bbcca-29829885-m47zz"
Sep 19 04:45:00.211588 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:00.211565    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829885-xlqqv"
Sep 19 04:45:00.220298 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:00.220262    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-eac6c5-29829885-pxvwr"]
Sep 19 04:45:00.238740 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:00.238496    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29829885-xlqqv"]
Sep 19 04:45:00.241160 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:00.241108    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8bbcca-29829885-m47zz"]
Sep 19 04:45:00.264478 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:00.264087    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-87e3e6-29829885-pfkpp"]
Sep 19 04:45:00.272955 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:00.272923    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-87e3e6-29829885-pfkpp"
Sep 19 04:45:00.281278 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:00.281243    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-87e3e6-29829885-pfkpp"]
Sep 19 04:45:00.303258 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:00.303159    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-87e3e6-29829885-pfkpp"
Sep 19 04:45:00.520178 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:00.520150    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-eac6c5-29829885-pxvwr"]
Sep 19 04:45:00.520501 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:45:00.520475    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dac9dfd_2820_46d9_b6ed_d0d691d033fe.slice/crio-d75a7f94925a3dfa8890f5125c8c3c5b296cdf6f59ce42804816abd218d7f7bf WatchSource:0}: Error finding container d75a7f94925a3dfa8890f5125c8c3c5b296cdf6f59ce42804816abd218d7f7bf: Status 404 returned error can't find the container with id d75a7f94925a3dfa8890f5125c8c3c5b296cdf6f59ce42804816abd218d7f7bf
Sep 19 04:45:00.617991 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:00.617875    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8bbcca-29829885-m47zz"]
Sep 19 04:45:00.619793 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:45:00.619755    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc91ac636_b399_4ae7_a092_f7b3c533de75.slice/crio-e7cc4bc99d1e8cc043571e92d520b88254eb30c7cf7527dbca58f7d79a7be8f6 WatchSource:0}: Error finding container e7cc4bc99d1e8cc043571e92d520b88254eb30c7cf7527dbca58f7d79a7be8f6: Status 404 returned error can't find the container with id e7cc4bc99d1e8cc043571e92d520b88254eb30c7cf7527dbca58f7d79a7be8f6
Sep 19 04:45:00.622665 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:00.622641    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29829885-xlqqv"]
Sep 19 04:45:00.622985 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:45:00.622965    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bd23183_a191_4026_9927_76cfe375ca16.slice/crio-2debbf0ebaedf85bffccc517e27f2222481aaa58475c63bcb7ee09cc7b488731 WatchSource:0}: Error finding container 2debbf0ebaedf85bffccc517e27f2222481aaa58475c63bcb7ee09cc7b488731: Status 404 returned error can't find the container with id 2debbf0ebaedf85bffccc517e27f2222481aaa58475c63bcb7ee09cc7b488731
Sep 19 04:45:00.664776 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:00.664747    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-87e3e6-29829885-pfkpp"]
Sep 19 04:45:00.665528 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:45:00.665507    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2f06779_4e8a_402b_91b0_3b8bd3664c94.slice/crio-5b0f807b3c39a9a84c7b213824f4484089eb4dde3a7ab217a1edfc3327aefeb5 WatchSource:0}: Error finding container 5b0f807b3c39a9a84c7b213824f4484089eb4dde3a7ab217a1edfc3327aefeb5: Status 404 returned error can't find the container with id 5b0f807b3c39a9a84c7b213824f4484089eb4dde3a7ab217a1edfc3327aefeb5
Sep 19 04:45:00.831977 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:00.831945    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8bbcca-29829885-m47zz" event={"ID":"c91ac636-b399-4ae7-a092-f7b3c533de75","Type":"ContainerStarted","Data":"e7cc4bc99d1e8cc043571e92d520b88254eb30c7cf7527dbca58f7d79a7be8f6"}
Sep 19 04:45:00.833253 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:00.833229    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829885-xlqqv" event={"ID":"0bd23183-a191-4026-9927-76cfe375ca16","Type":"ContainerStarted","Data":"2debbf0ebaedf85bffccc517e27f2222481aaa58475c63bcb7ee09cc7b488731"}
Sep 19 04:45:00.834869 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:00.834836    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-eac6c5-29829885-pxvwr" event={"ID":"8dac9dfd-2820-46d9-b6ed-d0d691d033fe","Type":"ContainerStarted","Data":"802d025c0cca96069e1066cb07bccf2bd655ae20d719587dc0dc72262053bcbf"}
Sep 19 04:45:00.834959 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:00.834885    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-eac6c5-29829885-pxvwr" event={"ID":"8dac9dfd-2820-46d9-b6ed-d0d691d033fe","Type":"ContainerStarted","Data":"d75a7f94925a3dfa8890f5125c8c3c5b296cdf6f59ce42804816abd218d7f7bf"}
Sep 19 04:45:00.836205 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:00.836176    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-87e3e6-29829885-pfkpp" event={"ID":"d2f06779-4e8a-402b-91b0-3b8bd3664c94","Type":"ContainerStarted","Data":"5b0f807b3c39a9a84c7b213824f4484089eb4dde3a7ab217a1edfc3327aefeb5"}
Sep 19 04:45:00.837438 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:00.837417    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w"
Sep 19 04:45:00.837519 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:00.837414    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w" event={"ID":"b427907c-0354-4e21-930b-e3ef32dff3b9","Type":"ContainerDied","Data":"1fa5588e00e60ee84eb1196d2dfe89142b4ff424ddff7cda6aa0590e63258f58"}
Sep 19 04:45:00.837561 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:00.837548    2579 scope.go:117] "RemoveContainer" containerID="5d4f57df6c58261a58ea006ef0fe764523ceaee00593a7f07ee09af4224454f7"
Sep 19 04:45:00.852070 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:00.852029    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-eac6c5-29829885-pxvwr" podStartSLOduration=0.852014823 podStartE2EDuration="852.014823ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 04:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:45:00.851728475 +0000 UTC m=+1880.034689624" watchObservedRunningTime="2026-09-19 04:45:00.852014823 +0000 UTC m=+1880.034975971"
Sep 19 04:45:00.867523 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:00.867494    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w"]
Sep 19 04:45:00.873352 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:00.873330    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-a0ed681d-openshift-default-58b9db579b-kdd7w"]
Sep 19 04:45:01.352641 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:01.352562    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b427907c-0354-4e21-930b-e3ef32dff3b9" path="/var/lib/kubelet/pods/b427907c-0354-4e21-930b-e3ef32dff3b9/volumes"
Sep 19 04:45:01.845230 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:01.845195    2579 generic.go:356] "Generic (PLEG): container finished" podID="0bd23183-a191-4026-9927-76cfe375ca16" containerID="420f01b3d7d9ca01f200a5c62e57cced296ebf578092a0a6436b142e2a0f7dc9" exitCode=0
Sep 19 04:45:01.845679 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:01.845282    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829885-xlqqv" event={"ID":"0bd23183-a191-4026-9927-76cfe375ca16","Type":"ContainerDied","Data":"420f01b3d7d9ca01f200a5c62e57cced296ebf578092a0a6436b142e2a0f7dc9"}
Sep 19 04:45:01.846730 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:01.846681    2579 generic.go:356] "Generic (PLEG): container finished" podID="8dac9dfd-2820-46d9-b6ed-d0d691d033fe" containerID="802d025c0cca96069e1066cb07bccf2bd655ae20d719587dc0dc72262053bcbf" exitCode=0
Sep 19 04:45:01.846851 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:01.846730    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-eac6c5-29829885-pxvwr" event={"ID":"8dac9dfd-2820-46d9-b6ed-d0d691d033fe","Type":"ContainerDied","Data":"802d025c0cca96069e1066cb07bccf2bd655ae20d719587dc0dc72262053bcbf"}
Sep 19 04:45:01.848317 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:01.848298    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-87e3e6-29829885-pfkpp" event={"ID":"d2f06779-4e8a-402b-91b0-3b8bd3664c94","Type":"ContainerStarted","Data":"0bbf97b0cbcbb518dc9f44fb032538b478d45d2a03d24fb4c7b9385d568d9012"}
Sep 19 04:45:01.850640 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:01.850613    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8bbcca-29829885-m47zz" event={"ID":"c91ac636-b399-4ae7-a092-f7b3c533de75","Type":"ContainerStarted","Data":"0da01db5c6bcddf2b3e4c65fe496523d17f729b037c0367d16032a0b01648f72"}
Sep 19 04:45:01.959747 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:01.959628    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8bbcca-29829885-m47zz" podStartSLOduration=1.959611075 podStartE2EDuration="1.959611075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 04:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:45:01.925826043 +0000 UTC m=+1881.108787190" watchObservedRunningTime="2026-09-19 04:45:01.959611075 +0000 UTC m=+1881.142572269"
Sep 19 04:45:01.986307 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:01.986246    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-87e3e6-29829885-pfkpp" podStartSLOduration=1.986225753 podStartE2EDuration="1.986225753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 04:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:45:01.985754676 +0000 UTC m=+1881.168715844" watchObservedRunningTime="2026-09-19 04:45:01.986225753 +0000 UTC m=+1881.169186904"
Sep 19 04:45:02.858941 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:02.858906    2579 generic.go:356] "Generic (PLEG): container finished" podID="c91ac636-b399-4ae7-a092-f7b3c533de75" containerID="0da01db5c6bcddf2b3e4c65fe496523d17f729b037c0367d16032a0b01648f72" exitCode=0
Sep 19 04:45:02.859325 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:02.858991    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8bbcca-29829885-m47zz" event={"ID":"c91ac636-b399-4ae7-a092-f7b3c533de75","Type":"ContainerDied","Data":"0da01db5c6bcddf2b3e4c65fe496523d17f729b037c0367d16032a0b01648f72"}
Sep 19 04:45:02.860637 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:02.860608    2579 generic.go:356] "Generic (PLEG): container finished" podID="d2f06779-4e8a-402b-91b0-3b8bd3664c94" containerID="0bbf97b0cbcbb518dc9f44fb032538b478d45d2a03d24fb4c7b9385d568d9012" exitCode=0
Sep 19 04:45:02.860789 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:02.860687    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-87e3e6-29829885-pfkpp" event={"ID":"d2f06779-4e8a-402b-91b0-3b8bd3664c94","Type":"ContainerDied","Data":"0bbf97b0cbcbb518dc9f44fb032538b478d45d2a03d24fb4c7b9385d568d9012"}
Sep 19 04:45:03.033954 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:03.033926    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829885-xlqqv"
Sep 19 04:45:03.036968 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:03.036943    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-eac6c5-29829885-pxvwr"
Sep 19 04:45:03.868972 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:03.868935    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829885-xlqqv" event={"ID":"0bd23183-a191-4026-9927-76cfe375ca16","Type":"ContainerDied","Data":"2debbf0ebaedf85bffccc517e27f2222481aaa58475c63bcb7ee09cc7b488731"}
Sep 19 04:45:03.869431 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:03.868993    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2debbf0ebaedf85bffccc517e27f2222481aaa58475c63bcb7ee09cc7b488731"
Sep 19 04:45:03.869431 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:03.868955    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829885-xlqqv"
Sep 19 04:45:03.870637 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:03.870618    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-eac6c5-29829885-pxvwr"
Sep 19 04:45:03.870839 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:03.870647    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-eac6c5-29829885-pxvwr" event={"ID":"8dac9dfd-2820-46d9-b6ed-d0d691d033fe","Type":"ContainerDied","Data":"d75a7f94925a3dfa8890f5125c8c3c5b296cdf6f59ce42804816abd218d7f7bf"}
Sep 19 04:45:03.870911 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:03.870686    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d75a7f94925a3dfa8890f5125c8c3c5b296cdf6f59ce42804816abd218d7f7bf"
Sep 19 04:45:04.026070 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:04.026044    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8bbcca-29829885-m47zz"
Sep 19 04:45:04.061021 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:04.060992    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-87e3e6-29829885-pfkpp"
Sep 19 04:45:04.879777 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:04.879748    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-87e3e6-29829885-pfkpp"
Sep 19 04:45:04.880236 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:04.879787    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-87e3e6-29829885-pfkpp" event={"ID":"d2f06779-4e8a-402b-91b0-3b8bd3664c94","Type":"ContainerDied","Data":"5b0f807b3c39a9a84c7b213824f4484089eb4dde3a7ab217a1edfc3327aefeb5"}
Sep 19 04:45:04.880236 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:04.879833    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b0f807b3c39a9a84c7b213824f4484089eb4dde3a7ab217a1edfc3327aefeb5"
Sep 19 04:45:04.881265 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:04.881245    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8bbcca-29829885-m47zz"
Sep 19 04:45:04.881387 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:04.881245    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8bbcca-29829885-m47zz" event={"ID":"c91ac636-b399-4ae7-a092-f7b3c533de75","Type":"ContainerDied","Data":"e7cc4bc99d1e8cc043571e92d520b88254eb30c7cf7527dbca58f7d79a7be8f6"}
Sep 19 04:45:04.881387 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:04.881364    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e7cc4bc99d1e8cc043571e92d520b88254eb30c7cf7527dbca58f7d79a7be8f6"
Sep 19 04:45:21.771473 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:21.771429    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-z2g72" podUID="d2cb2792-51a4-4365-95f5-de0f536efd2c" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.35:8081/readyz\": dial tcp 10.134.0.35:8081: connect: connection refused"
Sep 19 04:45:21.988393 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:21.988361    2579 generic.go:356] "Generic (PLEG): container finished" podID="d2cb2792-51a4-4365-95f5-de0f536efd2c" containerID="68de29a2166259ee1c510a5a49c0e986026e078e83b7b550650c3b46be061e68" exitCode=137
Sep 19 04:45:21.988559 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:21.988442    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-z2g72" event={"ID":"d2cb2792-51a4-4365-95f5-de0f536efd2c","Type":"ContainerDied","Data":"68de29a2166259ee1c510a5a49c0e986026e078e83b7b550650c3b46be061e68"}
Sep 19 04:45:21.988867 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:21.988851    2579 scope.go:117] "RemoveContainer" containerID="68de29a2166259ee1c510a5a49c0e986026e078e83b7b550650c3b46be061e68"
Sep 19 04:45:23.003583 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:23.003543    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-z2g72" event={"ID":"d2cb2792-51a4-4365-95f5-de0f536efd2c","Type":"ContainerStarted","Data":"f08e74eb0cda9f00f6e9f0f4341bce6864bb32f8d0b020582fdc21234c2ee654"}
Sep 19 04:45:23.004051 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:23.003837    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-z2g72"
Sep 19 04:45:34.011800 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:34.011766    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-z2g72"
Sep 19 04:45:41.652338 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:41.652302    2579 scope.go:117] "RemoveContainer" containerID="264ebccdcb0cf39fe6d8b74044b8be54cdd1bb76b2239ded09a5441371cb11d1"
Sep 19 04:45:58.655818 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:58.655788    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1a212c-6585f9b8ff-s877b"]
Sep 19 04:45:58.656276 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:58.656266    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2f06779-4e8a-402b-91b0-3b8bd3664c94" containerName="cleanup"
Sep 19 04:45:58.656322 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:58.656278    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2f06779-4e8a-402b-91b0-3b8bd3664c94" containerName="cleanup"
Sep 19 04:45:58.656322 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:58.656294    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8dac9dfd-2820-46d9-b6ed-d0d691d033fe" containerName="cleanup"
Sep 19 04:45:58.656322 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:58.656299    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8dac9dfd-2820-46d9-b6ed-d0d691d033fe" containerName="cleanup"
Sep 19 04:45:58.656322 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:58.656307    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c91ac636-b399-4ae7-a092-f7b3c533de75" containerName="cleanup"
Sep 19 04:45:58.656322 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:58.656312    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c91ac636-b399-4ae7-a092-f7b3c533de75" containerName="cleanup"
Sep 19 04:45:58.656488 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:58.656337    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bd23183-a191-4026-9927-76cfe375ca16" containerName="cleanup"
Sep 19 04:45:58.656488 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:58.656343    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bd23183-a191-4026-9927-76cfe375ca16" containerName="cleanup"
Sep 19 04:45:58.656488 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:58.656418    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c91ac636-b399-4ae7-a092-f7b3c533de75" containerName="cleanup"
Sep 19 04:45:58.656488 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:58.656426    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2f06779-4e8a-402b-91b0-3b8bd3664c94" containerName="cleanup"
Sep 19 04:45:58.656488 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:58.656435    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0bd23183-a191-4026-9927-76cfe375ca16" containerName="cleanup"
Sep 19 04:45:58.656488 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:58.656440    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8dac9dfd-2820-46d9-b6ed-d0d691d033fe" containerName="cleanup"
Sep 19 04:45:58.659795 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:58.659779    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1a212c-6585f9b8ff-s877b"
Sep 19 04:45:58.663458 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:58.663441    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-1a212c-dockercfg-6klzr\""
Sep 19 04:45:58.663802 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:58.663786    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-1a212c\""
Sep 19 04:45:58.688959 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:58.688926    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1a212c-6585f9b8ff-s877b"]
Sep 19 04:45:58.747000 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:58.746965    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/087b9c85-ead3-4c23-8e6e-9cfc0d3635bb-plugins-config-volume\") pod \"payload-processing-e2e-api-a-1a212c-6585f9b8ff-s877b\" (UID: \"087b9c85-ead3-4c23-8e6e-9cfc0d3635bb\") " pod="openshift-ingress/payload-processing-e2e-api-a-1a212c-6585f9b8ff-s877b"
Sep 19 04:45:58.747264 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:58.747008    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flnbz\" (UniqueName: \"kubernetes.io/projected/087b9c85-ead3-4c23-8e6e-9cfc0d3635bb-kube-api-access-flnbz\") pod \"payload-processing-e2e-api-a-1a212c-6585f9b8ff-s877b\" (UID: \"087b9c85-ead3-4c23-8e6e-9cfc0d3635bb\") " pod="openshift-ingress/payload-processing-e2e-api-a-1a212c-6585f9b8ff-s877b"
Sep 19 04:45:58.847868 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:58.847834    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/087b9c85-ead3-4c23-8e6e-9cfc0d3635bb-plugins-config-volume\") pod \"payload-processing-e2e-api-a-1a212c-6585f9b8ff-s877b\" (UID: \"087b9c85-ead3-4c23-8e6e-9cfc0d3635bb\") " pod="openshift-ingress/payload-processing-e2e-api-a-1a212c-6585f9b8ff-s877b"
Sep 19 04:45:58.847868 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:58.847871    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flnbz\" (UniqueName: \"kubernetes.io/projected/087b9c85-ead3-4c23-8e6e-9cfc0d3635bb-kube-api-access-flnbz\") pod \"payload-processing-e2e-api-a-1a212c-6585f9b8ff-s877b\" (UID: \"087b9c85-ead3-4c23-8e6e-9cfc0d3635bb\") " pod="openshift-ingress/payload-processing-e2e-api-a-1a212c-6585f9b8ff-s877b"
Sep 19 04:45:58.848562 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:58.848543    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/087b9c85-ead3-4c23-8e6e-9cfc0d3635bb-plugins-config-volume\") pod \"payload-processing-e2e-api-a-1a212c-6585f9b8ff-s877b\" (UID: \"087b9c85-ead3-4c23-8e6e-9cfc0d3635bb\") " pod="openshift-ingress/payload-processing-e2e-api-a-1a212c-6585f9b8ff-s877b"
Sep 19 04:45:58.861277 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:58.861244    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-flnbz\" (UniqueName: \"kubernetes.io/projected/087b9c85-ead3-4c23-8e6e-9cfc0d3635bb-kube-api-access-flnbz\") pod \"payload-processing-e2e-api-a-1a212c-6585f9b8ff-s877b\" (UID: \"087b9c85-ead3-4c23-8e6e-9cfc0d3635bb\") " pod="openshift-ingress/payload-processing-e2e-api-a-1a212c-6585f9b8ff-s877b"
Sep 19 04:45:58.970135 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:58.970049    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1a212c-6585f9b8ff-s877b"
Sep 19 04:45:59.256677 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:45:59.256648    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1a212c-6585f9b8ff-s877b"]
Sep 19 04:45:59.258257 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:45:59.258227    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod087b9c85_ead3_4c23_8e6e_9cfc0d3635bb.slice/crio-ad54e0f70a508b78cd53b64de15cfecfb080f75ab011cf9aa4db761c54f6587e WatchSource:0}: Error finding container ad54e0f70a508b78cd53b64de15cfecfb080f75ab011cf9aa4db761c54f6587e: Status 404 returned error can't find the container with id ad54e0f70a508b78cd53b64de15cfecfb080f75ab011cf9aa4db761c54f6587e
Sep 19 04:46:00.237891 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:46:00.237850    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1a212c-6585f9b8ff-s877b" event={"ID":"087b9c85-ead3-4c23-8e6e-9cfc0d3635bb","Type":"ContainerStarted","Data":"991b068dac945bc94531c7dc73d197a0b03173f383ef53cb15d7b26eca36a90d"}
Sep 19 04:46:00.238397 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:46:00.237918    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1a212c-6585f9b8ff-s877b" event={"ID":"087b9c85-ead3-4c23-8e6e-9cfc0d3635bb","Type":"ContainerStarted","Data":"ad54e0f70a508b78cd53b64de15cfecfb080f75ab011cf9aa4db761c54f6587e"}
Sep 19 04:46:00.238397 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:46:00.238026    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-1a212c-6585f9b8ff-s877b"
Sep 19 04:46:00.263132 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:46:00.263077    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-1a212c-6585f9b8ff-s877b" podStartSLOduration=2.263060695 podStartE2EDuration="2.263060695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 04:45:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:46:00.259577888 +0000 UTC m=+1939.442539038" watchObservedRunningTime="2026-09-19 04:46:00.263060695 +0000 UTC m=+1939.446021843"
Sep 19 04:46:11.252094 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:46:11.252063    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-1a212c-6585f9b8ff-s877b"
Sep 19 04:46:21.875246 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:46:21.875211    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-87e3e6/embedding-w0-87e3e6-kserve-64c8966d59-4r28w"]
Sep 19 04:46:21.883755 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:46:21.880373    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-87e3e6/embedding-w0-87e3e6-kserve-64c8966d59-4r28w"
Sep 19 04:46:21.884582 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:46:21.884548    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-87e3e6\"/\"embedding-w0-87e3e6-kserve-self-signed-certs\""
Sep 19 04:46:21.902910 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:46:21.902882    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-87e3e6/embedding-w0-87e3e6-kserve-64c8966d59-4r28w"]
Sep 19 04:46:21.993104 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:46:21.993064    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27bea02f-f375-4c7b-a50d-96123a14255f-dshm\") pod \"embedding-w0-87e3e6-kserve-64c8966d59-4r28w\" (UID: \"27bea02f-f375-4c7b-a50d-96123a14255f\") " pod="e2e-models-e2e-worker-w0-87e3e6/embedding-w0-87e3e6-kserve-64c8966d59-4r28w"
Sep 19 04:46:21.993284 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:46:21.993129    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27bea02f-f375-4c7b-a50d-96123a14255f-tls-certs\") pod \"embedding-w0-87e3e6-kserve-64c8966d59-4r28w\" (UID: \"27bea02f-f375-4c7b-a50d-96123a14255f\") " pod="e2e-models-e2e-worker-w0-87e3e6/embedding-w0-87e3e6-kserve-64c8966d59-4r28w"
Sep 19 04:46:21.993284 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:46:21.993175    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27bea02f-f375-4c7b-a50d-96123a14255f-tmp-dir\") pod \"embedding-w0-87e3e6-kserve-64c8966d59-4r28w\" (UID: \"27bea02f-f375-4c7b-a50d-96123a14255f\") " pod="e2e-models-e2e-worker-w0-87e3e6/embedding-w0-87e3e6-kserve-64c8966d59-4r28w"
Sep 19 04:46:21.993284 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:46:21.993202    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27bea02f-f375-4c7b-a50d-96123a14255f-home\") pod \"embedding-w0-87e3e6-kserve-64c8966d59-4r28w\" (UID: \"27bea02f-f375-4c7b-a50d-96123a14255f\") " pod="e2e-models-e2e-worker-w0-87e3e6/embedding-w0-87e3e6-kserve-64c8966d59-4r28w"
Sep 19 04:46:21.993284 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:46:21.993227    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27bea02f-f375-4c7b-a50d-96123a14255f-model-cache\") pod \"embedding-w0-87e3e6-kserve-64c8966d59-4r28w\" (UID: \"27bea02f-f375-4c7b-a50d-96123a14255f\") " pod="e2e-models-e2e-worker-w0-87e3e6/embedding-w0-87e3e6-kserve-64c8966d59-4r28w"
Sep 19 04:46:21.993492 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:46:21.993310    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ds8ng\" (UniqueName: \"kubernetes.io/projected/27bea02f-f375-4c7b-a50d-96123a14255f-kube-api-access-ds8ng\") pod \"embedding-w0-87e3e6-kserve-64c8966d59-4r28w\" (UID: \"27bea02f-f375-4c7b-a50d-96123a14255f\") " pod="e2e-models-e2e-worker-w0-87e3e6/embedding-w0-87e3e6-kserve-64c8966d59-4r28w"
Sep 19 04:46:22.094871 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:46:22.094831    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27bea02f-f375-4c7b-a50d-96123a14255f-tmp-dir\") pod \"embedding-w0-87e3e6-kserve-64c8966d59-4r28w\" (UID: \"27bea02f-f375-4c7b-a50d-96123a14255f\") " pod="e2e-models-e2e-worker-w0-87e3e6/embedding-w0-87e3e6-kserve-64c8966d59-4r28w"
Sep 19 04:46:22.095021 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:46:22.094888    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27bea02f-f375-4c7b-a50d-96123a14255f-home\") pod \"embedding-w0-87e3e6-kserve-64c8966d59-4r28w\" (UID: \"27bea02f-f375-4c7b-a50d-96123a14255f\") " pod="e2e-models-e2e-worker-w0-87e3e6/embedding-w0-87e3e6-kserve-64c8966d59-4r28w"
Sep 19 04:46:22.095021 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:46:22.094919    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27bea02f-f375-4c7b-a50d-96123a14255f-model-cache\") pod \"embedding-w0-87e3e6-kserve-64c8966d59-4r28w\" (UID: \"27bea02f-f375-4c7b-a50d-96123a14255f\") " pod="e2e-models-e2e-worker-w0-87e3e6/embedding-w0-87e3e6-kserve-64c8966d59-4r28w"
Sep 19 04:46:22.095021 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:46:22.095009    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ds8ng\" (UniqueName: \"kubernetes.io/projected/27bea02f-f375-4c7b-a50d-96123a14255f-kube-api-access-ds8ng\") pod \"embedding-w0-87e3e6-kserve-64c8966d59-4r28w\" (UID: \"27bea02f-f375-4c7b-a50d-96123a14255f\") " pod="e2e-models-e2e-worker-w0-87e3e6/embedding-w0-87e3e6-kserve-64c8966d59-4r28w"
Sep 19 04:46:22.095173 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:46:22.095125    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27bea02f-f375-4c7b-a50d-96123a14255f-dshm\") pod \"embedding-w0-87e3e6-kserve-64c8966d59-4r28w\" (UID: \"27bea02f-f375-4c7b-a50d-96123a14255f\") " pod="e2e-models-e2e-worker-w0-87e3e6/embedding-w0-87e3e6-kserve-64c8966d59-4r28w"
Sep 19 04:46:22.095231 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:46:22.095176    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27bea02f-f375-4c7b-a50d-96123a14255f-tls-certs\") pod \"embedding-w0-87e3e6-kserve-64c8966d59-4r28w\" (UID: \"27bea02f-f375-4c7b-a50d-96123a14255f\") " pod="e2e-models-e2e-worker-w0-87e3e6/embedding-w0-87e3e6-kserve-64c8966d59-4r28w"
Sep 19 04:46:22.095328 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:46:22.095302    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27bea02f-f375-4c7b-a50d-96123a14255f-home\") pod \"embedding-w0-87e3e6-kserve-64c8966d59-4r28w\" (UID: \"27bea02f-f375-4c7b-a50d-96123a14255f\") " pod="e2e-models-e2e-worker-w0-87e3e6/embedding-w0-87e3e6-kserve-64c8966d59-4r28w"
Sep 19 04:46:22.095328 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:46:22.095305    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27bea02f-f375-4c7b-a50d-96123a14255f-tmp-dir\") pod \"embedding-w0-87e3e6-kserve-64c8966d59-4r28w\" (UID: \"27bea02f-f375-4c7b-a50d-96123a14255f\") " pod="e2e-models-e2e-worker-w0-87e3e6/embedding-w0-87e3e6-kserve-64c8966d59-4r28w"
Sep 19 04:46:22.095474 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:46:22.095463    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27bea02f-f375-4c7b-a50d-96123a14255f-model-cache\") pod \"embedding-w0-87e3e6-kserve-64c8966d59-4r28w\" (UID: \"27bea02f-f375-4c7b-a50d-96123a14255f\") " pod="e2e-models-e2e-worker-w0-87e3e6/embedding-w0-87e3e6-kserve-64c8966d59-4r28w"
Sep 19 04:46:22.097614 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:46:22.097574    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27bea02f-f375-4c7b-a50d-96123a14255f-dshm\") pod \"embedding-w0-87e3e6-kserve-64c8966d59-4r28w\" (UID: \"27bea02f-f375-4c7b-a50d-96123a14255f\") " pod="e2e-models-e2e-worker-w0-87e3e6/embedding-w0-87e3e6-kserve-64c8966d59-4r28w"
Sep 19 04:46:22.097883 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:46:22.097857    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27bea02f-f375-4c7b-a50d-96123a14255f-tls-certs\") pod \"embedding-w0-87e3e6-kserve-64c8966d59-4r28w\" (UID: \"27bea02f-f375-4c7b-a50d-96123a14255f\") " pod="e2e-models-e2e-worker-w0-87e3e6/embedding-w0-87e3e6-kserve-64c8966d59-4r28w"
Sep 19 04:46:22.108312 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:46:22.108276    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ds8ng\" (UniqueName: \"kubernetes.io/projected/27bea02f-f375-4c7b-a50d-96123a14255f-kube-api-access-ds8ng\") pod \"embedding-w0-87e3e6-kserve-64c8966d59-4r28w\" (UID: \"27bea02f-f375-4c7b-a50d-96123a14255f\") " pod="e2e-models-e2e-worker-w0-87e3e6/embedding-w0-87e3e6-kserve-64c8966d59-4r28w"
Sep 19 04:46:22.196356 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:46:22.196102    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-87e3e6/embedding-w0-87e3e6-kserve-64c8966d59-4r28w"
Sep 19 04:46:22.478752 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:46:22.478690    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-87e3e6/embedding-w0-87e3e6-kserve-64c8966d59-4r28w"]
Sep 19 04:46:22.479770 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:46:22.479746    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod27bea02f_f375_4c7b_a50d_96123a14255f.slice/crio-382eb3f85b68db24b9ef3f574e631d826a0db1e09d15f5b765cd49c661bbcf18 WatchSource:0}: Error finding container 382eb3f85b68db24b9ef3f574e631d826a0db1e09d15f5b765cd49c661bbcf18: Status 404 returned error can't find the container with id 382eb3f85b68db24b9ef3f574e631d826a0db1e09d15f5b765cd49c661bbcf18
Sep 19 04:46:23.406662 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:46:23.406627    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-87e3e6/embedding-w0-87e3e6-kserve-64c8966d59-4r28w" event={"ID":"27bea02f-f375-4c7b-a50d-96123a14255f","Type":"ContainerStarted","Data":"e6cd84027bd83605d1d60fbb20ab6cbfad06ba4ae36c5dd90eccd71d4fe92ed7"}
Sep 19 04:46:23.406662 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:46:23.406668    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-87e3e6/embedding-w0-87e3e6-kserve-64c8966d59-4r28w" event={"ID":"27bea02f-f375-4c7b-a50d-96123a14255f","Type":"ContainerStarted","Data":"382eb3f85b68db24b9ef3f574e631d826a0db1e09d15f5b765cd49c661bbcf18"}
Sep 19 04:46:23.433647 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:46:23.433594    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-87e3e6/embedding-w0-87e3e6-kserve-64c8966d59-4r28w" podStartSLOduration=2.433577674 podStartE2EDuration="2.433577674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 04:46:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:46:23.432732264 +0000 UTC m=+1962.615693414" watchObservedRunningTime="2026-09-19 04:46:23.433577674 +0000 UTC m=+1962.616538812"
Sep 19 04:46:24.197075 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:46:24.197039    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-87e3e6/embedding-w0-87e3e6-kserve-64c8966d59-4r28w"
Sep 19 04:46:32.196888 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:46:32.196788    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-87e3e6/embedding-w0-87e3e6-kserve-64c8966d59-4r28w"
Sep 19 04:46:32.209775 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:46:32.209748    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-87e3e6/embedding-w0-87e3e6-kserve-64c8966d59-4r28w"
Sep 19 04:46:32.475767 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:46:32.475655    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-87e3e6/embedding-w0-87e3e6-kserve-64c8966d59-4r28w"
Sep 19 04:46:33.454323 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:46:33.454288    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-29f7a5-7fb745b74f-mfm7s"]
Sep 19 04:46:33.457844 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:46:33.457823    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-29f7a5-7fb745b74f-mfm7s"
Sep 19 04:46:33.460940 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:46:33.460916    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-29f7a5-dockercfg-6cbgf\""
Sep 19 04:46:33.461564 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:46:33.461540    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-29f7a5\""
Sep 19 04:46:33.474264 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:46:33.474237    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-29f7a5-7fb745b74f-mfm7s"]
Sep 19 04:46:33.628881 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:46:33.628841    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/637dcb9b-3570-496e-a009-134f1b3ea829-plugins-config-volume\") pod \"payload-processing-e2e-api-b-29f7a5-7fb745b74f-mfm7s\" (UID: \"637dcb9b-3570-496e-a009-134f1b3ea829\") " pod="openshift-ingress/payload-processing-e2e-api-b-29f7a5-7fb745b74f-mfm7s"
Sep 19 04:46:33.629068 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:46:33.628995    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkqn7\" (UniqueName: \"kubernetes.io/projected/637dcb9b-3570-496e-a009-134f1b3ea829-kube-api-access-nkqn7\") pod \"payload-processing-e2e-api-b-29f7a5-7fb745b74f-mfm7s\" (UID: \"637dcb9b-3570-496e-a009-134f1b3ea829\") " pod="openshift-ingress/payload-processing-e2e-api-b-29f7a5-7fb745b74f-mfm7s"
Sep 19 04:46:33.730259 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:46:33.730176    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/637dcb9b-3570-496e-a009-134f1b3ea829-plugins-config-volume\") pod \"payload-processing-e2e-api-b-29f7a5-7fb745b74f-mfm7s\" (UID: \"637dcb9b-3570-496e-a009-134f1b3ea829\") " pod="openshift-ingress/payload-processing-e2e-api-b-29f7a5-7fb745b74f-mfm7s"
Sep 19 04:46:33.730421 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:46:33.730294    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkqn7\" (UniqueName: \"kubernetes.io/projected/637dcb9b-3570-496e-a009-134f1b3ea829-kube-api-access-nkqn7\") pod \"payload-processing-e2e-api-b-29f7a5-7fb745b74f-mfm7s\" (UID: \"637dcb9b-3570-496e-a009-134f1b3ea829\") " pod="openshift-ingress/payload-processing-e2e-api-b-29f7a5-7fb745b74f-mfm7s"
Sep 19 04:46:33.731164 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:46:33.731144    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/637dcb9b-3570-496e-a009-134f1b3ea829-plugins-config-volume\") pod \"payload-processing-e2e-api-b-29f7a5-7fb745b74f-mfm7s\" (UID: \"637dcb9b-3570-496e-a009-134f1b3ea829\") " pod="openshift-ingress/payload-processing-e2e-api-b-29f7a5-7fb745b74f-mfm7s"
Sep 19 04:46:33.738961 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:46:33.738940    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkqn7\" (UniqueName: \"kubernetes.io/projected/637dcb9b-3570-496e-a009-134f1b3ea829-kube-api-access-nkqn7\") pod \"payload-processing-e2e-api-b-29f7a5-7fb745b74f-mfm7s\" (UID: \"637dcb9b-3570-496e-a009-134f1b3ea829\") " pod="openshift-ingress/payload-processing-e2e-api-b-29f7a5-7fb745b74f-mfm7s"
Sep 19 04:46:33.771688 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:46:33.771660    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-29f7a5-7fb745b74f-mfm7s"
Sep 19 04:46:34.052939 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:46:34.052910    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-29f7a5-7fb745b74f-mfm7s"]
Sep 19 04:46:34.053796 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:46:34.053768    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod637dcb9b_3570_496e_a009_134f1b3ea829.slice/crio-50d0bbc830ca24ac20d4471b963990a8814ad4b8870ff3c58d29b2b24beb623a WatchSource:0}: Error finding container 50d0bbc830ca24ac20d4471b963990a8814ad4b8870ff3c58d29b2b24beb623a: Status 404 returned error can't find the container with id 50d0bbc830ca24ac20d4471b963990a8814ad4b8870ff3c58d29b2b24beb623a
Sep 19 04:46:34.482436 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:46:34.482386    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-29f7a5-7fb745b74f-mfm7s" event={"ID":"637dcb9b-3570-496e-a009-134f1b3ea829","Type":"ContainerStarted","Data":"7ea332e1fd8b683e772d6955ebe7c0cf4c9250225659451959db1747310fc24d"}
Sep 19 04:46:34.482958 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:46:34.482453    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-29f7a5-7fb745b74f-mfm7s" event={"ID":"637dcb9b-3570-496e-a009-134f1b3ea829","Type":"ContainerStarted","Data":"50d0bbc830ca24ac20d4471b963990a8814ad4b8870ff3c58d29b2b24beb623a"}
Sep 19 04:46:34.483189 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:46:34.483166    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-29f7a5-7fb745b74f-mfm7s"
Sep 19 04:46:34.503885 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:46:34.503829    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-29f7a5-7fb745b74f-mfm7s" podStartSLOduration=1.503811526 podStartE2EDuration="1.503811526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 04:46:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:46:34.502016563 +0000 UTC m=+1973.684977702" watchObservedRunningTime="2026-09-19 04:46:34.503811526 +0000 UTC m=+1973.686772673"
Sep 19 04:46:41.679673 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:46:41.679634    2579 scope.go:117] "RemoveContainer" containerID="ec3aeeba1b7cb8917eb8777b0e9255ff2a190eb1befbc7d9a81a7336fb5afe6a"
Sep 19 04:46:45.491340 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:46:45.491303    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-29f7a5-7fb745b74f-mfm7s"
Sep 19 04:47:00.197173 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:00.197124    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1a212c-x2jcr"]
Sep 19 04:47:00.201189 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:00.201166    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1a212c-x2jcr"
Sep 19 04:47:00.203923 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:00.203900    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 19 04:47:00.204057 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:00.203904    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s6xbq\""
Sep 19 04:47:00.214359 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:00.214315    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1a212c-x2jcr"]
Sep 19 04:47:00.306064 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:00.306019    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/deaa594d-5680-4a27-b67c-b41b68bcc6f6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1a212c-x2jcr\" (UID: \"deaa594d-5680-4a27-b67c-b41b68bcc6f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1a212c-x2jcr"
Sep 19 04:47:00.407263 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:00.407221    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/deaa594d-5680-4a27-b67c-b41b68bcc6f6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1a212c-x2jcr\" (UID: \"deaa594d-5680-4a27-b67c-b41b68bcc6f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1a212c-x2jcr"
Sep 19 04:47:00.407978 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:00.407956    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/deaa594d-5680-4a27-b67c-b41b68bcc6f6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1a212c-x2jcr\" (UID: \"deaa594d-5680-4a27-b67c-b41b68bcc6f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1a212c-x2jcr"
Sep 19 04:47:00.515496 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:00.515409    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1a212c-x2jcr"
Sep 19 04:47:00.796973 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:00.796949    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1a212c-x2jcr"]
Sep 19 04:47:00.798874 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:47:00.798821    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddeaa594d_5680_4a27_b67c_b41b68bcc6f6.slice/crio-fae492650e17f3c1be57f342583d5e8c2f9c3267a1364bd2f6623a87de3a242e WatchSource:0}: Error finding container fae492650e17f3c1be57f342583d5e8c2f9c3267a1364bd2f6623a87de3a242e: Status 404 returned error can't find the container with id fae492650e17f3c1be57f342583d5e8c2f9c3267a1364bd2f6623a87de3a242e
Sep 19 04:47:01.665061 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:01.665020    2579 generic.go:356] "Generic (PLEG): container finished" podID="deaa594d-5680-4a27-b67c-b41b68bcc6f6" containerID="ce7f46575784b9c36c1f08357a6407c1be806833cc39098d3ea57426520c6887" exitCode=0
Sep 19 04:47:01.665503 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:01.665081    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1a212c-x2jcr" event={"ID":"deaa594d-5680-4a27-b67c-b41b68bcc6f6","Type":"ContainerDied","Data":"ce7f46575784b9c36c1f08357a6407c1be806833cc39098d3ea57426520c6887"}
Sep 19 04:47:01.665503 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:01.665145    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1a212c-x2jcr" event={"ID":"deaa594d-5680-4a27-b67c-b41b68bcc6f6","Type":"ContainerStarted","Data":"fae492650e17f3c1be57f342583d5e8c2f9c3267a1364bd2f6623a87de3a242e"}
Sep 19 04:47:02.826622 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:02.826597    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1a212c-x2jcr"
Sep 19 04:47:02.935143 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:02.935044    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/deaa594d-5680-4a27-b67c-b41b68bcc6f6-maas-api-service-ca\") pod \"deaa594d-5680-4a27-b67c-b41b68bcc6f6\" (UID: \"deaa594d-5680-4a27-b67c-b41b68bcc6f6\") "
Sep 19 04:47:02.935466 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:02.935439    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/deaa594d-5680-4a27-b67c-b41b68bcc6f6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "deaa594d-5680-4a27-b67c-b41b68bcc6f6" (UID: "deaa594d-5680-4a27-b67c-b41b68bcc6f6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 04:47:03.036597 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:03.036556    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/deaa594d-5680-4a27-b67c-b41b68bcc6f6-maas-api-service-ca\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:47:03.679755 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:03.679728    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1a212c-x2jcr"
Sep 19 04:47:03.679755 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:03.679734    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1a212c-x2jcr" event={"ID":"deaa594d-5680-4a27-b67c-b41b68bcc6f6","Type":"ContainerDied","Data":"fae492650e17f3c1be57f342583d5e8c2f9c3267a1364bd2f6623a87de3a242e"}
Sep 19 04:47:03.679983 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:03.679791    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fae492650e17f3c1be57f342583d5e8c2f9c3267a1364bd2f6623a87de3a242e"
Sep 19 04:47:16.091845 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:16.091812    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1a212c-6585f9b8ff-s877b"]
Sep 19 04:47:16.092446 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:16.092064    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-1a212c-6585f9b8ff-s877b" podUID="087b9c85-ead3-4c23-8e6e-9cfc0d3635bb" containerName="payload-processing" containerID="cri-o://991b068dac945bc94531c7dc73d197a0b03173f383ef53cb15d7b26eca36a90d" gracePeriod=30
Sep 19 04:47:16.329867 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:16.329836    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1a212c-x2jcr"]
Sep 19 04:47:16.335108 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:16.335074    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1a212c-x2jcr"]
Sep 19 04:47:16.389667 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:16.389618    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1a212c-6585f9b8ff-s877b"
Sep 19 04:47:16.474537 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:16.474504    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-flnbz\" (UniqueName: \"kubernetes.io/projected/087b9c85-ead3-4c23-8e6e-9cfc0d3635bb-kube-api-access-flnbz\") pod \"087b9c85-ead3-4c23-8e6e-9cfc0d3635bb\" (UID: \"087b9c85-ead3-4c23-8e6e-9cfc0d3635bb\") "
Sep 19 04:47:16.474768 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:16.474614    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/087b9c85-ead3-4c23-8e6e-9cfc0d3635bb-plugins-config-volume\") pod \"087b9c85-ead3-4c23-8e6e-9cfc0d3635bb\" (UID: \"087b9c85-ead3-4c23-8e6e-9cfc0d3635bb\") "
Sep 19 04:47:16.474992 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:16.474969    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/087b9c85-ead3-4c23-8e6e-9cfc0d3635bb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "087b9c85-ead3-4c23-8e6e-9cfc0d3635bb" (UID: "087b9c85-ead3-4c23-8e6e-9cfc0d3635bb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 04:47:16.477315 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:16.477285    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/087b9c85-ead3-4c23-8e6e-9cfc0d3635bb-kube-api-access-flnbz" (OuterVolumeSpecName: "kube-api-access-flnbz") pod "087b9c85-ead3-4c23-8e6e-9cfc0d3635bb" (UID: "087b9c85-ead3-4c23-8e6e-9cfc0d3635bb"). InnerVolumeSpecName "kube-api-access-flnbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 04:47:16.576019 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:16.575986    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flnbz\" (UniqueName: \"kubernetes.io/projected/087b9c85-ead3-4c23-8e6e-9cfc0d3635bb-kube-api-access-flnbz\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:47:16.576019 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:16.576011    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/087b9c85-ead3-4c23-8e6e-9cfc0d3635bb-plugins-config-volume\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:47:16.764799 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:16.764763    2579 generic.go:356] "Generic (PLEG): container finished" podID="087b9c85-ead3-4c23-8e6e-9cfc0d3635bb" containerID="991b068dac945bc94531c7dc73d197a0b03173f383ef53cb15d7b26eca36a90d" exitCode=0
Sep 19 04:47:16.764969 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:16.764808    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1a212c-6585f9b8ff-s877b"
Sep 19 04:47:16.764969 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:16.764850    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1a212c-6585f9b8ff-s877b" event={"ID":"087b9c85-ead3-4c23-8e6e-9cfc0d3635bb","Type":"ContainerDied","Data":"991b068dac945bc94531c7dc73d197a0b03173f383ef53cb15d7b26eca36a90d"}
Sep 19 04:47:16.764969 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:16.764902    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1a212c-6585f9b8ff-s877b" event={"ID":"087b9c85-ead3-4c23-8e6e-9cfc0d3635bb","Type":"ContainerDied","Data":"ad54e0f70a508b78cd53b64de15cfecfb080f75ab011cf9aa4db761c54f6587e"}
Sep 19 04:47:16.764969 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:16.764924    2579 scope.go:117] "RemoveContainer" containerID="991b068dac945bc94531c7dc73d197a0b03173f383ef53cb15d7b26eca36a90d"
Sep 19 04:47:16.774538 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:16.774519    2579 scope.go:117] "RemoveContainer" containerID="991b068dac945bc94531c7dc73d197a0b03173f383ef53cb15d7b26eca36a90d"
Sep 19 04:47:16.774871 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:47:16.774852    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"991b068dac945bc94531c7dc73d197a0b03173f383ef53cb15d7b26eca36a90d\": container with ID starting with 991b068dac945bc94531c7dc73d197a0b03173f383ef53cb15d7b26eca36a90d not found: ID does not exist" containerID="991b068dac945bc94531c7dc73d197a0b03173f383ef53cb15d7b26eca36a90d"
Sep 19 04:47:16.774924 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:16.774882    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"991b068dac945bc94531c7dc73d197a0b03173f383ef53cb15d7b26eca36a90d"} err="failed to get container status \"991b068dac945bc94531c7dc73d197a0b03173f383ef53cb15d7b26eca36a90d\": rpc error: code = NotFound desc = could not find container \"991b068dac945bc94531c7dc73d197a0b03173f383ef53cb15d7b26eca36a90d\": container with ID starting with 991b068dac945bc94531c7dc73d197a0b03173f383ef53cb15d7b26eca36a90d not found: ID does not exist"
Sep 19 04:47:16.792370 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:16.792325    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1a212c-6585f9b8ff-s877b"]
Sep 19 04:47:16.796517 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:16.796488    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1a212c-6585f9b8ff-s877b"]
Sep 19 04:47:17.352115 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:17.352077    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="087b9c85-ead3-4c23-8e6e-9cfc0d3635bb" path="/var/lib/kubelet/pods/087b9c85-ead3-4c23-8e6e-9cfc0d3635bb/volumes"
Sep 19 04:47:17.352518 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:17.352502    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="deaa594d-5680-4a27-b67c-b41b68bcc6f6" path="/var/lib/kubelet/pods/deaa594d-5680-4a27-b67c-b41b68bcc6f6/volumes"
Sep 19 04:47:24.456014 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:24.455981    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-29f7a5-vhzn9"]
Sep 19 04:47:24.456638 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:24.456619    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="087b9c85-ead3-4c23-8e6e-9cfc0d3635bb" containerName="payload-processing"
Sep 19 04:47:24.456740 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:24.456642    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="087b9c85-ead3-4c23-8e6e-9cfc0d3635bb" containerName="payload-processing"
Sep 19 04:47:24.456740 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:24.456660    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="deaa594d-5680-4a27-b67c-b41b68bcc6f6" containerName="revoke-keys"
Sep 19 04:47:24.456740 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:24.456668    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="deaa594d-5680-4a27-b67c-b41b68bcc6f6" containerName="revoke-keys"
Sep 19 04:47:24.456899 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:24.456811    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="087b9c85-ead3-4c23-8e6e-9cfc0d3635bb" containerName="payload-processing"
Sep 19 04:47:24.456899 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:24.456832    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="deaa594d-5680-4a27-b67c-b41b68bcc6f6" containerName="revoke-keys"
Sep 19 04:47:24.461201 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:24.461183    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-29f7a5-vhzn9"
Sep 19 04:47:24.464091 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:24.464071    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 19 04:47:24.464211 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:24.464182    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s6xbq\""
Sep 19 04:47:24.469935 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:24.469406    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-29f7a5-vhzn9"]
Sep 19 04:47:24.559637 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:24.559602    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d18923f-b4f9-4143-a6a9-a260564303d2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-29f7a5-vhzn9\" (UID: \"1d18923f-b4f9-4143-a6a9-a260564303d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-29f7a5-vhzn9"
Sep 19 04:47:24.660706 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:24.660660    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d18923f-b4f9-4143-a6a9-a260564303d2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-29f7a5-vhzn9\" (UID: \"1d18923f-b4f9-4143-a6a9-a260564303d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-29f7a5-vhzn9"
Sep 19 04:47:24.661289 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:24.661268    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d18923f-b4f9-4143-a6a9-a260564303d2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-29f7a5-vhzn9\" (UID: \"1d18923f-b4f9-4143-a6a9-a260564303d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-29f7a5-vhzn9"
Sep 19 04:47:24.775046 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:24.774959    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-29f7a5-vhzn9"
Sep 19 04:47:25.086442 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:25.086328    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-29f7a5-vhzn9"]
Sep 19 04:47:25.090300 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:47:25.090270    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d18923f_b4f9_4143_a6a9_a260564303d2.slice/crio-db0eee09a9cbef33a435e5fe5b12fa2f8a1652f75a3b14b07551b2a4cc8a45cb WatchSource:0}: Error finding container db0eee09a9cbef33a435e5fe5b12fa2f8a1652f75a3b14b07551b2a4cc8a45cb: Status 404 returned error can't find the container with id db0eee09a9cbef33a435e5fe5b12fa2f8a1652f75a3b14b07551b2a4cc8a45cb
Sep 19 04:47:25.828506 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:25.828467    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-29f7a5-vhzn9" event={"ID":"1d18923f-b4f9-4143-a6a9-a260564303d2","Type":"ContainerStarted","Data":"55a70d83a3aa2ad1efd9296fd646d7d6a513e3e447eb62b7cc17e26fec4522e8"}
Sep 19 04:47:25.828906 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:25.828530    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-29f7a5-vhzn9" event={"ID":"1d18923f-b4f9-4143-a6a9-a260564303d2","Type":"ContainerStarted","Data":"db0eee09a9cbef33a435e5fe5b12fa2f8a1652f75a3b14b07551b2a4cc8a45cb"}
Sep 19 04:47:25.847622 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:25.847573    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-29f7a5-vhzn9" podStartSLOduration=1.8475579629999999 podStartE2EDuration="1.847557963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 04:47:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:47:25.844461908 +0000 UTC m=+2025.027423055" watchObservedRunningTime="2026-09-19 04:47:25.847557963 +0000 UTC m=+2025.030519110"
Sep 19 04:47:26.837310 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:26.837272    2579 generic.go:356] "Generic (PLEG): container finished" podID="1d18923f-b4f9-4143-a6a9-a260564303d2" containerID="55a70d83a3aa2ad1efd9296fd646d7d6a513e3e447eb62b7cc17e26fec4522e8" exitCode=0
Sep 19 04:47:26.837733 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:26.837354    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-29f7a5-vhzn9" event={"ID":"1d18923f-b4f9-4143-a6a9-a260564303d2","Type":"ContainerDied","Data":"55a70d83a3aa2ad1efd9296fd646d7d6a513e3e447eb62b7cc17e26fec4522e8"}
Sep 19 04:47:28.001177 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:28.001145    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-29f7a5-vhzn9"
Sep 19 04:47:28.099325 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:28.099236    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d18923f-b4f9-4143-a6a9-a260564303d2-maas-api-service-ca\") pod \"1d18923f-b4f9-4143-a6a9-a260564303d2\" (UID: \"1d18923f-b4f9-4143-a6a9-a260564303d2\") "
Sep 19 04:47:28.099592 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:28.099569    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d18923f-b4f9-4143-a6a9-a260564303d2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1d18923f-b4f9-4143-a6a9-a260564303d2" (UID: "1d18923f-b4f9-4143-a6a9-a260564303d2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 04:47:28.200625 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:28.200594    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d18923f-b4f9-4143-a6a9-a260564303d2-maas-api-service-ca\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:47:28.852108 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:28.852077    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-29f7a5-vhzn9" event={"ID":"1d18923f-b4f9-4143-a6a9-a260564303d2","Type":"ContainerDied","Data":"db0eee09a9cbef33a435e5fe5b12fa2f8a1652f75a3b14b07551b2a4cc8a45cb"}
Sep 19 04:47:28.852280 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:28.852124    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db0eee09a9cbef33a435e5fe5b12fa2f8a1652f75a3b14b07551b2a4cc8a45cb"
Sep 19 04:47:28.852280 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:28.852089    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-29f7a5-vhzn9"
Sep 19 04:47:40.079306 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:40.079271    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-29f7a5-7fb745b74f-mfm7s"]
Sep 19 04:47:40.079952 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:40.079523    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-29f7a5-7fb745b74f-mfm7s" podUID="637dcb9b-3570-496e-a009-134f1b3ea829" containerName="payload-processing" containerID="cri-o://7ea332e1fd8b683e772d6955ebe7c0cf4c9250225659451959db1747310fc24d" gracePeriod=30
Sep 19 04:47:40.345452 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:40.345389    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-29f7a5-7fb745b74f-mfm7s"
Sep 19 04:47:40.346864 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:40.346838    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-29f7a5-vhzn9"]
Sep 19 04:47:40.360653 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:40.360627    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-29f7a5-vhzn9"]
Sep 19 04:47:40.527301 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:40.527262    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/637dcb9b-3570-496e-a009-134f1b3ea829-plugins-config-volume\") pod \"637dcb9b-3570-496e-a009-134f1b3ea829\" (UID: \"637dcb9b-3570-496e-a009-134f1b3ea829\") "
Sep 19 04:47:40.527497 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:40.527394    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nkqn7\" (UniqueName: \"kubernetes.io/projected/637dcb9b-3570-496e-a009-134f1b3ea829-kube-api-access-nkqn7\") pod \"637dcb9b-3570-496e-a009-134f1b3ea829\" (UID: \"637dcb9b-3570-496e-a009-134f1b3ea829\") "
Sep 19 04:47:40.527741 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:40.527682    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/637dcb9b-3570-496e-a009-134f1b3ea829-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "637dcb9b-3570-496e-a009-134f1b3ea829" (UID: "637dcb9b-3570-496e-a009-134f1b3ea829"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 04:47:40.529782 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:40.529761    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/637dcb9b-3570-496e-a009-134f1b3ea829-kube-api-access-nkqn7" (OuterVolumeSpecName: "kube-api-access-nkqn7") pod "637dcb9b-3570-496e-a009-134f1b3ea829" (UID: "637dcb9b-3570-496e-a009-134f1b3ea829"). InnerVolumeSpecName "kube-api-access-nkqn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 04:47:40.628969 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:40.628890    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkqn7\" (UniqueName: \"kubernetes.io/projected/637dcb9b-3570-496e-a009-134f1b3ea829-kube-api-access-nkqn7\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:47:40.628969 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:40.628923    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/637dcb9b-3570-496e-a009-134f1b3ea829-plugins-config-volume\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:47:40.929231 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:40.929189    2579 generic.go:356] "Generic (PLEG): container finished" podID="637dcb9b-3570-496e-a009-134f1b3ea829" containerID="7ea332e1fd8b683e772d6955ebe7c0cf4c9250225659451959db1747310fc24d" exitCode=0
Sep 19 04:47:40.929430 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:40.929303    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-29f7a5-7fb745b74f-mfm7s"
Sep 19 04:47:40.929430 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:40.929314    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-29f7a5-7fb745b74f-mfm7s" event={"ID":"637dcb9b-3570-496e-a009-134f1b3ea829","Type":"ContainerDied","Data":"7ea332e1fd8b683e772d6955ebe7c0cf4c9250225659451959db1747310fc24d"}
Sep 19 04:47:40.929430 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:40.929363    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-29f7a5-7fb745b74f-mfm7s" event={"ID":"637dcb9b-3570-496e-a009-134f1b3ea829","Type":"ContainerDied","Data":"50d0bbc830ca24ac20d4471b963990a8814ad4b8870ff3c58d29b2b24beb623a"}
Sep 19 04:47:40.929430 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:40.929392    2579 scope.go:117] "RemoveContainer" containerID="7ea332e1fd8b683e772d6955ebe7c0cf4c9250225659451959db1747310fc24d"
Sep 19 04:47:40.939903 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:40.939879    2579 scope.go:117] "RemoveContainer" containerID="7ea332e1fd8b683e772d6955ebe7c0cf4c9250225659451959db1747310fc24d"
Sep 19 04:47:40.940254 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:47:40.940218    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ea332e1fd8b683e772d6955ebe7c0cf4c9250225659451959db1747310fc24d\": container with ID starting with 7ea332e1fd8b683e772d6955ebe7c0cf4c9250225659451959db1747310fc24d not found: ID does not exist" containerID="7ea332e1fd8b683e772d6955ebe7c0cf4c9250225659451959db1747310fc24d"
Sep 19 04:47:40.940341 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:40.940250    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ea332e1fd8b683e772d6955ebe7c0cf4c9250225659451959db1747310fc24d"} err="failed to get container status \"7ea332e1fd8b683e772d6955ebe7c0cf4c9250225659451959db1747310fc24d\": rpc error: code = NotFound desc = could not find container \"7ea332e1fd8b683e772d6955ebe7c0cf4c9250225659451959db1747310fc24d\": container with ID starting with 7ea332e1fd8b683e772d6955ebe7c0cf4c9250225659451959db1747310fc24d not found: ID does not exist"
Sep 19 04:47:40.977446 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:40.977413    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-29f7a5-7fb745b74f-mfm7s"]
Sep 19 04:47:40.983853 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:40.983820    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-29f7a5-7fb745b74f-mfm7s"]
Sep 19 04:47:41.360967 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:41.360892    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d18923f-b4f9-4143-a6a9-a260564303d2" path="/var/lib/kubelet/pods/1d18923f-b4f9-4143-a6a9-a260564303d2/volumes"
Sep 19 04:47:41.361343 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:41.361245    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="637dcb9b-3570-496e-a009-134f1b3ea829" path="/var/lib/kubelet/pods/637dcb9b-3570-496e-a009-134f1b3ea829/volumes"
Sep 19 04:47:47.625354 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:47.625315    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c1a7b6-sggh4"]
Sep 19 04:47:47.625864 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:47.625745    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d18923f-b4f9-4143-a6a9-a260564303d2" containerName="revoke-keys"
Sep 19 04:47:47.625864 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:47.625757    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d18923f-b4f9-4143-a6a9-a260564303d2" containerName="revoke-keys"
Sep 19 04:47:47.625864 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:47.625780    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="637dcb9b-3570-496e-a009-134f1b3ea829" containerName="payload-processing"
Sep 19 04:47:47.625864 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:47.625787    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="637dcb9b-3570-496e-a009-134f1b3ea829" containerName="payload-processing"
Sep 19 04:47:47.625864 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:47.625856    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="637dcb9b-3570-496e-a009-134f1b3ea829" containerName="payload-processing"
Sep 19 04:47:47.625864 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:47.625864    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d18923f-b4f9-4143-a6a9-a260564303d2" containerName="revoke-keys"
Sep 19 04:47:47.629156 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:47.629139    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c1a7b6-sggh4"
Sep 19 04:47:47.633435 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:47.633410    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 19 04:47:47.633752 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:47.633731    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s6xbq\""
Sep 19 04:47:47.636546 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:47.636516    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c1a7b6-sggh4"]
Sep 19 04:47:47.700301 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:47.700258    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba0aa7b7-a8a8-499a-895b-9c6f61aae2fb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-c1a7b6-sggh4\" (UID: \"ba0aa7b7-a8a8-499a-895b-9c6f61aae2fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c1a7b6-sggh4"
Sep 19 04:47:47.801463 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:47.801424    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba0aa7b7-a8a8-499a-895b-9c6f61aae2fb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-c1a7b6-sggh4\" (UID: \"ba0aa7b7-a8a8-499a-895b-9c6f61aae2fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c1a7b6-sggh4"
Sep 19 04:47:47.802072 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:47.802056    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba0aa7b7-a8a8-499a-895b-9c6f61aae2fb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-c1a7b6-sggh4\" (UID: \"ba0aa7b7-a8a8-499a-895b-9c6f61aae2fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c1a7b6-sggh4"
Sep 19 04:47:47.941733 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:47.941683    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c1a7b6-sggh4"
Sep 19 04:47:48.220872 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:48.220836    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c1a7b6-sggh4"]
Sep 19 04:47:48.222346 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:47:48.222321    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba0aa7b7_a8a8_499a_895b_9c6f61aae2fb.slice/crio-709a12ed0823009f22a4f21f4e42884404b578e9cc33419b2332878f256ee22c WatchSource:0}: Error finding container 709a12ed0823009f22a4f21f4e42884404b578e9cc33419b2332878f256ee22c: Status 404 returned error can't find the container with id 709a12ed0823009f22a4f21f4e42884404b578e9cc33419b2332878f256ee22c
Sep 19 04:47:48.303900 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:48.303868    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-87e3e6-xfz6p"]
Sep 19 04:47:48.308593 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:48.308570    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-87e3e6-xfz6p"
Sep 19 04:47:48.328491 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:48.328413    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-87e3e6-xfz6p"]
Sep 19 04:47:48.409967 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:48.409939    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c3abaa9-3d34-4e36-b08e-4d2477e5cc87-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-87e3e6-xfz6p\" (UID: \"3c3abaa9-3d34-4e36-b08e-4d2477e5cc87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-87e3e6-xfz6p"
Sep 19 04:47:48.510572 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:48.510544    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c3abaa9-3d34-4e36-b08e-4d2477e5cc87-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-87e3e6-xfz6p\" (UID: \"3c3abaa9-3d34-4e36-b08e-4d2477e5cc87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-87e3e6-xfz6p"
Sep 19 04:47:48.511668 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:48.511642    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c3abaa9-3d34-4e36-b08e-4d2477e5cc87-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-87e3e6-xfz6p\" (UID: \"3c3abaa9-3d34-4e36-b08e-4d2477e5cc87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-87e3e6-xfz6p"
Sep 19 04:47:48.618997 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:48.618962    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-87e3e6-xfz6p"
Sep 19 04:47:48.897370 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:48.897340    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-87e3e6-xfz6p"]
Sep 19 04:47:48.899382 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:47:48.899352    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c3abaa9_3d34_4e36_b08e_4d2477e5cc87.slice/crio-b9bed115da90b422a61b7f15ad5e0f7f93c453fed2ce1dfe5ffe00cf0a0bb9d5 WatchSource:0}: Error finding container b9bed115da90b422a61b7f15ad5e0f7f93c453fed2ce1dfe5ffe00cf0a0bb9d5: Status 404 returned error can't find the container with id b9bed115da90b422a61b7f15ad5e0f7f93c453fed2ce1dfe5ffe00cf0a0bb9d5
Sep 19 04:47:48.986175 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:48.986134    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c1a7b6-sggh4" event={"ID":"ba0aa7b7-a8a8-499a-895b-9c6f61aae2fb","Type":"ContainerStarted","Data":"93caab62c6bdd7a122f129c7ab9a4119aca982ba4299fda90afefff2f4da6acd"}
Sep 19 04:47:48.986342 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:48.986205    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c1a7b6-sggh4" event={"ID":"ba0aa7b7-a8a8-499a-895b-9c6f61aae2fb","Type":"ContainerStarted","Data":"709a12ed0823009f22a4f21f4e42884404b578e9cc33419b2332878f256ee22c"}
Sep 19 04:47:48.987739 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:48.987673    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-87e3e6-xfz6p" event={"ID":"3c3abaa9-3d34-4e36-b08e-4d2477e5cc87","Type":"ContainerStarted","Data":"b9bed115da90b422a61b7f15ad5e0f7f93c453fed2ce1dfe5ffe00cf0a0bb9d5"}
Sep 19 04:47:49.006073 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:49.006030    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c1a7b6-sggh4" podStartSLOduration=2.006014907 podStartE2EDuration="2.006014907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 04:47:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:47:49.003954722 +0000 UTC m=+2048.186915870" watchObservedRunningTime="2026-09-19 04:47:49.006014907 +0000 UTC m=+2048.188976054"
Sep 19 04:47:49.997578 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:49.997486    2579 generic.go:356] "Generic (PLEG): container finished" podID="ba0aa7b7-a8a8-499a-895b-9c6f61aae2fb" containerID="93caab62c6bdd7a122f129c7ab9a4119aca982ba4299fda90afefff2f4da6acd" exitCode=0
Sep 19 04:47:49.998016 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:49.997580    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c1a7b6-sggh4" event={"ID":"ba0aa7b7-a8a8-499a-895b-9c6f61aae2fb","Type":"ContainerDied","Data":"93caab62c6bdd7a122f129c7ab9a4119aca982ba4299fda90afefff2f4da6acd"}
Sep 19 04:47:49.999321 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:49.999298    2579 generic.go:356] "Generic (PLEG): container finished" podID="3c3abaa9-3d34-4e36-b08e-4d2477e5cc87" containerID="5044c1dce773da7a478137025418addec88a0508702bdac710d91d1f0e021211" exitCode=0
Sep 19 04:47:49.999428 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:49.999382    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-87e3e6-xfz6p" event={"ID":"3c3abaa9-3d34-4e36-b08e-4d2477e5cc87","Type":"ContainerDied","Data":"5044c1dce773da7a478137025418addec88a0508702bdac710d91d1f0e021211"}
Sep 19 04:47:51.160362 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:51.160324    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c1a7b6-sggh4"
Sep 19 04:47:51.196815 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:51.196785    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-87e3e6-xfz6p"
Sep 19 04:47:51.240390 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:51.240360    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba0aa7b7-a8a8-499a-895b-9c6f61aae2fb-maas-api-service-ca\") pod \"ba0aa7b7-a8a8-499a-895b-9c6f61aae2fb\" (UID: \"ba0aa7b7-a8a8-499a-895b-9c6f61aae2fb\") "
Sep 19 04:47:51.240595 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:51.240453    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c3abaa9-3d34-4e36-b08e-4d2477e5cc87-maas-api-service-ca\") pod \"3c3abaa9-3d34-4e36-b08e-4d2477e5cc87\" (UID: \"3c3abaa9-3d34-4e36-b08e-4d2477e5cc87\") "
Sep 19 04:47:51.240780 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:51.240752    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba0aa7b7-a8a8-499a-895b-9c6f61aae2fb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ba0aa7b7-a8a8-499a-895b-9c6f61aae2fb" (UID: "ba0aa7b7-a8a8-499a-895b-9c6f61aae2fb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 04:47:51.240866 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:51.240810    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c3abaa9-3d34-4e36-b08e-4d2477e5cc87-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3c3abaa9-3d34-4e36-b08e-4d2477e5cc87" (UID: "3c3abaa9-3d34-4e36-b08e-4d2477e5cc87"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 04:47:51.240939 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:51.240916    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba0aa7b7-a8a8-499a-895b-9c6f61aae2fb-maas-api-service-ca\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:47:51.241046 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:51.240942    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c3abaa9-3d34-4e36-b08e-4d2477e5cc87-maas-api-service-ca\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:47:52.013429 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:52.013400    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c1a7b6-sggh4"
Sep 19 04:47:52.013615 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:52.013432    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c1a7b6-sggh4" event={"ID":"ba0aa7b7-a8a8-499a-895b-9c6f61aae2fb","Type":"ContainerDied","Data":"709a12ed0823009f22a4f21f4e42884404b578e9cc33419b2332878f256ee22c"}
Sep 19 04:47:52.013615 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:52.013487    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="709a12ed0823009f22a4f21f4e42884404b578e9cc33419b2332878f256ee22c"
Sep 19 04:47:52.014991 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:52.014964    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-87e3e6-xfz6p" event={"ID":"3c3abaa9-3d34-4e36-b08e-4d2477e5cc87","Type":"ContainerDied","Data":"b9bed115da90b422a61b7f15ad5e0f7f93c453fed2ce1dfe5ffe00cf0a0bb9d5"}
Sep 19 04:47:52.015115 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:52.015007    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b9bed115da90b422a61b7f15ad5e0f7f93c453fed2ce1dfe5ffe00cf0a0bb9d5"
Sep 19 04:47:52.015115 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:52.014977    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-87e3e6-xfz6p"
Sep 19 04:47:53.124582 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:53.124540    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-87e3e6/distinct-2-w0-87e3e6-kserve-7694d86767-6sz6p"]
Sep 19 04:47:53.128977 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:53.128952    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-87e3e6/distinct-w0-87e3e6-kserve-54c675b78b-vsvns"]
Sep 19 04:47:53.142283 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:53.142252    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-87e3e6/embedding-w0-87e3e6-kserve-64c8966d59-4r28w"]
Sep 19 04:47:53.156533 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:53.156503    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-87e3e6/premium-w0-87e3e6-kserve-f48dd7f99-dsx42"]
Sep 19 04:47:58.505293 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:58.505245    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c1a7b6-746cd84ccd-kwzs2"]
Sep 19 04:47:58.505805 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:58.505489    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c1a7b6-746cd84ccd-kwzs2" podUID="9b1c7a21-3050-4d74-8754-9c960a4dee9b" containerName="payload-pre-processing" containerID="cri-o://44e7f419bb5b14ac7c9fa976fdc7f116ace19255c079bfc0ee9bd59823ff7e75" gracePeriod=30
Sep 19 04:47:58.697813 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:58.697778    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c1a7b6-sggh4"]
Sep 19 04:47:58.700473 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:58.700443    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c1a7b6-sggh4"]
Sep 19 04:47:58.796566 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:58.796533    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c1a7b6-746cd84ccd-kwzs2"
Sep 19 04:47:58.924734 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:58.924675    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fbdxs\" (UniqueName: \"kubernetes.io/projected/9b1c7a21-3050-4d74-8754-9c960a4dee9b-kube-api-access-fbdxs\") pod \"9b1c7a21-3050-4d74-8754-9c960a4dee9b\" (UID: \"9b1c7a21-3050-4d74-8754-9c960a4dee9b\") "
Sep 19 04:47:58.924919 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:58.924836    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b1c7a21-3050-4d74-8754-9c960a4dee9b-plugins-config-volume\") pod \"9b1c7a21-3050-4d74-8754-9c960a4dee9b\" (UID: \"9b1c7a21-3050-4d74-8754-9c960a4dee9b\") "
Sep 19 04:47:58.925234 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:58.925205    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b1c7a21-3050-4d74-8754-9c960a4dee9b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9b1c7a21-3050-4d74-8754-9c960a4dee9b" (UID: "9b1c7a21-3050-4d74-8754-9c960a4dee9b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 04:47:58.927029 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:58.927008    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b1c7a21-3050-4d74-8754-9c960a4dee9b-kube-api-access-fbdxs" (OuterVolumeSpecName: "kube-api-access-fbdxs") pod "9b1c7a21-3050-4d74-8754-9c960a4dee9b" (UID: "9b1c7a21-3050-4d74-8754-9c960a4dee9b"). InnerVolumeSpecName "kube-api-access-fbdxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 04:47:59.026761 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:59.026668    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbdxs\" (UniqueName: \"kubernetes.io/projected/9b1c7a21-3050-4d74-8754-9c960a4dee9b-kube-api-access-fbdxs\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:47:59.026761 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:59.026734    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b1c7a21-3050-4d74-8754-9c960a4dee9b-plugins-config-volume\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:47:59.065806 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:59.065768    2579 generic.go:356] "Generic (PLEG): container finished" podID="9b1c7a21-3050-4d74-8754-9c960a4dee9b" containerID="44e7f419bb5b14ac7c9fa976fdc7f116ace19255c079bfc0ee9bd59823ff7e75" exitCode=0
Sep 19 04:47:59.065992 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:59.065820    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c1a7b6-746cd84ccd-kwzs2"
Sep 19 04:47:59.065992 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:59.065867    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c1a7b6-746cd84ccd-kwzs2" event={"ID":"9b1c7a21-3050-4d74-8754-9c960a4dee9b","Type":"ContainerDied","Data":"44e7f419bb5b14ac7c9fa976fdc7f116ace19255c079bfc0ee9bd59823ff7e75"}
Sep 19 04:47:59.065992 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:59.065922    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c1a7b6-746cd84ccd-kwzs2" event={"ID":"9b1c7a21-3050-4d74-8754-9c960a4dee9b","Type":"ContainerDied","Data":"3ce0b1901e8853a2724b430a25e50a91a0c41944e7721c90e27b0fc08a6eff85"}
Sep 19 04:47:59.065992 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:59.065946    2579 scope.go:117] "RemoveContainer" containerID="44e7f419bb5b14ac7c9fa976fdc7f116ace19255c079bfc0ee9bd59823ff7e75"
Sep 19 04:47:59.076890 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:59.076871    2579 scope.go:117] "RemoveContainer" containerID="44e7f419bb5b14ac7c9fa976fdc7f116ace19255c079bfc0ee9bd59823ff7e75"
Sep 19 04:47:59.077173 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:47:59.077147    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44e7f419bb5b14ac7c9fa976fdc7f116ace19255c079bfc0ee9bd59823ff7e75\": container with ID starting with 44e7f419bb5b14ac7c9fa976fdc7f116ace19255c079bfc0ee9bd59823ff7e75 not found: ID does not exist" containerID="44e7f419bb5b14ac7c9fa976fdc7f116ace19255c079bfc0ee9bd59823ff7e75"
Sep 19 04:47:59.077274 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:59.077179    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44e7f419bb5b14ac7c9fa976fdc7f116ace19255c079bfc0ee9bd59823ff7e75"} err="failed to get container status \"44e7f419bb5b14ac7c9fa976fdc7f116ace19255c079bfc0ee9bd59823ff7e75\": rpc error: code = NotFound desc = could not find container \"44e7f419bb5b14ac7c9fa976fdc7f116ace19255c079bfc0ee9bd59823ff7e75\": container with ID starting with 44e7f419bb5b14ac7c9fa976fdc7f116ace19255c079bfc0ee9bd59823ff7e75 not found: ID does not exist"
Sep 19 04:47:59.092547 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:59.092512    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c1a7b6-746cd84ccd-kwzs2"]
Sep 19 04:47:59.097852 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:59.097829    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c1a7b6-746cd84ccd-kwzs2"]
Sep 19 04:47:59.351611 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:59.351535    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b1c7a21-3050-4d74-8754-9c960a4dee9b" path="/var/lib/kubelet/pods/9b1c7a21-3050-4d74-8754-9c960a4dee9b/volumes"
Sep 19 04:47:59.351933 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:47:59.351919    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba0aa7b7-a8a8-499a-895b-9c6f61aae2fb" path="/var/lib/kubelet/pods/ba0aa7b7-a8a8-499a-895b-9c6f61aae2fb/volumes"
Sep 19 04:48:02.313870 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:02.313789    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-87e3e6-29829885-pfkpp"]
Sep 19 04:48:02.317513 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:02.317489    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-87e3e6-29829885-pfkpp"]
Sep 19 04:48:02.350652 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:02.350622    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-87e3e6-7f7f6b4cc9-p22q5"]
Sep 19 04:48:02.350881 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:02.350826    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-87e3e6-7f7f6b4cc9-p22q5" podUID="ee412ff9-2199-4f68-a185-fe1b4c59cf78" containerName="payload-processing" containerID="cri-o://0f8224599559efb6ebfd193dbc1f309df9047b3fc23f2893773e2d4ca8b3bc8f" gracePeriod=30
Sep 19 04:48:02.357264 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:02.357185    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-87e3e6-749ffc4498-lv2h9"]
Sep 19 04:48:02.462183 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:48:02.462148    2579 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-worker-w0-87e3e6: configmap "payload-processing-plugins-e2e-worker-w0-87e3e6" not found
Sep 19 04:48:02.462344 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:48:02.462289    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/be72f493-a1a7-4627-8130-cc828a2fdd82-plugins-config-volume podName:be72f493-a1a7-4627-8130-cc828a2fdd82 nodeName:}" failed. No retries permitted until 2026-09-19 04:48:02.962273359 +0000 UTC m=+2062.145234489 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/be72f493-a1a7-4627-8130-cc828a2fdd82-plugins-config-volume") pod "payload-pre-processing-e2e-worker-w0-87e3e6-749ffc4498-lv2h9" (UID: "be72f493-a1a7-4627-8130-cc828a2fdd82") : configmap "payload-processing-plugins-e2e-worker-w0-87e3e6" not found
Sep 19 04:48:02.537184 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:02.537134    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-87e3e6-xfz6p"]
Sep 19 04:48:02.543756 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:02.543623    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-87e3e6-xfz6p"]
Sep 19 04:48:02.633510 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:02.633488    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-87e3e6-7f7f6b4cc9-p22q5"
Sep 19 04:48:02.765984 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:02.765951    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-77vbk\" (UniqueName: \"kubernetes.io/projected/ee412ff9-2199-4f68-a185-fe1b4c59cf78-kube-api-access-77vbk\") pod \"ee412ff9-2199-4f68-a185-fe1b4c59cf78\" (UID: \"ee412ff9-2199-4f68-a185-fe1b4c59cf78\") "
Sep 19 04:48:02.766171 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:02.766009    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee412ff9-2199-4f68-a185-fe1b4c59cf78-plugins-config-volume\") pod \"ee412ff9-2199-4f68-a185-fe1b4c59cf78\" (UID: \"ee412ff9-2199-4f68-a185-fe1b4c59cf78\") "
Sep 19 04:48:02.766412 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:02.766386    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee412ff9-2199-4f68-a185-fe1b4c59cf78-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ee412ff9-2199-4f68-a185-fe1b4c59cf78" (UID: "ee412ff9-2199-4f68-a185-fe1b4c59cf78"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 04:48:02.768364 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:02.768332    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee412ff9-2199-4f68-a185-fe1b4c59cf78-kube-api-access-77vbk" (OuterVolumeSpecName: "kube-api-access-77vbk") pod "ee412ff9-2199-4f68-a185-fe1b4c59cf78" (UID: "ee412ff9-2199-4f68-a185-fe1b4c59cf78"). InnerVolumeSpecName "kube-api-access-77vbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 04:48:02.867449 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:02.867364    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee412ff9-2199-4f68-a185-fe1b4c59cf78-plugins-config-volume\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:48:02.867449 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:02.867396    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77vbk\" (UniqueName: \"kubernetes.io/projected/ee412ff9-2199-4f68-a185-fe1b4c59cf78-kube-api-access-77vbk\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:48:02.969383 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:48:02.969353    2579 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-worker-w0-87e3e6: configmap "payload-processing-plugins-e2e-worker-w0-87e3e6" not found
Sep 19 04:48:02.969559 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:48:02.969423    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/be72f493-a1a7-4627-8130-cc828a2fdd82-plugins-config-volume podName:be72f493-a1a7-4627-8130-cc828a2fdd82 nodeName:}" failed. No retries permitted until 2026-09-19 04:48:03.969407859 +0000 UTC m=+2063.152368985 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/be72f493-a1a7-4627-8130-cc828a2fdd82-plugins-config-volume") pod "payload-pre-processing-e2e-worker-w0-87e3e6-749ffc4498-lv2h9" (UID: "be72f493-a1a7-4627-8130-cc828a2fdd82") : configmap "payload-processing-plugins-e2e-worker-w0-87e3e6" not found
Sep 19 04:48:03.096631 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:03.096594    2579 generic.go:356] "Generic (PLEG): container finished" podID="ee412ff9-2199-4f68-a185-fe1b4c59cf78" containerID="0f8224599559efb6ebfd193dbc1f309df9047b3fc23f2893773e2d4ca8b3bc8f" exitCode=0
Sep 19 04:48:03.096825 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:03.096645    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-87e3e6-7f7f6b4cc9-p22q5"
Sep 19 04:48:03.096825 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:03.096681    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-87e3e6-7f7f6b4cc9-p22q5" event={"ID":"ee412ff9-2199-4f68-a185-fe1b4c59cf78","Type":"ContainerDied","Data":"0f8224599559efb6ebfd193dbc1f309df9047b3fc23f2893773e2d4ca8b3bc8f"}
Sep 19 04:48:03.096825 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:03.096761    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-87e3e6-7f7f6b4cc9-p22q5" event={"ID":"ee412ff9-2199-4f68-a185-fe1b4c59cf78","Type":"ContainerDied","Data":"ee209eef5c41d59796dd5b4e4fd7e9eb1aaf2690457e882f943353edf7651bf9"}
Sep 19 04:48:03.096825 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:03.096790    2579 scope.go:117] "RemoveContainer" containerID="0f8224599559efb6ebfd193dbc1f309df9047b3fc23f2893773e2d4ca8b3bc8f"
Sep 19 04:48:03.097057 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:03.096939    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-87e3e6-749ffc4498-lv2h9" podUID="be72f493-a1a7-4627-8130-cc828a2fdd82" containerName="payload-pre-processing" containerID="cri-o://0cefaf98f7ea4927e376d3bcc9161b443f26e971bbdfff1514c34bacf07ce862" gracePeriod=30
Sep 19 04:48:03.107474 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:03.107452    2579 scope.go:117] "RemoveContainer" containerID="0f8224599559efb6ebfd193dbc1f309df9047b3fc23f2893773e2d4ca8b3bc8f"
Sep 19 04:48:03.107822 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:48:03.107802    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f8224599559efb6ebfd193dbc1f309df9047b3fc23f2893773e2d4ca8b3bc8f\": container with ID starting with 0f8224599559efb6ebfd193dbc1f309df9047b3fc23f2893773e2d4ca8b3bc8f not found: ID does not exist" containerID="0f8224599559efb6ebfd193dbc1f309df9047b3fc23f2893773e2d4ca8b3bc8f"
Sep 19 04:48:03.107884 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:03.107832    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f8224599559efb6ebfd193dbc1f309df9047b3fc23f2893773e2d4ca8b3bc8f"} err="failed to get container status \"0f8224599559efb6ebfd193dbc1f309df9047b3fc23f2893773e2d4ca8b3bc8f\": rpc error: code = NotFound desc = could not find container \"0f8224599559efb6ebfd193dbc1f309df9047b3fc23f2893773e2d4ca8b3bc8f\": container with ID starting with 0f8224599559efb6ebfd193dbc1f309df9047b3fc23f2893773e2d4ca8b3bc8f not found: ID does not exist"
Sep 19 04:48:03.139028 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:03.138963    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-87e3e6-7f7f6b4cc9-p22q5"]
Sep 19 04:48:03.141534 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:03.141512    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-87e3e6-7f7f6b4cc9-p22q5"]
Sep 19 04:48:03.345418 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:03.345393    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-87e3e6-749ffc4498-lv2h9"
Sep 19 04:48:03.352582 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:03.352557    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c3abaa9-3d34-4e36-b08e-4d2477e5cc87" path="/var/lib/kubelet/pods/3c3abaa9-3d34-4e36-b08e-4d2477e5cc87/volumes"
Sep 19 04:48:03.352920 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:03.352906    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2f06779-4e8a-402b-91b0-3b8bd3664c94" path="/var/lib/kubelet/pods/d2f06779-4e8a-402b-91b0-3b8bd3664c94/volumes"
Sep 19 04:48:03.353220 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:03.353208    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee412ff9-2199-4f68-a185-fe1b4c59cf78" path="/var/lib/kubelet/pods/ee412ff9-2199-4f68-a185-fe1b4c59cf78/volumes"
Sep 19 04:48:03.476026 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:03.475997    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be72f493-a1a7-4627-8130-cc828a2fdd82-plugins-config-volume\") pod \"be72f493-a1a7-4627-8130-cc828a2fdd82\" (UID: \"be72f493-a1a7-4627-8130-cc828a2fdd82\") "
Sep 19 04:48:03.476199 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:03.476179    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-snfkz\" (UniqueName: \"kubernetes.io/projected/be72f493-a1a7-4627-8130-cc828a2fdd82-kube-api-access-snfkz\") pod \"be72f493-a1a7-4627-8130-cc828a2fdd82\" (UID: \"be72f493-a1a7-4627-8130-cc828a2fdd82\") "
Sep 19 04:48:03.476358 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:03.476330    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be72f493-a1a7-4627-8130-cc828a2fdd82-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "be72f493-a1a7-4627-8130-cc828a2fdd82" (UID: "be72f493-a1a7-4627-8130-cc828a2fdd82"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 04:48:03.476785 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:03.476651    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be72f493-a1a7-4627-8130-cc828a2fdd82-plugins-config-volume\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:48:03.478395 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:03.478371    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be72f493-a1a7-4627-8130-cc828a2fdd82-kube-api-access-snfkz" (OuterVolumeSpecName: "kube-api-access-snfkz") pod "be72f493-a1a7-4627-8130-cc828a2fdd82" (UID: "be72f493-a1a7-4627-8130-cc828a2fdd82"). InnerVolumeSpecName "kube-api-access-snfkz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 04:48:03.577424 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:03.577385    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snfkz\" (UniqueName: \"kubernetes.io/projected/be72f493-a1a7-4627-8130-cc828a2fdd82-kube-api-access-snfkz\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:48:04.104741 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:04.104669    2579 generic.go:356] "Generic (PLEG): container finished" podID="be72f493-a1a7-4627-8130-cc828a2fdd82" containerID="0cefaf98f7ea4927e376d3bcc9161b443f26e971bbdfff1514c34bacf07ce862" exitCode=0
Sep 19 04:48:04.104741 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:04.104746    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-87e3e6-749ffc4498-lv2h9"
Sep 19 04:48:04.105114 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:04.104768    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-87e3e6-749ffc4498-lv2h9" event={"ID":"be72f493-a1a7-4627-8130-cc828a2fdd82","Type":"ContainerDied","Data":"0cefaf98f7ea4927e376d3bcc9161b443f26e971bbdfff1514c34bacf07ce862"}
Sep 19 04:48:04.105114 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:04.104836    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-87e3e6-749ffc4498-lv2h9" event={"ID":"be72f493-a1a7-4627-8130-cc828a2fdd82","Type":"ContainerDied","Data":"db7eafb5a5122746f3f2bf9648211af6714fc7cbeda3ceeba985c621c379a86b"}
Sep 19 04:48:04.105114 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:04.104866    2579 scope.go:117] "RemoveContainer" containerID="0cefaf98f7ea4927e376d3bcc9161b443f26e971bbdfff1514c34bacf07ce862"
Sep 19 04:48:04.115302 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:04.115284    2579 scope.go:117] "RemoveContainer" containerID="0cefaf98f7ea4927e376d3bcc9161b443f26e971bbdfff1514c34bacf07ce862"
Sep 19 04:48:04.115601 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:48:04.115582    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cefaf98f7ea4927e376d3bcc9161b443f26e971bbdfff1514c34bacf07ce862\": container with ID starting with 0cefaf98f7ea4927e376d3bcc9161b443f26e971bbdfff1514c34bacf07ce862 not found: ID does not exist" containerID="0cefaf98f7ea4927e376d3bcc9161b443f26e971bbdfff1514c34bacf07ce862"
Sep 19 04:48:04.115672 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:04.115616    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cefaf98f7ea4927e376d3bcc9161b443f26e971bbdfff1514c34bacf07ce862"} err="failed to get container status \"0cefaf98f7ea4927e376d3bcc9161b443f26e971bbdfff1514c34bacf07ce862\": rpc error: code = NotFound desc = could not find container \"0cefaf98f7ea4927e376d3bcc9161b443f26e971bbdfff1514c34bacf07ce862\": container with ID starting with 0cefaf98f7ea4927e376d3bcc9161b443f26e971bbdfff1514c34bacf07ce862 not found: ID does not exist"
Sep 19 04:48:04.129078 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:04.129053    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-87e3e6-749ffc4498-lv2h9"]
Sep 19 04:48:04.140722 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:04.140681    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-87e3e6-749ffc4498-lv2h9"]
Sep 19 04:48:05.352194 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:05.352157    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be72f493-a1a7-4627-8130-cc828a2fdd82" path="/var/lib/kubelet/pods/be72f493-a1a7-4627-8130-cc828a2fdd82/volumes"
Sep 19 04:48:06.600190 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:06.600141    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-eac6c5-fd8pl"]
Sep 19 04:48:06.600737 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:06.600572    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be72f493-a1a7-4627-8130-cc828a2fdd82" containerName="payload-pre-processing"
Sep 19 04:48:06.600737 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:06.600584    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="be72f493-a1a7-4627-8130-cc828a2fdd82" containerName="payload-pre-processing"
Sep 19 04:48:06.600737 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:06.600599    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b1c7a21-3050-4d74-8754-9c960a4dee9b" containerName="payload-pre-processing"
Sep 19 04:48:06.600737 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:06.600605    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b1c7a21-3050-4d74-8754-9c960a4dee9b" containerName="payload-pre-processing"
Sep 19 04:48:06.600737 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:06.600612    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee412ff9-2199-4f68-a185-fe1b4c59cf78" containerName="payload-processing"
Sep 19 04:48:06.600737 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:06.600618    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee412ff9-2199-4f68-a185-fe1b4c59cf78" containerName="payload-processing"
Sep 19 04:48:06.600737 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:06.600626    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c3abaa9-3d34-4e36-b08e-4d2477e5cc87" containerName="revoke-keys"
Sep 19 04:48:06.600737 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:06.600632    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c3abaa9-3d34-4e36-b08e-4d2477e5cc87" containerName="revoke-keys"
Sep 19 04:48:06.600737 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:06.600649    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba0aa7b7-a8a8-499a-895b-9c6f61aae2fb" containerName="revoke-keys"
Sep 19 04:48:06.600737 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:06.600654    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba0aa7b7-a8a8-499a-895b-9c6f61aae2fb" containerName="revoke-keys"
Sep 19 04:48:06.601276 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:06.600757    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b1c7a21-3050-4d74-8754-9c960a4dee9b" containerName="payload-pre-processing"
Sep 19 04:48:06.601276 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:06.600769    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba0aa7b7-a8a8-499a-895b-9c6f61aae2fb" containerName="revoke-keys"
Sep 19 04:48:06.601276 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:06.600777    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee412ff9-2199-4f68-a185-fe1b4c59cf78" containerName="payload-processing"
Sep 19 04:48:06.601276 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:06.600786    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="be72f493-a1a7-4627-8130-cc828a2fdd82" containerName="payload-pre-processing"
Sep 19 04:48:06.601276 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:06.600796    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c3abaa9-3d34-4e36-b08e-4d2477e5cc87" containerName="revoke-keys"
Sep 19 04:48:06.605475 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:06.605448    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-eac6c5-fd8pl"
Sep 19 04:48:06.609103 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:06.609079    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 19 04:48:06.609103 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:06.609086    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s6xbq\""
Sep 19 04:48:06.619534 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:06.619504    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-eac6c5-fd8pl"]
Sep 19 04:48:06.713637 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:06.713608    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad3a96ee-5575-4a39-8788-9cf602c80f7d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-eac6c5-fd8pl\" (UID: \"ad3a96ee-5575-4a39-8788-9cf602c80f7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-eac6c5-fd8pl"
Sep 19 04:48:06.814895 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:06.814854    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad3a96ee-5575-4a39-8788-9cf602c80f7d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-eac6c5-fd8pl\" (UID: \"ad3a96ee-5575-4a39-8788-9cf602c80f7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-eac6c5-fd8pl"
Sep 19 04:48:06.815477 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:06.815459    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad3a96ee-5575-4a39-8788-9cf602c80f7d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-eac6c5-fd8pl\" (UID: \"ad3a96ee-5575-4a39-8788-9cf602c80f7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-eac6c5-fd8pl"
Sep 19 04:48:06.917347 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:06.917310    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-eac6c5-fd8pl"
Sep 19 04:48:07.204479 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:07.204401    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-eac6c5-fd8pl"]
Sep 19 04:48:07.207089 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:48:07.207062    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad3a96ee_5575_4a39_8788_9cf602c80f7d.slice/crio-152b16ec4dda2d1585c6040a0f29afb5cf1036b2e591cd3638c332b2ee8ea583 WatchSource:0}: Error finding container 152b16ec4dda2d1585c6040a0f29afb5cf1036b2e591cd3638c332b2ee8ea583: Status 404 returned error can't find the container with id 152b16ec4dda2d1585c6040a0f29afb5cf1036b2e591cd3638c332b2ee8ea583
Sep 19 04:48:08.134674 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:08.134634    2579 generic.go:356] "Generic (PLEG): container finished" podID="ad3a96ee-5575-4a39-8788-9cf602c80f7d" containerID="2c82ecd5f8dd296bc3c3fa18135fcbeffab68e6185d8db45b2182ee7705cd814" exitCode=0
Sep 19 04:48:08.135157 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:08.134733    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-eac6c5-fd8pl" event={"ID":"ad3a96ee-5575-4a39-8788-9cf602c80f7d","Type":"ContainerDied","Data":"2c82ecd5f8dd296bc3c3fa18135fcbeffab68e6185d8db45b2182ee7705cd814"}
Sep 19 04:48:08.135157 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:08.134783    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-eac6c5-fd8pl" event={"ID":"ad3a96ee-5575-4a39-8788-9cf602c80f7d","Type":"ContainerStarted","Data":"152b16ec4dda2d1585c6040a0f29afb5cf1036b2e591cd3638c332b2ee8ea583"}
Sep 19 04:48:08.146106 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:08.146080    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-87e3e6/distinct-2-w0-87e3e6-kserve-7694d86767-6sz6p" podUID="98309c08-4d67-45bc-9848-ec80ec181224" containerName="main" containerID="cri-o://4afb6217682191470e34b1df907f36996b64787ff7ec2d9a4ec06d64374b440a" gracePeriod=45
Sep 19 04:48:08.148510 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:08.148479    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-87e3e6/distinct-w0-87e3e6-kserve-54c675b78b-vsvns" podUID="91cd381a-3058-4df7-954e-984be634029e" containerName="main" containerID="cri-o://e266d067b1572459b72323e6eea2f6a7aff885511f2239ef9768b5d6cb995c3f" gracePeriod=45
Sep 19 04:48:08.163000 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:08.162971    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-87e3e6/embedding-w0-87e3e6-kserve-64c8966d59-4r28w" podUID="27bea02f-f375-4c7b-a50d-96123a14255f" containerName="main" containerID="cri-o://e6cd84027bd83605d1d60fbb20ab6cbfad06ba4ae36c5dd90eccd71d4fe92ed7" gracePeriod=45
Sep 19 04:48:08.179075 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:08.179048    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-87e3e6/premium-w0-87e3e6-kserve-f48dd7f99-dsx42" podUID="4ca3da4a-b847-47c4-a1cf-32e7f8643cea" containerName="main" containerID="cri-o://d781ac527d13a23ba126249cdc8bc8cb6540d4e528706e7ae8188ccfe9840c65" gracePeriod=45
Sep 19 04:48:08.184740 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:08.184710    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-87e3e6/distinct-w0-87e3e6-kserve-54c675b78b-vsvns" podUID="91cd381a-3058-4df7-954e-984be634029e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": dial tcp 10.134.0.73:8000: connect: connection refused"
Sep 19 04:48:08.455926 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:08.455897    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-87e3e6/distinct-2-w0-87e3e6-kserve-7694d86767-6sz6p"
Sep 19 04:48:08.460518 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:08.460122    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-87e3e6/distinct-w0-87e3e6-kserve-54c675b78b-vsvns"
Sep 19 04:48:08.474674 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:08.474652    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-87e3e6/embedding-w0-87e3e6-kserve-64c8966d59-4r28w"
Sep 19 04:48:08.508327 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:08.508301    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-87e3e6/premium-w0-87e3e6-kserve-f48dd7f99-dsx42"
Sep 19 04:48:08.634126 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:08.634041    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8r822\" (UniqueName: \"kubernetes.io/projected/91cd381a-3058-4df7-954e-984be634029e-kube-api-access-8r822\") pod \"91cd381a-3058-4df7-954e-984be634029e\" (UID: \"91cd381a-3058-4df7-954e-984be634029e\") "
Sep 19 04:48:08.634126 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:08.634077    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91cd381a-3058-4df7-954e-984be634029e-model-cache\") pod \"91cd381a-3058-4df7-954e-984be634029e\" (UID: \"91cd381a-3058-4df7-954e-984be634029e\") "
Sep 19 04:48:08.634126 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:08.634099    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27bea02f-f375-4c7b-a50d-96123a14255f-model-cache\") pod \"27bea02f-f375-4c7b-a50d-96123a14255f\" (UID: \"27bea02f-f375-4c7b-a50d-96123a14255f\") "
Sep 19 04:48:08.634126 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:08.634122    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ds8ng\" (UniqueName: \"kubernetes.io/projected/27bea02f-f375-4c7b-a50d-96123a14255f-kube-api-access-ds8ng\") pod \"27bea02f-f375-4c7b-a50d-96123a14255f\" (UID: \"27bea02f-f375-4c7b-a50d-96123a14255f\") "
Sep 19 04:48:08.634471 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:08.634147    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98309c08-4d67-45bc-9848-ec80ec181224-tmp-dir\") pod \"98309c08-4d67-45bc-9848-ec80ec181224\" (UID: \"98309c08-4d67-45bc-9848-ec80ec181224\") "
Sep 19 04:48:08.634471 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:08.634213    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91cd381a-3058-4df7-954e-984be634029e-tmp-dir\") pod \"91cd381a-3058-4df7-954e-984be634029e\" (UID: \"91cd381a-3058-4df7-954e-984be634029e\") "
Sep 19 04:48:08.634471 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:08.634249    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98309c08-4d67-45bc-9848-ec80ec181224-dshm\") pod \"98309c08-4d67-45bc-9848-ec80ec181224\" (UID: \"98309c08-4d67-45bc-9848-ec80ec181224\") "
Sep 19 04:48:08.634471 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:08.634282    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98309c08-4d67-45bc-9848-ec80ec181224-tls-certs\") pod \"98309c08-4d67-45bc-9848-ec80ec181224\" (UID: \"98309c08-4d67-45bc-9848-ec80ec181224\") "
Sep 19 04:48:08.634471 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:08.634311    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s8f9k\" (UniqueName: \"kubernetes.io/projected/98309c08-4d67-45bc-9848-ec80ec181224-kube-api-access-s8f9k\") pod \"98309c08-4d67-45bc-9848-ec80ec181224\" (UID: \"98309c08-4d67-45bc-9848-ec80ec181224\") "
Sep 19 04:48:08.634471 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:08.634346    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ca3da4a-b847-47c4-a1cf-32e7f8643cea-tmp-dir\") pod \"4ca3da4a-b847-47c4-a1cf-32e7f8643cea\" (UID: \"4ca3da4a-b847-47c4-a1cf-32e7f8643cea\") "
Sep 19 04:48:08.634471 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:08.634343    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91cd381a-3058-4df7-954e-984be634029e-model-cache" (OuterVolumeSpecName: "model-cache") pod "91cd381a-3058-4df7-954e-984be634029e" (UID: "91cd381a-3058-4df7-954e-984be634029e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:48:08.634471 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:08.634363    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27bea02f-f375-4c7b-a50d-96123a14255f-model-cache" (OuterVolumeSpecName: "model-cache") pod "27bea02f-f375-4c7b-a50d-96123a14255f" (UID: "27bea02f-f375-4c7b-a50d-96123a14255f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:48:08.634471 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:08.634386    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98309c08-4d67-45bc-9848-ec80ec181224-model-cache\") pod \"98309c08-4d67-45bc-9848-ec80ec181224\" (UID: \"98309c08-4d67-45bc-9848-ec80ec181224\") "
Sep 19 04:48:08.634471 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:08.634455    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ca3da4a-b847-47c4-a1cf-32e7f8643cea-home\") pod \"4ca3da4a-b847-47c4-a1cf-32e7f8643cea\" (UID: \"4ca3da4a-b847-47c4-a1cf-32e7f8643cea\") "
Sep 19 04:48:08.635013 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:08.634485    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27bea02f-f375-4c7b-a50d-96123a14255f-tls-certs\") pod \"27bea02f-f375-4c7b-a50d-96123a14255f\" (UID: \"27bea02f-f375-4c7b-a50d-96123a14255f\") "
Sep 19 04:48:08.635013 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:08.634477    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98309c08-4d67-45bc-9848-ec80ec181224-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "98309c08-4d67-45bc-9848-ec80ec181224" (UID: "98309c08-4d67-45bc-9848-ec80ec181224"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:48:08.635013 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:08.634509    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91cd381a-3058-4df7-954e-984be634029e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "91cd381a-3058-4df7-954e-984be634029e" (UID: "91cd381a-3058-4df7-954e-984be634029e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:48:08.635013 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:08.634544    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ca3da4a-b847-47c4-a1cf-32e7f8643cea-dshm\") pod \"4ca3da4a-b847-47c4-a1cf-32e7f8643cea\" (UID: \"4ca3da4a-b847-47c4-a1cf-32e7f8643cea\") "
Sep 19 04:48:08.635013 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:08.634575    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98309c08-4d67-45bc-9848-ec80ec181224-home\") pod \"98309c08-4d67-45bc-9848-ec80ec181224\" (UID: \"98309c08-4d67-45bc-9848-ec80ec181224\") "
Sep 19 04:48:08.635013 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:08.634596    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98309c08-4d67-45bc-9848-ec80ec181224-model-cache" (OuterVolumeSpecName: "model-cache") pod "98309c08-4d67-45bc-9848-ec80ec181224" (UID: "98309c08-4d67-45bc-9848-ec80ec181224"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:48:08.635013 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:08.634606    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ca3da4a-b847-47c4-a1cf-32e7f8643cea-tls-certs\") pod \"4ca3da4a-b847-47c4-a1cf-32e7f8643cea\" (UID: \"4ca3da4a-b847-47c4-a1cf-32e7f8643cea\") "
Sep 19 04:48:08.635013 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:08.634631    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91cd381a-3058-4df7-954e-984be634029e-home\") pod \"91cd381a-3058-4df7-954e-984be634029e\" (UID: \"91cd381a-3058-4df7-954e-984be634029e\") "
Sep 19 04:48:08.635013 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:08.634655    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27bea02f-f375-4c7b-a50d-96123a14255f-home\") pod \"27bea02f-f375-4c7b-a50d-96123a14255f\" (UID: \"27bea02f-f375-4c7b-a50d-96123a14255f\") "
Sep 19 04:48:08.635013 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:08.634710    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27bea02f-f375-4c7b-a50d-96123a14255f-dshm\") pod \"27bea02f-f375-4c7b-a50d-96123a14255f\" (UID: \"27bea02f-f375-4c7b-a50d-96123a14255f\") "
Sep 19 04:48:08.635013 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:08.634739    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91cd381a-3058-4df7-954e-984be634029e-tls-certs\") pod \"91cd381a-3058-4df7-954e-984be634029e\" (UID: \"91cd381a-3058-4df7-954e-984be634029e\") "
Sep 19 04:48:08.635013 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:08.634759    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ca3da4a-b847-47c4-a1cf-32e7f8643cea-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4ca3da4a-b847-47c4-a1cf-32e7f8643cea" (UID: "4ca3da4a-b847-47c4-a1cf-32e7f8643cea"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:48:08.635013 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:08.634775    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8586w\" (UniqueName: \"kubernetes.io/projected/4ca3da4a-b847-47c4-a1cf-32e7f8643cea-kube-api-access-8586w\") pod \"4ca3da4a-b847-47c4-a1cf-32e7f8643cea\" (UID: \"4ca3da4a-b847-47c4-a1cf-32e7f8643cea\") "
Sep 19 04:48:08.635013 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:08.634806    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ca3da4a-b847-47c4-a1cf-32e7f8643cea-model-cache\") pod \"4ca3da4a-b847-47c4-a1cf-32e7f8643cea\" (UID: \"4ca3da4a-b847-47c4-a1cf-32e7f8643cea\") "
Sep 19 04:48:08.635013 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:08.634855    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91cd381a-3058-4df7-954e-984be634029e-dshm\") pod \"91cd381a-3058-4df7-954e-984be634029e\" (UID: \"91cd381a-3058-4df7-954e-984be634029e\") "
Sep 19 04:48:08.635013 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:08.634894    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27bea02f-f375-4c7b-a50d-96123a14255f-tmp-dir\") pod \"27bea02f-f375-4c7b-a50d-96123a14255f\" (UID: \"27bea02f-f375-4c7b-a50d-96123a14255f\") "
Sep 19 04:48:08.635013 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:08.634927    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91cd381a-3058-4df7-954e-984be634029e-home" (OuterVolumeSpecName: "home") pod "91cd381a-3058-4df7-954e-984be634029e" (UID: "91cd381a-3058-4df7-954e-984be634029e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:48:08.635861 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:08.635281    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91cd381a-3058-4df7-954e-984be634029e-model-cache\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:48:08.635861 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:08.635299    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27bea02f-f375-4c7b-a50d-96123a14255f-model-cache\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:48:08.635861 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:08.635313    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98309c08-4d67-45bc-9848-ec80ec181224-tmp-dir\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:48:08.635861 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:08.635325    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91cd381a-3058-4df7-954e-984be634029e-tmp-dir\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:48:08.635861 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:08.635337    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ca3da4a-b847-47c4-a1cf-32e7f8643cea-tmp-dir\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:48:08.635861 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:08.635348    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98309c08-4d67-45bc-9848-ec80ec181224-model-cache\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:48:08.635861 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:08.635359    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91cd381a-3058-4df7-954e-984be634029e-home\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:48:08.635861 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:08.635568    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27bea02f-f375-4c7b-a50d-96123a14255f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "27bea02f-f375-4c7b-a50d-96123a14255f" (UID: "27bea02f-f375-4c7b-a50d-96123a14255f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:48:08.635861 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:08.635808    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27bea02f-f375-4c7b-a50d-96123a14255f-home" (OuterVolumeSpecName: "home") pod "27bea02f-f375-4c7b-a50d-96123a14255f" (UID: "27bea02f-f375-4c7b-a50d-96123a14255f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:48:08.638077 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:08.637328    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98309c08-4d67-45bc-9848-ec80ec181224-kube-api-access-s8f9k" (OuterVolumeSpecName: "kube-api-access-s8f9k") pod "98309c08-4d67-45bc-9848-ec80ec181224" (UID: "98309c08-4d67-45bc-9848-ec80ec181224"). InnerVolumeSpecName "kube-api-access-s8f9k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 04:48:08.638077 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:08.637559    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ca3da4a-b847-47c4-a1cf-32e7f8643cea-home" (OuterVolumeSpecName: "home") pod "4ca3da4a-b847-47c4-a1cf-32e7f8643cea" (UID: "4ca3da4a-b847-47c4-a1cf-32e7f8643cea"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:48:08.638077 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:08.637609    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98309c08-4d67-45bc-9848-ec80ec181224-dshm" (OuterVolumeSpecName: "dshm") pod "98309c08-4d67-45bc-9848-ec80ec181224" (UID: "98309c08-4d67-45bc-9848-ec80ec181224"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:48:08.638077 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:08.637738    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91cd381a-3058-4df7-954e-984be634029e-kube-api-access-8r822" (OuterVolumeSpecName: "kube-api-access-8r822") pod "91cd381a-3058-4df7-954e-984be634029e" (UID: "91cd381a-3058-4df7-954e-984be634029e"). InnerVolumeSpecName "kube-api-access-8r822". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 04:48:08.638077 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:08.637769    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98309c08-4d67-45bc-9848-ec80ec181224-home" (OuterVolumeSpecName: "home") pod "98309c08-4d67-45bc-9848-ec80ec181224" (UID: "98309c08-4d67-45bc-9848-ec80ec181224"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:48:08.638077 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:08.637812    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27bea02f-f375-4c7b-a50d-96123a14255f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "27bea02f-f375-4c7b-a50d-96123a14255f" (UID: "27bea02f-f375-4c7b-a50d-96123a14255f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 04:48:08.638077 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:08.637819    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ca3da4a-b847-47c4-a1cf-32e7f8643cea-dshm" (OuterVolumeSpecName: "dshm") pod "4ca3da4a-b847-47c4-a1cf-32e7f8643cea" (UID: "4ca3da4a-b847-47c4-a1cf-32e7f8643cea"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:48:08.638077 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:08.638018    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27bea02f-f375-4c7b-a50d-96123a14255f-kube-api-access-ds8ng" (OuterVolumeSpecName: "kube-api-access-ds8ng") pod "27bea02f-f375-4c7b-a50d-96123a14255f" (UID: "27bea02f-f375-4c7b-a50d-96123a14255f"). InnerVolumeSpecName "kube-api-access-ds8ng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 04:48:08.638077 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:08.638078    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ca3da4a-b847-47c4-a1cf-32e7f8643cea-model-cache" (OuterVolumeSpecName: "model-cache") pod "4ca3da4a-b847-47c4-a1cf-32e7f8643cea" (UID: "4ca3da4a-b847-47c4-a1cf-32e7f8643cea"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:48:08.638856 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:08.638751    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27bea02f-f375-4c7b-a50d-96123a14255f-dshm" (OuterVolumeSpecName: "dshm") pod "27bea02f-f375-4c7b-a50d-96123a14255f" (UID: "27bea02f-f375-4c7b-a50d-96123a14255f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:48:08.638856 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:08.638771    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98309c08-4d67-45bc-9848-ec80ec181224-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "98309c08-4d67-45bc-9848-ec80ec181224" (UID: "98309c08-4d67-45bc-9848-ec80ec181224"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 04:48:08.639923 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:08.639901    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91cd381a-3058-4df7-954e-984be634029e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "91cd381a-3058-4df7-954e-984be634029e" (UID: "91cd381a-3058-4df7-954e-984be634029e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 04:48:08.640035 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:08.639931    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ca3da4a-b847-47c4-a1cf-32e7f8643cea-kube-api-access-8586w" (OuterVolumeSpecName: "kube-api-access-8586w") pod "4ca3da4a-b847-47c4-a1cf-32e7f8643cea" (UID: "4ca3da4a-b847-47c4-a1cf-32e7f8643cea"). InnerVolumeSpecName "kube-api-access-8586w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 04:48:08.640370 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:08.640353    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91cd381a-3058-4df7-954e-984be634029e-dshm" (OuterVolumeSpecName: "dshm") pod "91cd381a-3058-4df7-954e-984be634029e" (UID: "91cd381a-3058-4df7-954e-984be634029e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:48:08.640370 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:08.640363    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ca3da4a-b847-47c4-a1cf-32e7f8643cea-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4ca3da4a-b847-47c4-a1cf-32e7f8643cea" (UID: "4ca3da4a-b847-47c4-a1cf-32e7f8643cea"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 04:48:08.736267 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:08.736230    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8r822\" (UniqueName: \"kubernetes.io/projected/91cd381a-3058-4df7-954e-984be634029e-kube-api-access-8r822\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:48:08.736267 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:08.736259    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ds8ng\" (UniqueName: \"kubernetes.io/projected/27bea02f-f375-4c7b-a50d-96123a14255f-kube-api-access-ds8ng\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:48:08.736267 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:08.736269    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98309c08-4d67-45bc-9848-ec80ec181224-dshm\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:48:08.736501 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:08.736279    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98309c08-4d67-45bc-9848-ec80ec181224-tls-certs\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:48:08.736501 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:08.736289    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8f9k\" (UniqueName: \"kubernetes.io/projected/98309c08-4d67-45bc-9848-ec80ec181224-kube-api-access-s8f9k\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:48:08.736501 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:08.736296    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ca3da4a-b847-47c4-a1cf-32e7f8643cea-home\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:48:08.736501 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:08.736304    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27bea02f-f375-4c7b-a50d-96123a14255f-tls-certs\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:48:08.736501 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:08.736311    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ca3da4a-b847-47c4-a1cf-32e7f8643cea-dshm\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:48:08.736501 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:08.736317    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98309c08-4d67-45bc-9848-ec80ec181224-home\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:48:08.736501 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:08.736324    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ca3da4a-b847-47c4-a1cf-32e7f8643cea-tls-certs\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:48:08.736501 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:08.736331    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27bea02f-f375-4c7b-a50d-96123a14255f-home\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:48:08.736501 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:08.736337    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27bea02f-f375-4c7b-a50d-96123a14255f-dshm\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:48:08.736501 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:08.736351    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91cd381a-3058-4df7-954e-984be634029e-tls-certs\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:48:08.736501 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:08.736360    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8586w\" (UniqueName: \"kubernetes.io/projected/4ca3da4a-b847-47c4-a1cf-32e7f8643cea-kube-api-access-8586w\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:48:08.736501 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:08.736368    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ca3da4a-b847-47c4-a1cf-32e7f8643cea-model-cache\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:48:08.736501 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:08.736375    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91cd381a-3058-4df7-954e-984be634029e-dshm\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:48:08.736501 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:08.736382    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27bea02f-f375-4c7b-a50d-96123a14255f-tmp-dir\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:48:09.142389 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:09.142348    2579 generic.go:356] "Generic (PLEG): container finished" podID="27bea02f-f375-4c7b-a50d-96123a14255f" containerID="e6cd84027bd83605d1d60fbb20ab6cbfad06ba4ae36c5dd90eccd71d4fe92ed7" exitCode=0
Sep 19 04:48:09.142838 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:09.142408    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-87e3e6/embedding-w0-87e3e6-kserve-64c8966d59-4r28w"
Sep 19 04:48:09.142838 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:09.142435    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-87e3e6/embedding-w0-87e3e6-kserve-64c8966d59-4r28w" event={"ID":"27bea02f-f375-4c7b-a50d-96123a14255f","Type":"ContainerDied","Data":"e6cd84027bd83605d1d60fbb20ab6cbfad06ba4ae36c5dd90eccd71d4fe92ed7"}
Sep 19 04:48:09.142838 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:09.142487    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-87e3e6/embedding-w0-87e3e6-kserve-64c8966d59-4r28w" event={"ID":"27bea02f-f375-4c7b-a50d-96123a14255f","Type":"ContainerDied","Data":"382eb3f85b68db24b9ef3f574e631d826a0db1e09d15f5b765cd49c661bbcf18"}
Sep 19 04:48:09.142838 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:09.142504    2579 scope.go:117] "RemoveContainer" containerID="e6cd84027bd83605d1d60fbb20ab6cbfad06ba4ae36c5dd90eccd71d4fe92ed7"
Sep 19 04:48:09.144176 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:09.144153    2579 generic.go:356] "Generic (PLEG): container finished" podID="91cd381a-3058-4df7-954e-984be634029e" containerID="e266d067b1572459b72323e6eea2f6a7aff885511f2239ef9768b5d6cb995c3f" exitCode=0
Sep 19 04:48:09.144297 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:09.144218    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-87e3e6/distinct-w0-87e3e6-kserve-54c675b78b-vsvns"
Sep 19 04:48:09.144297 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:09.144252    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-87e3e6/distinct-w0-87e3e6-kserve-54c675b78b-vsvns" event={"ID":"91cd381a-3058-4df7-954e-984be634029e","Type":"ContainerDied","Data":"e266d067b1572459b72323e6eea2f6a7aff885511f2239ef9768b5d6cb995c3f"}
Sep 19 04:48:09.144470 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:09.144327    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-87e3e6/distinct-w0-87e3e6-kserve-54c675b78b-vsvns" event={"ID":"91cd381a-3058-4df7-954e-984be634029e","Type":"ContainerDied","Data":"cd73e1cfd97471f1db15a4acb93d579a7a99ef164f8cbf2e1b092e6a373e3adb"}
Sep 19 04:48:09.145923 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:09.145898    2579 generic.go:356] "Generic (PLEG): container finished" podID="4ca3da4a-b847-47c4-a1cf-32e7f8643cea" containerID="d781ac527d13a23ba126249cdc8bc8cb6540d4e528706e7ae8188ccfe9840c65" exitCode=0
Sep 19 04:48:09.146037 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:09.145956    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-87e3e6/premium-w0-87e3e6-kserve-f48dd7f99-dsx42"
Sep 19 04:48:09.146037 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:09.146009    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-87e3e6/premium-w0-87e3e6-kserve-f48dd7f99-dsx42" event={"ID":"4ca3da4a-b847-47c4-a1cf-32e7f8643cea","Type":"ContainerDied","Data":"d781ac527d13a23ba126249cdc8bc8cb6540d4e528706e7ae8188ccfe9840c65"}
Sep 19 04:48:09.146169 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:09.146046    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-87e3e6/premium-w0-87e3e6-kserve-f48dd7f99-dsx42" event={"ID":"4ca3da4a-b847-47c4-a1cf-32e7f8643cea","Type":"ContainerDied","Data":"c2e152812ac45758991e496ca431075a1675f6bf9931eb83edfb1579b76069ca"}
Sep 19 04:48:09.147647 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:09.147623    2579 generic.go:356] "Generic (PLEG): container finished" podID="98309c08-4d67-45bc-9848-ec80ec181224" containerID="4afb6217682191470e34b1df907f36996b64787ff7ec2d9a4ec06d64374b440a" exitCode=0
Sep 19 04:48:09.147805 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:09.147676    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-87e3e6/distinct-2-w0-87e3e6-kserve-7694d86767-6sz6p"
Sep 19 04:48:09.147805 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:09.147728    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-87e3e6/distinct-2-w0-87e3e6-kserve-7694d86767-6sz6p" event={"ID":"98309c08-4d67-45bc-9848-ec80ec181224","Type":"ContainerDied","Data":"4afb6217682191470e34b1df907f36996b64787ff7ec2d9a4ec06d64374b440a"}
Sep 19 04:48:09.147805 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:09.147761    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-87e3e6/distinct-2-w0-87e3e6-kserve-7694d86767-6sz6p" event={"ID":"98309c08-4d67-45bc-9848-ec80ec181224","Type":"ContainerDied","Data":"8ad8a9517274808d206933ef5fec1fd8dc0ccda51a7f3454285bf5e82a95ac59"}
Sep 19 04:48:09.155377 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:09.155359    2579 scope.go:117] "RemoveContainer" containerID="e6cd84027bd83605d1d60fbb20ab6cbfad06ba4ae36c5dd90eccd71d4fe92ed7"
Sep 19 04:48:09.155824 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:48:09.155803    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6cd84027bd83605d1d60fbb20ab6cbfad06ba4ae36c5dd90eccd71d4fe92ed7\": container with ID starting with e6cd84027bd83605d1d60fbb20ab6cbfad06ba4ae36c5dd90eccd71d4fe92ed7 not found: ID does not exist" containerID="e6cd84027bd83605d1d60fbb20ab6cbfad06ba4ae36c5dd90eccd71d4fe92ed7"
Sep 19 04:48:09.155904 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:09.155832    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6cd84027bd83605d1d60fbb20ab6cbfad06ba4ae36c5dd90eccd71d4fe92ed7"} err="failed to get container status \"e6cd84027bd83605d1d60fbb20ab6cbfad06ba4ae36c5dd90eccd71d4fe92ed7\": rpc error: code = NotFound desc = could not find container \"e6cd84027bd83605d1d60fbb20ab6cbfad06ba4ae36c5dd90eccd71d4fe92ed7\": container with ID starting with e6cd84027bd83605d1d60fbb20ab6cbfad06ba4ae36c5dd90eccd71d4fe92ed7 not found: ID does not exist"
Sep 19 04:48:09.155904 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:09.155849    2579 scope.go:117] "RemoveContainer" containerID="e266d067b1572459b72323e6eea2f6a7aff885511f2239ef9768b5d6cb995c3f"
Sep 19 04:48:09.173506 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:09.173479    2579 scope.go:117] "RemoveContainer" containerID="e266d067b1572459b72323e6eea2f6a7aff885511f2239ef9768b5d6cb995c3f"
Sep 19 04:48:09.174581 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:48:09.174542    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e266d067b1572459b72323e6eea2f6a7aff885511f2239ef9768b5d6cb995c3f\": container with ID starting with e266d067b1572459b72323e6eea2f6a7aff885511f2239ef9768b5d6cb995c3f not found: ID does not exist" containerID="e266d067b1572459b72323e6eea2f6a7aff885511f2239ef9768b5d6cb995c3f"
Sep 19 04:48:09.174744 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:09.174578    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e266d067b1572459b72323e6eea2f6a7aff885511f2239ef9768b5d6cb995c3f"} err="failed to get container status \"e266d067b1572459b72323e6eea2f6a7aff885511f2239ef9768b5d6cb995c3f\": rpc error: code = NotFound desc = could not find container \"e266d067b1572459b72323e6eea2f6a7aff885511f2239ef9768b5d6cb995c3f\": container with ID starting with e266d067b1572459b72323e6eea2f6a7aff885511f2239ef9768b5d6cb995c3f not found: ID does not exist"
Sep 19 04:48:09.174744 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:09.174599    2579 scope.go:117] "RemoveContainer" containerID="d781ac527d13a23ba126249cdc8bc8cb6540d4e528706e7ae8188ccfe9840c65"
Sep 19 04:48:09.175681 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:09.175655    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-87e3e6/embedding-w0-87e3e6-kserve-64c8966d59-4r28w"]
Sep 19 04:48:09.185622 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:09.185602    2579 scope.go:117] "RemoveContainer" containerID="d781ac527d13a23ba126249cdc8bc8cb6540d4e528706e7ae8188ccfe9840c65"
Sep 19 04:48:09.187389 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:48:09.186052    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d781ac527d13a23ba126249cdc8bc8cb6540d4e528706e7ae8188ccfe9840c65\": container with ID starting with d781ac527d13a23ba126249cdc8bc8cb6540d4e528706e7ae8188ccfe9840c65 not found: ID does not exist" containerID="d781ac527d13a23ba126249cdc8bc8cb6540d4e528706e7ae8188ccfe9840c65"
Sep 19 04:48:09.187389 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:09.186107    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d781ac527d13a23ba126249cdc8bc8cb6540d4e528706e7ae8188ccfe9840c65"} err="failed to get container status \"d781ac527d13a23ba126249cdc8bc8cb6540d4e528706e7ae8188ccfe9840c65\": rpc error: code = NotFound desc = could not find container \"d781ac527d13a23ba126249cdc8bc8cb6540d4e528706e7ae8188ccfe9840c65\": container with ID starting with d781ac527d13a23ba126249cdc8bc8cb6540d4e528706e7ae8188ccfe9840c65 not found: ID does not exist"
Sep 19 04:48:09.187389 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:09.186132    2579 scope.go:117] "RemoveContainer" containerID="4afb6217682191470e34b1df907f36996b64787ff7ec2d9a4ec06d64374b440a"
Sep 19 04:48:09.187389 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:09.186525    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-87e3e6/embedding-w0-87e3e6-kserve-64c8966d59-4r28w"]
Sep 19 04:48:09.198287 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:09.198265    2579 scope.go:117] "RemoveContainer" containerID="4afb6217682191470e34b1df907f36996b64787ff7ec2d9a4ec06d64374b440a"
Sep 19 04:48:09.198674 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:48:09.198624    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4afb6217682191470e34b1df907f36996b64787ff7ec2d9a4ec06d64374b440a\": container with ID starting with 4afb6217682191470e34b1df907f36996b64787ff7ec2d9a4ec06d64374b440a not found: ID does not exist" containerID="4afb6217682191470e34b1df907f36996b64787ff7ec2d9a4ec06d64374b440a"
Sep 19 04:48:09.198809 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:09.198682    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4afb6217682191470e34b1df907f36996b64787ff7ec2d9a4ec06d64374b440a"} err="failed to get container status \"4afb6217682191470e34b1df907f36996b64787ff7ec2d9a4ec06d64374b440a\": rpc error: code = NotFound desc = could not find container \"4afb6217682191470e34b1df907f36996b64787ff7ec2d9a4ec06d64374b440a\": container with ID starting with 4afb6217682191470e34b1df907f36996b64787ff7ec2d9a4ec06d64374b440a not found: ID does not exist"
Sep 19 04:48:09.203411 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:09.203385    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-87e3e6/distinct-w0-87e3e6-kserve-54c675b78b-vsvns"]
Sep 19 04:48:09.209520 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:09.209498    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-87e3e6/distinct-w0-87e3e6-kserve-54c675b78b-vsvns"]
Sep 19 04:48:09.225566 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:09.225541    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-87e3e6/premium-w0-87e3e6-kserve-f48dd7f99-dsx42"]
Sep 19 04:48:09.229273 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:09.229250    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-87e3e6/premium-w0-87e3e6-kserve-f48dd7f99-dsx42"]
Sep 19 04:48:09.245724 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:09.245644    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-87e3e6/distinct-2-w0-87e3e6-kserve-7694d86767-6sz6p"]
Sep 19 04:48:09.252592 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:09.252569    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-87e3e6/distinct-2-w0-87e3e6-kserve-7694d86767-6sz6p"]
Sep 19 04:48:09.302855 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:09.302829    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-eac6c5-fd8pl"
Sep 19 04:48:09.351972 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:09.351943    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27bea02f-f375-4c7b-a50d-96123a14255f" path="/var/lib/kubelet/pods/27bea02f-f375-4c7b-a50d-96123a14255f/volumes"
Sep 19 04:48:09.352369 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:09.352335    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ca3da4a-b847-47c4-a1cf-32e7f8643cea" path="/var/lib/kubelet/pods/4ca3da4a-b847-47c4-a1cf-32e7f8643cea/volumes"
Sep 19 04:48:09.352815 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:09.352798    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91cd381a-3058-4df7-954e-984be634029e" path="/var/lib/kubelet/pods/91cd381a-3058-4df7-954e-984be634029e/volumes"
Sep 19 04:48:09.353472 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:09.353457    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98309c08-4d67-45bc-9848-ec80ec181224" path="/var/lib/kubelet/pods/98309c08-4d67-45bc-9848-ec80ec181224/volumes"
Sep 19 04:48:09.442513 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:09.442484    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad3a96ee-5575-4a39-8788-9cf602c80f7d-maas-api-service-ca\") pod \"ad3a96ee-5575-4a39-8788-9cf602c80f7d\" (UID: \"ad3a96ee-5575-4a39-8788-9cf602c80f7d\") "
Sep 19 04:48:09.442903 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:09.442872    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad3a96ee-5575-4a39-8788-9cf602c80f7d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ad3a96ee-5575-4a39-8788-9cf602c80f7d" (UID: "ad3a96ee-5575-4a39-8788-9cf602c80f7d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 04:48:09.443301 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:09.443149    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad3a96ee-5575-4a39-8788-9cf602c80f7d-maas-api-service-ca\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:48:09.465375 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:09.465343    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-87e3e6/embedding-w0-87e3e6-kserve-64c8966d59-4r28w" podUID="27bea02f-f375-4c7b-a50d-96123a14255f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded"
Sep 19 04:48:10.159470 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:10.159438    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-eac6c5-fd8pl"
Sep 19 04:48:10.159470 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:10.159467    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-eac6c5-fd8pl" event={"ID":"ad3a96ee-5575-4a39-8788-9cf602c80f7d","Type":"ContainerDied","Data":"152b16ec4dda2d1585c6040a0f29afb5cf1036b2e591cd3638c332b2ee8ea583"}
Sep 19 04:48:10.160012 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:10.159510    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="152b16ec4dda2d1585c6040a0f29afb5cf1036b2e591cd3638c332b2ee8ea583"
Sep 19 04:48:10.996715 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:10.996663    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-87e3e6-openshift-default-8989bfbdd-fwgnd"]
Sep 19 04:48:10.996924 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:10.996882    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-87e3e6-openshift-default-8989bfbdd-fwgnd" podUID="39081cb6-0611-41e5-b50e-bfe7759ef529" containerName="istio-proxy" containerID="cri-o://7767b9aaefb207bc507d2ed7cf4cf100de783b3e1c49dc120633ec33b925c219" gracePeriod=30
Sep 19 04:48:16.202302 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:16.202268    2579 generic.go:356] "Generic (PLEG): container finished" podID="39081cb6-0611-41e5-b50e-bfe7759ef529" containerID="7767b9aaefb207bc507d2ed7cf4cf100de783b3e1c49dc120633ec33b925c219" exitCode=0
Sep 19 04:48:16.202785 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:16.202335    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-87e3e6-openshift-default-8989bfbdd-fwgnd" event={"ID":"39081cb6-0611-41e5-b50e-bfe7759ef529","Type":"ContainerDied","Data":"7767b9aaefb207bc507d2ed7cf4cf100de783b3e1c49dc120633ec33b925c219"}
Sep 19 04:48:16.202785 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:16.202406    2579 scope.go:117] "RemoveContainer" containerID="40df442f47aa4c4fbcbd07b89753e89ee38a79d4b0ce19eed2f6d50264f2d0cc"
Sep 19 04:48:16.263769 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:16.263743    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-87e3e6-openshift-default-8989bfbdd-fwgnd"
Sep 19 04:48:16.305879 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:16.305799    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-44rjb\" (UniqueName: \"kubernetes.io/projected/39081cb6-0611-41e5-b50e-bfe7759ef529-kube-api-access-44rjb\") pod \"39081cb6-0611-41e5-b50e-bfe7759ef529\" (UID: \"39081cb6-0611-41e5-b50e-bfe7759ef529\") "
Sep 19 04:48:16.305879 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:16.305857    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39081cb6-0611-41e5-b50e-bfe7759ef529-istio-token\") pod \"39081cb6-0611-41e5-b50e-bfe7759ef529\" (UID: \"39081cb6-0611-41e5-b50e-bfe7759ef529\") "
Sep 19 04:48:16.306141 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:16.305898    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39081cb6-0611-41e5-b50e-bfe7759ef529-credential-socket\") pod \"39081cb6-0611-41e5-b50e-bfe7759ef529\" (UID: \"39081cb6-0611-41e5-b50e-bfe7759ef529\") "
Sep 19 04:48:16.306141 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:16.305925    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39081cb6-0611-41e5-b50e-bfe7759ef529-istio-envoy\") pod \"39081cb6-0611-41e5-b50e-bfe7759ef529\" (UID: \"39081cb6-0611-41e5-b50e-bfe7759ef529\") "
Sep 19 04:48:16.306141 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:16.305961    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39081cb6-0611-41e5-b50e-bfe7759ef529-istio-podinfo\") pod \"39081cb6-0611-41e5-b50e-bfe7759ef529\" (UID: \"39081cb6-0611-41e5-b50e-bfe7759ef529\") "
Sep 19 04:48:16.306141 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:16.306010    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39081cb6-0611-41e5-b50e-bfe7759ef529-workload-certs\") pod \"39081cb6-0611-41e5-b50e-bfe7759ef529\" (UID: \"39081cb6-0611-41e5-b50e-bfe7759ef529\") "
Sep 19 04:48:16.306141 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:16.306036    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39081cb6-0611-41e5-b50e-bfe7759ef529-workload-socket\") pod \"39081cb6-0611-41e5-b50e-bfe7759ef529\" (UID: \"39081cb6-0611-41e5-b50e-bfe7759ef529\") "
Sep 19 04:48:16.306141 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:16.306062    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39081cb6-0611-41e5-b50e-bfe7759ef529-istio-data\") pod \"39081cb6-0611-41e5-b50e-bfe7759ef529\" (UID: \"39081cb6-0611-41e5-b50e-bfe7759ef529\") "
Sep 19 04:48:16.306141 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:16.306107    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39081cb6-0611-41e5-b50e-bfe7759ef529-istiod-ca-cert\") pod \"39081cb6-0611-41e5-b50e-bfe7759ef529\" (UID: \"39081cb6-0611-41e5-b50e-bfe7759ef529\") "
Sep 19 04:48:16.306464 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:16.306181    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39081cb6-0611-41e5-b50e-bfe7759ef529-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "39081cb6-0611-41e5-b50e-bfe7759ef529" (UID: "39081cb6-0611-41e5-b50e-bfe7759ef529"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:48:16.306464 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:16.306351    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39081cb6-0611-41e5-b50e-bfe7759ef529-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "39081cb6-0611-41e5-b50e-bfe7759ef529" (UID: "39081cb6-0611-41e5-b50e-bfe7759ef529"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:48:16.306628 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:16.306363    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39081cb6-0611-41e5-b50e-bfe7759ef529-istio-data" (OuterVolumeSpecName: "istio-data") pod "39081cb6-0611-41e5-b50e-bfe7759ef529" (UID: "39081cb6-0611-41e5-b50e-bfe7759ef529"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:48:16.306628 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:16.306405    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39081cb6-0611-41e5-b50e-bfe7759ef529-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "39081cb6-0611-41e5-b50e-bfe7759ef529" (UID: "39081cb6-0611-41e5-b50e-bfe7759ef529"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:48:16.306773 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:16.306464    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39081cb6-0611-41e5-b50e-bfe7759ef529-credential-socket\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:48:16.306830 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:16.306781    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39081cb6-0611-41e5-b50e-bfe7759ef529-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "39081cb6-0611-41e5-b50e-bfe7759ef529" (UID: "39081cb6-0611-41e5-b50e-bfe7759ef529"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 04:48:16.308676 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:16.308643    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/39081cb6-0611-41e5-b50e-bfe7759ef529-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "39081cb6-0611-41e5-b50e-bfe7759ef529" (UID: "39081cb6-0611-41e5-b50e-bfe7759ef529"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 19 04:48:16.308940 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:16.308913    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39081cb6-0611-41e5-b50e-bfe7759ef529-istio-token" (OuterVolumeSpecName: "istio-token") pod "39081cb6-0611-41e5-b50e-bfe7759ef529" (UID: "39081cb6-0611-41e5-b50e-bfe7759ef529"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 04:48:16.309167 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:16.309144    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39081cb6-0611-41e5-b50e-bfe7759ef529-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "39081cb6-0611-41e5-b50e-bfe7759ef529" (UID: "39081cb6-0611-41e5-b50e-bfe7759ef529"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:48:16.309262 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:16.309226    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39081cb6-0611-41e5-b50e-bfe7759ef529-kube-api-access-44rjb" (OuterVolumeSpecName: "kube-api-access-44rjb") pod "39081cb6-0611-41e5-b50e-bfe7759ef529" (UID: "39081cb6-0611-41e5-b50e-bfe7759ef529"). InnerVolumeSpecName "kube-api-access-44rjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 04:48:16.407172 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:16.407133    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44rjb\" (UniqueName: \"kubernetes.io/projected/39081cb6-0611-41e5-b50e-bfe7759ef529-kube-api-access-44rjb\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:48:16.407172 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:16.407167    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39081cb6-0611-41e5-b50e-bfe7759ef529-istio-token\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:48:16.407172 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:16.407176    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39081cb6-0611-41e5-b50e-bfe7759ef529-istio-envoy\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:48:16.407398 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:16.407184    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39081cb6-0611-41e5-b50e-bfe7759ef529-istio-podinfo\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:48:16.407398 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:16.407192    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39081cb6-0611-41e5-b50e-bfe7759ef529-workload-certs\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:48:16.407398 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:16.407202    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39081cb6-0611-41e5-b50e-bfe7759ef529-workload-socket\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:48:16.407398 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:16.407209    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39081cb6-0611-41e5-b50e-bfe7759ef529-istio-data\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:48:16.407398 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:16.407217    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39081cb6-0611-41e5-b50e-bfe7759ef529-istiod-ca-cert\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:48:17.210970 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:17.210935    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-87e3e6-openshift-default-8989bfbdd-fwgnd" event={"ID":"39081cb6-0611-41e5-b50e-bfe7759ef529","Type":"ContainerDied","Data":"bfcf9ee85a7595e5fd6c96f103ba105618ffcd540435bca411ac82b62d02c43a"}
Sep 19 04:48:17.210970 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:17.210963    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-87e3e6-openshift-default-8989bfbdd-fwgnd"
Sep 19 04:48:17.211458 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:17.210994    2579 scope.go:117] "RemoveContainer" containerID="7767b9aaefb207bc507d2ed7cf4cf100de783b3e1c49dc120633ec33b925c219"
Sep 19 04:48:17.235907 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:17.235880    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-87e3e6-openshift-default-8989bfbdd-fwgnd"]
Sep 19 04:48:17.241801 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:17.241776    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-87e3e6-openshift-default-8989bfbdd-fwgnd"]
Sep 19 04:48:17.351888 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:17.351853    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39081cb6-0611-41e5-b50e-bfe7759ef529" path="/var/lib/kubelet/pods/39081cb6-0611-41e5-b50e-bfe7759ef529/volumes"
Sep 19 04:48:17.450864 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:17.450831    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-eac6c5-29829885-pxvwr"]
Sep 19 04:48:17.457905 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:17.457874    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-eac6c5-29829885-pxvwr"]
Sep 19 04:48:17.678423 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:17.678385    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-eac6c5-fd8pl"]
Sep 19 04:48:17.680676 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:17.680645    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-eac6c5-fd8pl"]
Sep 19 04:48:19.351763 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:19.351731    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8dac9dfd-2820-46d9-b6ed-d0d691d033fe" path="/var/lib/kubelet/pods/8dac9dfd-2820-46d9-b6ed-d0d691d033fe/volumes"
Sep 19 04:48:19.352117 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:19.352036    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad3a96ee-5575-4a39-8788-9cf602c80f7d" path="/var/lib/kubelet/pods/ad3a96ee-5575-4a39-8788-9cf602c80f7d/volumes"
Sep 19 04:48:24.790305 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:24.790257    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-eac6c5-openshift-default-5bc95d8f8b-lc62q"]
Sep 19 04:48:24.790937 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:24.790536    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-eac6c5-openshift-default-5bc95d8f8b-lc62q" podUID="a0c01ced-87c7-4491-aa9c-2eabd3789794" containerName="istio-proxy" containerID="cri-o://7cba260d9bfc04edb10a51cf1c512bb9605110e83a882a0910dde07f414e6860" gracePeriod=30
Sep 19 04:48:30.046468 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:30.046443    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-eac6c5-openshift-default-5bc95d8f8b-lc62q"
Sep 19 04:48:30.127745 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:30.127624    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0c01ced-87c7-4491-aa9c-2eabd3789794-workload-certs\") pod \"a0c01ced-87c7-4491-aa9c-2eabd3789794\" (UID: \"a0c01ced-87c7-4491-aa9c-2eabd3789794\") "
Sep 19 04:48:30.127745 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:30.127667    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0c01ced-87c7-4491-aa9c-2eabd3789794-istiod-ca-cert\") pod \"a0c01ced-87c7-4491-aa9c-2eabd3789794\" (UID: \"a0c01ced-87c7-4491-aa9c-2eabd3789794\") "
Sep 19 04:48:30.127975 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:30.127751    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0c01ced-87c7-4491-aa9c-2eabd3789794-istio-token\") pod \"a0c01ced-87c7-4491-aa9c-2eabd3789794\" (UID: \"a0c01ced-87c7-4491-aa9c-2eabd3789794\") "
Sep 19 04:48:30.127975 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:30.127784    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0c01ced-87c7-4491-aa9c-2eabd3789794-istio-podinfo\") pod \"a0c01ced-87c7-4491-aa9c-2eabd3789794\" (UID: \"a0c01ced-87c7-4491-aa9c-2eabd3789794\") "
Sep 19 04:48:30.127975 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:30.127939    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c6ncd\" (UniqueName: \"kubernetes.io/projected/a0c01ced-87c7-4491-aa9c-2eabd3789794-kube-api-access-c6ncd\") pod \"a0c01ced-87c7-4491-aa9c-2eabd3789794\" (UID: \"a0c01ced-87c7-4491-aa9c-2eabd3789794\") "
Sep 19 04:48:30.127975 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:30.127941    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0c01ced-87c7-4491-aa9c-2eabd3789794-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a0c01ced-87c7-4491-aa9c-2eabd3789794" (UID: "a0c01ced-87c7-4491-aa9c-2eabd3789794"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:48:30.128193 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:30.128032    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0c01ced-87c7-4491-aa9c-2eabd3789794-istio-envoy\") pod \"a0c01ced-87c7-4491-aa9c-2eabd3789794\" (UID: \"a0c01ced-87c7-4491-aa9c-2eabd3789794\") "
Sep 19 04:48:30.128193 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:30.128129    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0c01ced-87c7-4491-aa9c-2eabd3789794-workload-socket\") pod \"a0c01ced-87c7-4491-aa9c-2eabd3789794\" (UID: \"a0c01ced-87c7-4491-aa9c-2eabd3789794\") "
Sep 19 04:48:30.128193 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:30.128168    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0c01ced-87c7-4491-aa9c-2eabd3789794-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a0c01ced-87c7-4491-aa9c-2eabd3789794" (UID: "a0c01ced-87c7-4491-aa9c-2eabd3789794"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 04:48:30.128193 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:30.128181    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0c01ced-87c7-4491-aa9c-2eabd3789794-istio-data\") pod \"a0c01ced-87c7-4491-aa9c-2eabd3789794\" (UID: \"a0c01ced-87c7-4491-aa9c-2eabd3789794\") "
Sep 19 04:48:30.128368 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:30.128212    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0c01ced-87c7-4491-aa9c-2eabd3789794-credential-socket\") pod \"a0c01ced-87c7-4491-aa9c-2eabd3789794\" (UID: \"a0c01ced-87c7-4491-aa9c-2eabd3789794\") "
Sep 19 04:48:30.128426 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:30.128357    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0c01ced-87c7-4491-aa9c-2eabd3789794-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a0c01ced-87c7-4491-aa9c-2eabd3789794" (UID: "a0c01ced-87c7-4491-aa9c-2eabd3789794"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:48:30.128562 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:30.128510    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0c01ced-87c7-4491-aa9c-2eabd3789794-istio-data" (OuterVolumeSpecName: "istio-data") pod "a0c01ced-87c7-4491-aa9c-2eabd3789794" (UID: "a0c01ced-87c7-4491-aa9c-2eabd3789794"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:48:30.128782 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:30.128578    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0c01ced-87c7-4491-aa9c-2eabd3789794-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a0c01ced-87c7-4491-aa9c-2eabd3789794" (UID: "a0c01ced-87c7-4491-aa9c-2eabd3789794"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:48:30.128782 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:30.128636    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0c01ced-87c7-4491-aa9c-2eabd3789794-workload-certs\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:48:30.128782 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:30.128649    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0c01ced-87c7-4491-aa9c-2eabd3789794-istiod-ca-cert\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:48:30.128782 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:30.128657    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0c01ced-87c7-4491-aa9c-2eabd3789794-workload-socket\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:48:30.128782 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:30.128666    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0c01ced-87c7-4491-aa9c-2eabd3789794-istio-data\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:48:30.130229 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:30.130202    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0c01ced-87c7-4491-aa9c-2eabd3789794-istio-token" (OuterVolumeSpecName: "istio-token") pod "a0c01ced-87c7-4491-aa9c-2eabd3789794" (UID: "a0c01ced-87c7-4491-aa9c-2eabd3789794"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 04:48:30.130455 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:30.130430    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0c01ced-87c7-4491-aa9c-2eabd3789794-kube-api-access-c6ncd" (OuterVolumeSpecName: "kube-api-access-c6ncd") pod "a0c01ced-87c7-4491-aa9c-2eabd3789794" (UID: "a0c01ced-87c7-4491-aa9c-2eabd3789794"). InnerVolumeSpecName "kube-api-access-c6ncd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 04:48:30.130731 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:30.130709    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a0c01ced-87c7-4491-aa9c-2eabd3789794-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a0c01ced-87c7-4491-aa9c-2eabd3789794" (UID: "a0c01ced-87c7-4491-aa9c-2eabd3789794"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 19 04:48:30.130835 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:30.130739    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0c01ced-87c7-4491-aa9c-2eabd3789794-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a0c01ced-87c7-4491-aa9c-2eabd3789794" (UID: "a0c01ced-87c7-4491-aa9c-2eabd3789794"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:48:30.229348 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:30.229310    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0c01ced-87c7-4491-aa9c-2eabd3789794-istio-token\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:48:30.229348 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:30.229341    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0c01ced-87c7-4491-aa9c-2eabd3789794-istio-podinfo\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:48:30.229348 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:30.229350    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6ncd\" (UniqueName: \"kubernetes.io/projected/a0c01ced-87c7-4491-aa9c-2eabd3789794-kube-api-access-c6ncd\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:48:30.229348 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:30.229358    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0c01ced-87c7-4491-aa9c-2eabd3789794-istio-envoy\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:48:30.229614 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:30.229366    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0c01ced-87c7-4491-aa9c-2eabd3789794-credential-socket\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:48:30.303792 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:30.303749    2579 generic.go:356] "Generic (PLEG): container finished" podID="a0c01ced-87c7-4491-aa9c-2eabd3789794" containerID="7cba260d9bfc04edb10a51cf1c512bb9605110e83a882a0910dde07f414e6860" exitCode=0
Sep 19 04:48:30.303979 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:30.303806    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-eac6c5-openshift-default-5bc95d8f8b-lc62q"
Sep 19 04:48:30.303979 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:30.303829    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-eac6c5-openshift-default-5bc95d8f8b-lc62q" event={"ID":"a0c01ced-87c7-4491-aa9c-2eabd3789794","Type":"ContainerDied","Data":"7cba260d9bfc04edb10a51cf1c512bb9605110e83a882a0910dde07f414e6860"}
Sep 19 04:48:30.303979 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:30.303901    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-eac6c5-openshift-default-5bc95d8f8b-lc62q" event={"ID":"a0c01ced-87c7-4491-aa9c-2eabd3789794","Type":"ContainerDied","Data":"c5678405880d45e2bf36e5da99c8feac9abb3444b8e9d571e9081ac5ee0bd850"}
Sep 19 04:48:30.303979 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:30.303936    2579 scope.go:117] "RemoveContainer" containerID="7cba260d9bfc04edb10a51cf1c512bb9605110e83a882a0910dde07f414e6860"
Sep 19 04:48:30.314954 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:30.314935    2579 scope.go:117] "RemoveContainer" containerID="7a90f1a74af62738e0bb8312e425d3ae5d4868c1e49b4c76bb2ad12f24097a49"
Sep 19 04:48:30.324637 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:30.324615    2579 scope.go:117] "RemoveContainer" containerID="7cba260d9bfc04edb10a51cf1c512bb9605110e83a882a0910dde07f414e6860"
Sep 19 04:48:30.324967 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:48:30.324950    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cba260d9bfc04edb10a51cf1c512bb9605110e83a882a0910dde07f414e6860\": container with ID starting with 7cba260d9bfc04edb10a51cf1c512bb9605110e83a882a0910dde07f414e6860 not found: ID does not exist" containerID="7cba260d9bfc04edb10a51cf1c512bb9605110e83a882a0910dde07f414e6860"
Sep 19 04:48:30.325027 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:30.324975    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cba260d9bfc04edb10a51cf1c512bb9605110e83a882a0910dde07f414e6860"} err="failed to get container status \"7cba260d9bfc04edb10a51cf1c512bb9605110e83a882a0910dde07f414e6860\": rpc error: code = NotFound desc = could not find container \"7cba260d9bfc04edb10a51cf1c512bb9605110e83a882a0910dde07f414e6860\": container with ID starting with 7cba260d9bfc04edb10a51cf1c512bb9605110e83a882a0910dde07f414e6860 not found: ID does not exist"
Sep 19 04:48:30.325027 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:30.324993    2579 scope.go:117] "RemoveContainer" containerID="7a90f1a74af62738e0bb8312e425d3ae5d4868c1e49b4c76bb2ad12f24097a49"
Sep 19 04:48:30.325287 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:48:30.325272    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a90f1a74af62738e0bb8312e425d3ae5d4868c1e49b4c76bb2ad12f24097a49\": container with ID starting with 7a90f1a74af62738e0bb8312e425d3ae5d4868c1e49b4c76bb2ad12f24097a49 not found: ID does not exist" containerID="7a90f1a74af62738e0bb8312e425d3ae5d4868c1e49b4c76bb2ad12f24097a49"
Sep 19 04:48:30.325359 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:30.325291    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a90f1a74af62738e0bb8312e425d3ae5d4868c1e49b4c76bb2ad12f24097a49"} err="failed to get container status \"7a90f1a74af62738e0bb8312e425d3ae5d4868c1e49b4c76bb2ad12f24097a49\": rpc error: code = NotFound desc = could not find container \"7a90f1a74af62738e0bb8312e425d3ae5d4868c1e49b4c76bb2ad12f24097a49\": container with ID starting with 7a90f1a74af62738e0bb8312e425d3ae5d4868c1e49b4c76bb2ad12f24097a49 not found: ID does not exist"
Sep 19 04:48:30.329293 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:30.329272    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-eac6c5-openshift-default-5bc95d8f8b-lc62q"]
Sep 19 04:48:30.337030 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:30.337007    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-eac6c5-openshift-default-5bc95d8f8b-lc62q"]
Sep 19 04:48:31.352423 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:31.352388    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0c01ced-87c7-4491-aa9c-2eabd3789794" path="/var/lib/kubelet/pods/a0c01ced-87c7-4491-aa9c-2eabd3789794/volumes"
Sep 19 04:48:41.722610 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:41.722575    2579 scope.go:117] "RemoveContainer" containerID="89ffce6f354c6cebba8024128404ade1e2b591a415625a493a411f5ad22d2f22"
Sep 19 04:48:41.731599 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:48:41.731575    2579 scope.go:117] "RemoveContainer" containerID="372c0e635ec2b448bd4cbf591e9e452ab01f6d2792bd46320e424e98f734eff6"
Sep 19 04:49:09.546688 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:09.546654    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-10379-8a4e4b"]
Sep 19 04:49:09.547211 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:09.547072    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91cd381a-3058-4df7-954e-984be634029e" containerName="main"
Sep 19 04:49:09.547211 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:09.547085    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="91cd381a-3058-4df7-954e-984be634029e" containerName="main"
Sep 19 04:49:09.547211 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:09.547095    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0c01ced-87c7-4491-aa9c-2eabd3789794" containerName="istio-proxy"
Sep 19 04:49:09.547211 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:09.547102    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0c01ced-87c7-4491-aa9c-2eabd3789794" containerName="istio-proxy"
Sep 19 04:49:09.547211 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:09.547112    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98309c08-4d67-45bc-9848-ec80ec181224" containerName="main"
Sep 19 04:49:09.547211 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:09.547120    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="98309c08-4d67-45bc-9848-ec80ec181224" containerName="main"
Sep 19 04:49:09.547211 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:09.547132    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39081cb6-0611-41e5-b50e-bfe7759ef529" containerName="istio-proxy"
Sep 19 04:49:09.547211 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:09.547137    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="39081cb6-0611-41e5-b50e-bfe7759ef529" containerName="istio-proxy"
Sep 19 04:49:09.547211 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:09.547142    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39081cb6-0611-41e5-b50e-bfe7759ef529" containerName="istio-proxy"
Sep 19 04:49:09.547211 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:09.547147    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="39081cb6-0611-41e5-b50e-bfe7759ef529" containerName="istio-proxy"
Sep 19 04:49:09.547211 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:09.547154    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ca3da4a-b847-47c4-a1cf-32e7f8643cea" containerName="main"
Sep 19 04:49:09.547211 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:09.547158    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ca3da4a-b847-47c4-a1cf-32e7f8643cea" containerName="main"
Sep 19 04:49:09.547211 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:09.547168    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0c01ced-87c7-4491-aa9c-2eabd3789794" containerName="istio-proxy"
Sep 19 04:49:09.547211 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:09.547172    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0c01ced-87c7-4491-aa9c-2eabd3789794" containerName="istio-proxy"
Sep 19 04:49:09.547211 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:09.547179    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27bea02f-f375-4c7b-a50d-96123a14255f" containerName="main"
Sep 19 04:49:09.547211 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:09.547185    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="27bea02f-f375-4c7b-a50d-96123a14255f" containerName="main"
Sep 19 04:49:09.547211 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:09.547195    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad3a96ee-5575-4a39-8788-9cf602c80f7d" containerName="revoke-keys"
Sep 19 04:49:09.547211 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:09.547200    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad3a96ee-5575-4a39-8788-9cf602c80f7d" containerName="revoke-keys"
Sep 19 04:49:09.547821 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:09.547253    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad3a96ee-5575-4a39-8788-9cf602c80f7d" containerName="revoke-keys"
Sep 19 04:49:09.547821 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:09.547263    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="39081cb6-0611-41e5-b50e-bfe7759ef529" containerName="istio-proxy"
Sep 19 04:49:09.547821 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:09.547273    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="39081cb6-0611-41e5-b50e-bfe7759ef529" containerName="istio-proxy"
Sep 19 04:49:09.547821 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:09.547281    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0c01ced-87c7-4491-aa9c-2eabd3789794" containerName="istio-proxy"
Sep 19 04:49:09.547821 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:09.547286    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0c01ced-87c7-4491-aa9c-2eabd3789794" containerName="istio-proxy"
Sep 19 04:49:09.547821 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:09.547292    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="91cd381a-3058-4df7-954e-984be634029e" containerName="main"
Sep 19 04:49:09.547821 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:09.547297    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="98309c08-4d67-45bc-9848-ec80ec181224" containerName="main"
Sep 19 04:49:09.547821 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:09.547302    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ca3da4a-b847-47c4-a1cf-32e7f8643cea" containerName="main"
Sep 19 04:49:09.547821 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:09.547309    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="27bea02f-f375-4c7b-a50d-96123a14255f" containerName="main"
Sep 19 04:49:09.550552 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:09.550530    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10379-8a4e4b"
Sep 19 04:49:09.553389 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:09.553350    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-6vmt9\""
Sep 19 04:49:09.555095 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:09.555069    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-10379-8a4e4b"]
Sep 19 04:49:09.703203 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:09.703168    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwbfq\" (UniqueName: \"kubernetes.io/projected/ac357fa0-8331-45b9-8ab6-73092d8d8d99-kube-api-access-hwbfq\") pod \"test-curl-10379-8a4e4b\" (UID: \"ac357fa0-8331-45b9-8ab6-73092d8d8d99\") " pod="opendatahub/test-curl-10379-8a4e4b"
Sep 19 04:49:09.804402 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:09.804306    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwbfq\" (UniqueName: \"kubernetes.io/projected/ac357fa0-8331-45b9-8ab6-73092d8d8d99-kube-api-access-hwbfq\") pod \"test-curl-10379-8a4e4b\" (UID: \"ac357fa0-8331-45b9-8ab6-73092d8d8d99\") " pod="opendatahub/test-curl-10379-8a4e4b"
Sep 19 04:49:09.813594 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:09.813569    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwbfq\" (UniqueName: \"kubernetes.io/projected/ac357fa0-8331-45b9-8ab6-73092d8d8d99-kube-api-access-hwbfq\") pod \"test-curl-10379-8a4e4b\" (UID: \"ac357fa0-8331-45b9-8ab6-73092d8d8d99\") " pod="opendatahub/test-curl-10379-8a4e4b"
Sep 19 04:49:09.861659 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:09.861630    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10379-8a4e4b"
Sep 19 04:49:10.134194 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:10.132878    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-10379-8a4e4b"]
Sep 19 04:49:10.135421 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:49:10.135388    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podac357fa0_8331_45b9_8ab6_73092d8d8d99.slice/crio-f1a91bacdcab6830fa22ef730fbe742bf8134214ae0cd28af0dbcc594f263853 WatchSource:0}: Error finding container f1a91bacdcab6830fa22ef730fbe742bf8134214ae0cd28af0dbcc594f263853: Status 404 returned error can't find the container with id f1a91bacdcab6830fa22ef730fbe742bf8134214ae0cd28af0dbcc594f263853
Sep 19 04:49:10.137681 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:10.137662    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 19 04:49:10.557622 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:10.557589    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-10379-8a4e4b" event={"ID":"ac357fa0-8331-45b9-8ab6-73092d8d8d99","Type":"ContainerStarted","Data":"f1a91bacdcab6830fa22ef730fbe742bf8134214ae0cd28af0dbcc594f263853"}
Sep 19 04:49:12.571937 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:12.571894    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-10379-8a4e4b" event={"ID":"ac357fa0-8331-45b9-8ab6-73092d8d8d99","Type":"ContainerStarted","Data":"50efb411f85afbdf2742e014194e5549042eb22a566035937bd0fef9be0724c0"}
Sep 19 04:49:12.595773 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:12.595662    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-10379-8a4e4b" podStartSLOduration=1.39172235 podStartE2EDuration="3.595647731s" totalImagesPullingTime="2.203925381s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 04:49:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:49:12.593519364 +0000 UTC m=+2131.776480511" watchObservedRunningTime="2026-09-19 04:49:12.595647731 +0000 UTC m=+2131.778608878"
Sep 19 04:49:13.223292 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:13.223254    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-10379-8a4e4b"]
Sep 19 04:49:13.227839 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:13.227813    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-10379-8a4e4b"]
Sep 19 04:49:13.578119 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:49:13.578086    2579 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-10379-8a4e4b" container="test-curl-10379-8a4e4b"
Sep 19 04:49:14.583623 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:14.583570    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-10379-8a4e4b" podUID="ac357fa0-8331-45b9-8ab6-73092d8d8d99" containerName="test-curl-10379-8a4e4b" containerID="cri-o://50efb411f85afbdf2742e014194e5549042eb22a566035937bd0fef9be0724c0" gracePeriod=2
Sep 19 04:49:14.691460 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:14.691419    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-561bb1/auth-test-model-561bb1-kserve-747479f548-x5znp"]
Sep 19 04:49:16.775632 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:16.775601    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10379-8a4e4b"
Sep 19 04:49:16.963461 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:16.963430    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hwbfq\" (UniqueName: \"kubernetes.io/projected/ac357fa0-8331-45b9-8ab6-73092d8d8d99-kube-api-access-hwbfq\") pod \"ac357fa0-8331-45b9-8ab6-73092d8d8d99\" (UID: \"ac357fa0-8331-45b9-8ab6-73092d8d8d99\") "
Sep 19 04:49:16.965639 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:16.965614    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac357fa0-8331-45b9-8ab6-73092d8d8d99-kube-api-access-hwbfq" (OuterVolumeSpecName: "kube-api-access-hwbfq") pod "ac357fa0-8331-45b9-8ab6-73092d8d8d99" (UID: "ac357fa0-8331-45b9-8ab6-73092d8d8d99"). InnerVolumeSpecName "kube-api-access-hwbfq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 04:49:17.064649 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:17.064597    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwbfq\" (UniqueName: \"kubernetes.io/projected/ac357fa0-8331-45b9-8ab6-73092d8d8d99-kube-api-access-hwbfq\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:49:17.352013 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:17.351931    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac357fa0-8331-45b9-8ab6-73092d8d8d99" path="/var/lib/kubelet/pods/ac357fa0-8331-45b9-8ab6-73092d8d8d99/volumes"
Sep 19 04:49:17.604115 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:17.604008    2579 generic.go:356] "Generic (PLEG): container finished" podID="ac357fa0-8331-45b9-8ab6-73092d8d8d99" containerID="50efb411f85afbdf2742e014194e5549042eb22a566035937bd0fef9be0724c0" exitCode=137
Sep 19 04:49:17.604115 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:17.604055    2579 scope.go:117] "RemoveContainer" containerID="50efb411f85afbdf2742e014194e5549042eb22a566035937bd0fef9be0724c0"
Sep 19 04:49:17.604115 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:17.604062    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10379-8a4e4b"
Sep 19 04:49:17.613910 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:17.613886    2579 scope.go:117] "RemoveContainer" containerID="50efb411f85afbdf2742e014194e5549042eb22a566035937bd0fef9be0724c0"
Sep 19 04:49:17.614186 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:49:17.614166    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50efb411f85afbdf2742e014194e5549042eb22a566035937bd0fef9be0724c0\": container with ID starting with 50efb411f85afbdf2742e014194e5549042eb22a566035937bd0fef9be0724c0 not found: ID does not exist" containerID="50efb411f85afbdf2742e014194e5549042eb22a566035937bd0fef9be0724c0"
Sep 19 04:49:17.614240 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:17.614194    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50efb411f85afbdf2742e014194e5549042eb22a566035937bd0fef9be0724c0"} err="failed to get container status \"50efb411f85afbdf2742e014194e5549042eb22a566035937bd0fef9be0724c0\": rpc error: code = NotFound desc = could not find container \"50efb411f85afbdf2742e014194e5549042eb22a566035937bd0fef9be0724c0\": container with ID starting with 50efb411f85afbdf2742e014194e5549042eb22a566035937bd0fef9be0724c0 not found: ID does not exist"
Sep 19 04:49:28.759350 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:28.759320    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-561bb1/sub-test-model-561bb1-kserve-755b9c9866-28mqt"]
Sep 19 04:49:28.759830 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:28.759684    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac357fa0-8331-45b9-8ab6-73092d8d8d99" containerName="test-curl-10379-8a4e4b"
Sep 19 04:49:28.759830 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:28.759717    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac357fa0-8331-45b9-8ab6-73092d8d8d99" containerName="test-curl-10379-8a4e4b"
Sep 19 04:49:28.759830 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:28.759799    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac357fa0-8331-45b9-8ab6-73092d8d8d99" containerName="test-curl-10379-8a4e4b"
Sep 19 04:49:28.764086 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:28.764068    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-561bb1/sub-test-model-561bb1-kserve-755b9c9866-28mqt"
Sep 19 04:49:28.766668 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:28.766649    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-561bb1\"/\"sub-test-model-561bb1-kserve-self-signed-certs\""
Sep 19 04:49:28.778477 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:28.778438    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-561bb1/sub-test-model-561bb1-kserve-755b9c9866-28mqt"]
Sep 19 04:49:28.865544 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:28.865499    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/188b3bbc-7ac0-4244-9ee1-87d90652f6ff-home\") pod \"sub-test-model-561bb1-kserve-755b9c9866-28mqt\" (UID: \"188b3bbc-7ac0-4244-9ee1-87d90652f6ff\") " pod="ai-tenant-e2e-shared-a-w3-561bb1/sub-test-model-561bb1-kserve-755b9c9866-28mqt"
Sep 19 04:49:28.865544 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:28.865543    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/188b3bbc-7ac0-4244-9ee1-87d90652f6ff-tls-certs\") pod \"sub-test-model-561bb1-kserve-755b9c9866-28mqt\" (UID: \"188b3bbc-7ac0-4244-9ee1-87d90652f6ff\") " pod="ai-tenant-e2e-shared-a-w3-561bb1/sub-test-model-561bb1-kserve-755b9c9866-28mqt"
Sep 19 04:49:28.865788 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:28.865572    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/188b3bbc-7ac0-4244-9ee1-87d90652f6ff-dshm\") pod \"sub-test-model-561bb1-kserve-755b9c9866-28mqt\" (UID: \"188b3bbc-7ac0-4244-9ee1-87d90652f6ff\") " pod="ai-tenant-e2e-shared-a-w3-561bb1/sub-test-model-561bb1-kserve-755b9c9866-28mqt"
Sep 19 04:49:28.865788 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:28.865747    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/188b3bbc-7ac0-4244-9ee1-87d90652f6ff-tmp-dir\") pod \"sub-test-model-561bb1-kserve-755b9c9866-28mqt\" (UID: \"188b3bbc-7ac0-4244-9ee1-87d90652f6ff\") " pod="ai-tenant-e2e-shared-a-w3-561bb1/sub-test-model-561bb1-kserve-755b9c9866-28mqt"
Sep 19 04:49:28.865858 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:28.865793    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/188b3bbc-7ac0-4244-9ee1-87d90652f6ff-model-cache\") pod \"sub-test-model-561bb1-kserve-755b9c9866-28mqt\" (UID: \"188b3bbc-7ac0-4244-9ee1-87d90652f6ff\") " pod="ai-tenant-e2e-shared-a-w3-561bb1/sub-test-model-561bb1-kserve-755b9c9866-28mqt"
Sep 19 04:49:28.865898 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:28.865877    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwhpn\" (UniqueName: \"kubernetes.io/projected/188b3bbc-7ac0-4244-9ee1-87d90652f6ff-kube-api-access-pwhpn\") pod \"sub-test-model-561bb1-kserve-755b9c9866-28mqt\" (UID: \"188b3bbc-7ac0-4244-9ee1-87d90652f6ff\") " pod="ai-tenant-e2e-shared-a-w3-561bb1/sub-test-model-561bb1-kserve-755b9c9866-28mqt"
Sep 19 04:49:28.966491 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:28.966452    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/188b3bbc-7ac0-4244-9ee1-87d90652f6ff-tmp-dir\") pod \"sub-test-model-561bb1-kserve-755b9c9866-28mqt\" (UID: \"188b3bbc-7ac0-4244-9ee1-87d90652f6ff\") " pod="ai-tenant-e2e-shared-a-w3-561bb1/sub-test-model-561bb1-kserve-755b9c9866-28mqt"
Sep 19 04:49:28.966491 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:28.966491    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/188b3bbc-7ac0-4244-9ee1-87d90652f6ff-model-cache\") pod \"sub-test-model-561bb1-kserve-755b9c9866-28mqt\" (UID: \"188b3bbc-7ac0-4244-9ee1-87d90652f6ff\") " pod="ai-tenant-e2e-shared-a-w3-561bb1/sub-test-model-561bb1-kserve-755b9c9866-28mqt"
Sep 19 04:49:28.966778 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:28.966528    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwhpn\" (UniqueName: \"kubernetes.io/projected/188b3bbc-7ac0-4244-9ee1-87d90652f6ff-kube-api-access-pwhpn\") pod \"sub-test-model-561bb1-kserve-755b9c9866-28mqt\" (UID: \"188b3bbc-7ac0-4244-9ee1-87d90652f6ff\") " pod="ai-tenant-e2e-shared-a-w3-561bb1/sub-test-model-561bb1-kserve-755b9c9866-28mqt"
Sep 19 04:49:28.966778 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:28.966548    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/188b3bbc-7ac0-4244-9ee1-87d90652f6ff-home\") pod \"sub-test-model-561bb1-kserve-755b9c9866-28mqt\" (UID: \"188b3bbc-7ac0-4244-9ee1-87d90652f6ff\") " pod="ai-tenant-e2e-shared-a-w3-561bb1/sub-test-model-561bb1-kserve-755b9c9866-28mqt"
Sep 19 04:49:28.966778 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:28.966571    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/188b3bbc-7ac0-4244-9ee1-87d90652f6ff-tls-certs\") pod \"sub-test-model-561bb1-kserve-755b9c9866-28mqt\" (UID: \"188b3bbc-7ac0-4244-9ee1-87d90652f6ff\") " pod="ai-tenant-e2e-shared-a-w3-561bb1/sub-test-model-561bb1-kserve-755b9c9866-28mqt"
Sep 19 04:49:28.967061 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:28.967033    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/188b3bbc-7ac0-4244-9ee1-87d90652f6ff-tmp-dir\") pod \"sub-test-model-561bb1-kserve-755b9c9866-28mqt\" (UID: \"188b3bbc-7ac0-4244-9ee1-87d90652f6ff\") " pod="ai-tenant-e2e-shared-a-w3-561bb1/sub-test-model-561bb1-kserve-755b9c9866-28mqt"
Sep 19 04:49:28.967147 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:28.967044    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/188b3bbc-7ac0-4244-9ee1-87d90652f6ff-dshm\") pod \"sub-test-model-561bb1-kserve-755b9c9866-28mqt\" (UID: \"188b3bbc-7ac0-4244-9ee1-87d90652f6ff\") " pod="ai-tenant-e2e-shared-a-w3-561bb1/sub-test-model-561bb1-kserve-755b9c9866-28mqt"
Sep 19 04:49:28.967241 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:28.967224    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/188b3bbc-7ac0-4244-9ee1-87d90652f6ff-model-cache\") pod \"sub-test-model-561bb1-kserve-755b9c9866-28mqt\" (UID: \"188b3bbc-7ac0-4244-9ee1-87d90652f6ff\") " pod="ai-tenant-e2e-shared-a-w3-561bb1/sub-test-model-561bb1-kserve-755b9c9866-28mqt"
Sep 19 04:49:28.967408 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:28.967306    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/188b3bbc-7ac0-4244-9ee1-87d90652f6ff-home\") pod \"sub-test-model-561bb1-kserve-755b9c9866-28mqt\" (UID: \"188b3bbc-7ac0-4244-9ee1-87d90652f6ff\") " pod="ai-tenant-e2e-shared-a-w3-561bb1/sub-test-model-561bb1-kserve-755b9c9866-28mqt"
Sep 19 04:49:28.969943 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:28.969917    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/188b3bbc-7ac0-4244-9ee1-87d90652f6ff-tls-certs\") pod \"sub-test-model-561bb1-kserve-755b9c9866-28mqt\" (UID: \"188b3bbc-7ac0-4244-9ee1-87d90652f6ff\") " pod="ai-tenant-e2e-shared-a-w3-561bb1/sub-test-model-561bb1-kserve-755b9c9866-28mqt"
Sep 19 04:49:28.970115 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:28.970087    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/188b3bbc-7ac0-4244-9ee1-87d90652f6ff-dshm\") pod \"sub-test-model-561bb1-kserve-755b9c9866-28mqt\" (UID: \"188b3bbc-7ac0-4244-9ee1-87d90652f6ff\") " pod="ai-tenant-e2e-shared-a-w3-561bb1/sub-test-model-561bb1-kserve-755b9c9866-28mqt"
Sep 19 04:49:28.976957 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:28.976933    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwhpn\" (UniqueName: \"kubernetes.io/projected/188b3bbc-7ac0-4244-9ee1-87d90652f6ff-kube-api-access-pwhpn\") pod \"sub-test-model-561bb1-kserve-755b9c9866-28mqt\" (UID: \"188b3bbc-7ac0-4244-9ee1-87d90652f6ff\") " pod="ai-tenant-e2e-shared-a-w3-561bb1/sub-test-model-561bb1-kserve-755b9c9866-28mqt"
Sep 19 04:49:29.074279 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:29.074180    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-561bb1/sub-test-model-561bb1-kserve-755b9c9866-28mqt"
Sep 19 04:49:29.353525 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:29.353502    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-561bb1/sub-test-model-561bb1-kserve-755b9c9866-28mqt"]
Sep 19 04:49:29.354990 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:49:29.354967    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod188b3bbc_7ac0_4244_9ee1_87d90652f6ff.slice/crio-f01cf38c76fa67dca17204b28812e9b542259ad23cd9783c440632fdcf5b684e WatchSource:0}: Error finding container f01cf38c76fa67dca17204b28812e9b542259ad23cd9783c440632fdcf5b684e: Status 404 returned error can't find the container with id f01cf38c76fa67dca17204b28812e9b542259ad23cd9783c440632fdcf5b684e
Sep 19 04:49:29.678469 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:29.678421    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-561bb1/sub-test-model-561bb1-kserve-755b9c9866-28mqt" event={"ID":"188b3bbc-7ac0-4244-9ee1-87d90652f6ff","Type":"ContainerStarted","Data":"e93e8f5a01fe82013923ca874db927baf013a7b5e3de760d282385b67a925e30"}
Sep 19 04:49:29.678661 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:29.678482    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-561bb1/sub-test-model-561bb1-kserve-755b9c9866-28mqt" event={"ID":"188b3bbc-7ac0-4244-9ee1-87d90652f6ff","Type":"ContainerStarted","Data":"f01cf38c76fa67dca17204b28812e9b542259ad23cd9783c440632fdcf5b684e"}
Sep 19 04:49:29.704372 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:29.704315    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-561bb1/sub-test-model-561bb1-kserve-755b9c9866-28mqt" podStartSLOduration=1.70429915 podStartE2EDuration="1.70429915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 04:49:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:49:29.701986849 +0000 UTC m=+2148.884947997" watchObservedRunningTime="2026-09-19 04:49:29.70429915 +0000 UTC m=+2148.887260297"
Sep 19 04:49:29.710602 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:29.710558    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-561bb1/auth-test-model-561bb1-kserve-747479f548-x5znp" podUID="8e185da2-a21a-48ea-858f-e2c225df87d6" containerName="main" containerID="cri-o://14308c49e918f6062fd6b5da38953f17ff1ff0f5aa0b97b4647ed37c1c87de99" gracePeriod=45
Sep 19 04:49:29.952221 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:29.952187    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-561bb1/auth-test-model-561bb1-kserve-747479f548-x5znp"
Sep 19 04:49:29.977533 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:29.977504    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6mddz\" (UniqueName: \"kubernetes.io/projected/8e185da2-a21a-48ea-858f-e2c225df87d6-kube-api-access-6mddz\") pod \"8e185da2-a21a-48ea-858f-e2c225df87d6\" (UID: \"8e185da2-a21a-48ea-858f-e2c225df87d6\") "
Sep 19 04:49:29.977722 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:29.977552    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e185da2-a21a-48ea-858f-e2c225df87d6-dshm\") pod \"8e185da2-a21a-48ea-858f-e2c225df87d6\" (UID: \"8e185da2-a21a-48ea-858f-e2c225df87d6\") "
Sep 19 04:49:29.977722 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:29.977587    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e185da2-a21a-48ea-858f-e2c225df87d6-tmp-dir\") pod \"8e185da2-a21a-48ea-858f-e2c225df87d6\" (UID: \"8e185da2-a21a-48ea-858f-e2c225df87d6\") "
Sep 19 04:49:29.977722 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:29.977615    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e185da2-a21a-48ea-858f-e2c225df87d6-home\") pod \"8e185da2-a21a-48ea-858f-e2c225df87d6\" (UID: \"8e185da2-a21a-48ea-858f-e2c225df87d6\") "
Sep 19 04:49:29.977722 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:29.977653    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e185da2-a21a-48ea-858f-e2c225df87d6-tls-certs\") pod \"8e185da2-a21a-48ea-858f-e2c225df87d6\" (UID: \"8e185da2-a21a-48ea-858f-e2c225df87d6\") "
Sep 19 04:49:29.977722 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:29.977677    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e185da2-a21a-48ea-858f-e2c225df87d6-model-cache\") pod \"8e185da2-a21a-48ea-858f-e2c225df87d6\" (UID: \"8e185da2-a21a-48ea-858f-e2c225df87d6\") "
Sep 19 04:49:29.978001 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:29.977892    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e185da2-a21a-48ea-858f-e2c225df87d6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8e185da2-a21a-48ea-858f-e2c225df87d6" (UID: "8e185da2-a21a-48ea-858f-e2c225df87d6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:49:29.978001 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:29.977955    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e185da2-a21a-48ea-858f-e2c225df87d6-model-cache" (OuterVolumeSpecName: "model-cache") pod "8e185da2-a21a-48ea-858f-e2c225df87d6" (UID: "8e185da2-a21a-48ea-858f-e2c225df87d6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:49:29.978110 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:29.978030    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e185da2-a21a-48ea-858f-e2c225df87d6-tmp-dir\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:49:29.978110 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:29.978050    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e185da2-a21a-48ea-858f-e2c225df87d6-model-cache\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:49:29.978340 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:29.978312    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e185da2-a21a-48ea-858f-e2c225df87d6-home" (OuterVolumeSpecName: "home") pod "8e185da2-a21a-48ea-858f-e2c225df87d6" (UID: "8e185da2-a21a-48ea-858f-e2c225df87d6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:49:29.980089 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:29.980059    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e185da2-a21a-48ea-858f-e2c225df87d6-kube-api-access-6mddz" (OuterVolumeSpecName: "kube-api-access-6mddz") pod "8e185da2-a21a-48ea-858f-e2c225df87d6" (UID: "8e185da2-a21a-48ea-858f-e2c225df87d6"). InnerVolumeSpecName "kube-api-access-6mddz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 04:49:29.980179 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:29.980083    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e185da2-a21a-48ea-858f-e2c225df87d6-dshm" (OuterVolumeSpecName: "dshm") pod "8e185da2-a21a-48ea-858f-e2c225df87d6" (UID: "8e185da2-a21a-48ea-858f-e2c225df87d6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:49:29.980179 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:29.980163    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e185da2-a21a-48ea-858f-e2c225df87d6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8e185da2-a21a-48ea-858f-e2c225df87d6" (UID: "8e185da2-a21a-48ea-858f-e2c225df87d6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 04:49:30.074328 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:30.074291    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-561bb1/sub-test-model-561bb1-kserve-755b9c9866-28mqt"
Sep 19 04:49:30.078857 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:30.078835    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mddz\" (UniqueName: \"kubernetes.io/projected/8e185da2-a21a-48ea-858f-e2c225df87d6-kube-api-access-6mddz\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:49:30.078949 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:30.078860    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e185da2-a21a-48ea-858f-e2c225df87d6-dshm\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:49:30.078949 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:30.078869    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e185da2-a21a-48ea-858f-e2c225df87d6-home\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:49:30.078949 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:30.078877    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e185da2-a21a-48ea-858f-e2c225df87d6-tls-certs\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:49:30.686576 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:30.686543    2579 generic.go:356] "Generic (PLEG): container finished" podID="8e185da2-a21a-48ea-858f-e2c225df87d6" containerID="14308c49e918f6062fd6b5da38953f17ff1ff0f5aa0b97b4647ed37c1c87de99" exitCode=0
Sep 19 04:49:30.686817 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:30.686601    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-561bb1/auth-test-model-561bb1-kserve-747479f548-x5znp"
Sep 19 04:49:30.686817 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:30.686630    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-561bb1/auth-test-model-561bb1-kserve-747479f548-x5znp" event={"ID":"8e185da2-a21a-48ea-858f-e2c225df87d6","Type":"ContainerDied","Data":"14308c49e918f6062fd6b5da38953f17ff1ff0f5aa0b97b4647ed37c1c87de99"}
Sep 19 04:49:30.686817 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:30.686680    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-561bb1/auth-test-model-561bb1-kserve-747479f548-x5znp" event={"ID":"8e185da2-a21a-48ea-858f-e2c225df87d6","Type":"ContainerDied","Data":"54b12f6d7b2806278bb91252a86bc89b9d1f849835d4412a0110a4efbc3f1e70"}
Sep 19 04:49:30.686817 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:30.686732    2579 scope.go:117] "RemoveContainer" containerID="14308c49e918f6062fd6b5da38953f17ff1ff0f5aa0b97b4647ed37c1c87de99"
Sep 19 04:49:30.697110 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:30.697093    2579 scope.go:117] "RemoveContainer" containerID="14308c49e918f6062fd6b5da38953f17ff1ff0f5aa0b97b4647ed37c1c87de99"
Sep 19 04:49:30.697372 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:49:30.697354    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14308c49e918f6062fd6b5da38953f17ff1ff0f5aa0b97b4647ed37c1c87de99\": container with ID starting with 14308c49e918f6062fd6b5da38953f17ff1ff0f5aa0b97b4647ed37c1c87de99 not found: ID does not exist" containerID="14308c49e918f6062fd6b5da38953f17ff1ff0f5aa0b97b4647ed37c1c87de99"
Sep 19 04:49:30.697428 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:30.697380    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14308c49e918f6062fd6b5da38953f17ff1ff0f5aa0b97b4647ed37c1c87de99"} err="failed to get container status \"14308c49e918f6062fd6b5da38953f17ff1ff0f5aa0b97b4647ed37c1c87de99\": rpc error: code = NotFound desc = could not find container \"14308c49e918f6062fd6b5da38953f17ff1ff0f5aa0b97b4647ed37c1c87de99\": container with ID starting with 14308c49e918f6062fd6b5da38953f17ff1ff0f5aa0b97b4647ed37c1c87de99 not found: ID does not exist"
Sep 19 04:49:30.721243 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:30.721219    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-561bb1/auth-test-model-561bb1-kserve-747479f548-x5znp"]
Sep 19 04:49:30.725110 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:30.725089    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-561bb1/auth-test-model-561bb1-kserve-747479f548-x5znp"]
Sep 19 04:49:31.351019 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:31.350984    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e185da2-a21a-48ea-858f-e2c225df87d6" path="/var/lib/kubelet/pods/8e185da2-a21a-48ea-858f-e2c225df87d6/volumes"
Sep 19 04:49:39.075372 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:39.075282    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-561bb1/sub-test-model-561bb1-kserve-755b9c9866-28mqt"
Sep 19 04:49:39.088824 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:39.088792    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-561bb1/sub-test-model-561bb1-kserve-755b9c9866-28mqt"
Sep 19 04:49:39.749233 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:39.749195    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-561bb1/sub-test-model-561bb1-kserve-755b9c9866-28mqt"
Sep 19 04:49:41.796816 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:41.796783    2579 scope.go:117] "RemoveContainer" containerID="cfe7610fd69df562a3852f1b2b72bc5e48948d6f5f6038dcfc9bb8f7d7f2feb9"
Sep 19 04:49:54.651257 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:54.651204    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-8bbcca/sub-test-model-8bbcca-kserve-bf68fd574-cg6sz"]
Sep 19 04:49:54.651783 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:54.651763    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e185da2-a21a-48ea-858f-e2c225df87d6" containerName="main"
Sep 19 04:49:54.651839 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:54.651786    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e185da2-a21a-48ea-858f-e2c225df87d6" containerName="main"
Sep 19 04:49:54.651876 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:54.651866    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e185da2-a21a-48ea-858f-e2c225df87d6" containerName="main"
Sep 19 04:49:54.655100 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:54.655083    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-8bbcca/sub-test-model-8bbcca-kserve-bf68fd574-cg6sz"
Sep 19 04:49:54.661965 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:54.661944    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-8bbcca\"/\"sub-test-model-8bbcca-kserve-self-signed-certs\""
Sep 19 04:49:54.662053 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:54.661974    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-8bbcca\"/\"openshift-service-ca.crt\""
Sep 19 04:49:54.662053 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:54.662030    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-8bbcca\"/\"default-dockercfg-pv8fk\""
Sep 19 04:49:54.662163 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:54.661980    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-8bbcca\"/\"kube-root-ca.crt\""
Sep 19 04:49:54.668863 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:54.668839    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8bbcca/sub-test-model-8bbcca-kserve-bf68fd574-cg6sz"]
Sep 19 04:49:54.812327 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:54.812283    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/568ba356-2a61-4f5f-bb23-4d261e3f9c42-model-cache\") pod \"sub-test-model-8bbcca-kserve-bf68fd574-cg6sz\" (UID: \"568ba356-2a61-4f5f-bb23-4d261e3f9c42\") " pod="ai-tenant-e2e-shared-b-w3-8bbcca/sub-test-model-8bbcca-kserve-bf68fd574-cg6sz"
Sep 19 04:49:54.812327 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:54.812320    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/568ba356-2a61-4f5f-bb23-4d261e3f9c42-dshm\") pod \"sub-test-model-8bbcca-kserve-bf68fd574-cg6sz\" (UID: \"568ba356-2a61-4f5f-bb23-4d261e3f9c42\") " pod="ai-tenant-e2e-shared-b-w3-8bbcca/sub-test-model-8bbcca-kserve-bf68fd574-cg6sz"
Sep 19 04:49:54.812596 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:54.812422    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/568ba356-2a61-4f5f-bb23-4d261e3f9c42-tls-certs\") pod \"sub-test-model-8bbcca-kserve-bf68fd574-cg6sz\" (UID: \"568ba356-2a61-4f5f-bb23-4d261e3f9c42\") " pod="ai-tenant-e2e-shared-b-w3-8bbcca/sub-test-model-8bbcca-kserve-bf68fd574-cg6sz"
Sep 19 04:49:54.812596 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:54.812458    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvgkj\" (UniqueName: \"kubernetes.io/projected/568ba356-2a61-4f5f-bb23-4d261e3f9c42-kube-api-access-pvgkj\") pod \"sub-test-model-8bbcca-kserve-bf68fd574-cg6sz\" (UID: \"568ba356-2a61-4f5f-bb23-4d261e3f9c42\") " pod="ai-tenant-e2e-shared-b-w3-8bbcca/sub-test-model-8bbcca-kserve-bf68fd574-cg6sz"
Sep 19 04:49:54.812596 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:54.812552    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/568ba356-2a61-4f5f-bb23-4d261e3f9c42-tmp-dir\") pod \"sub-test-model-8bbcca-kserve-bf68fd574-cg6sz\" (UID: \"568ba356-2a61-4f5f-bb23-4d261e3f9c42\") " pod="ai-tenant-e2e-shared-b-w3-8bbcca/sub-test-model-8bbcca-kserve-bf68fd574-cg6sz"
Sep 19 04:49:54.812810 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:54.812615    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/568ba356-2a61-4f5f-bb23-4d261e3f9c42-home\") pod \"sub-test-model-8bbcca-kserve-bf68fd574-cg6sz\" (UID: \"568ba356-2a61-4f5f-bb23-4d261e3f9c42\") " pod="ai-tenant-e2e-shared-b-w3-8bbcca/sub-test-model-8bbcca-kserve-bf68fd574-cg6sz"
Sep 19 04:49:54.913630 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:54.913538    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/568ba356-2a61-4f5f-bb23-4d261e3f9c42-tmp-dir\") pod \"sub-test-model-8bbcca-kserve-bf68fd574-cg6sz\" (UID: \"568ba356-2a61-4f5f-bb23-4d261e3f9c42\") " pod="ai-tenant-e2e-shared-b-w3-8bbcca/sub-test-model-8bbcca-kserve-bf68fd574-cg6sz"
Sep 19 04:49:54.913630 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:54.913618    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/568ba356-2a61-4f5f-bb23-4d261e3f9c42-home\") pod \"sub-test-model-8bbcca-kserve-bf68fd574-cg6sz\" (UID: \"568ba356-2a61-4f5f-bb23-4d261e3f9c42\") " pod="ai-tenant-e2e-shared-b-w3-8bbcca/sub-test-model-8bbcca-kserve-bf68fd574-cg6sz"
Sep 19 04:49:54.913898 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:54.913736    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/568ba356-2a61-4f5f-bb23-4d261e3f9c42-model-cache\") pod \"sub-test-model-8bbcca-kserve-bf68fd574-cg6sz\" (UID: \"568ba356-2a61-4f5f-bb23-4d261e3f9c42\") " pod="ai-tenant-e2e-shared-b-w3-8bbcca/sub-test-model-8bbcca-kserve-bf68fd574-cg6sz"
Sep 19 04:49:54.913898 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:54.913763    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/568ba356-2a61-4f5f-bb23-4d261e3f9c42-dshm\") pod \"sub-test-model-8bbcca-kserve-bf68fd574-cg6sz\" (UID: \"568ba356-2a61-4f5f-bb23-4d261e3f9c42\") " pod="ai-tenant-e2e-shared-b-w3-8bbcca/sub-test-model-8bbcca-kserve-bf68fd574-cg6sz"
Sep 19 04:49:54.913898 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:54.913815    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/568ba356-2a61-4f5f-bb23-4d261e3f9c42-tls-certs\") pod \"sub-test-model-8bbcca-kserve-bf68fd574-cg6sz\" (UID: \"568ba356-2a61-4f5f-bb23-4d261e3f9c42\") " pod="ai-tenant-e2e-shared-b-w3-8bbcca/sub-test-model-8bbcca-kserve-bf68fd574-cg6sz"
Sep 19 04:49:54.913898 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:54.913839    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvgkj\" (UniqueName: \"kubernetes.io/projected/568ba356-2a61-4f5f-bb23-4d261e3f9c42-kube-api-access-pvgkj\") pod \"sub-test-model-8bbcca-kserve-bf68fd574-cg6sz\" (UID: \"568ba356-2a61-4f5f-bb23-4d261e3f9c42\") " pod="ai-tenant-e2e-shared-b-w3-8bbcca/sub-test-model-8bbcca-kserve-bf68fd574-cg6sz"
Sep 19 04:49:54.913898 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:54.913815    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/568ba356-2a61-4f5f-bb23-4d261e3f9c42-tmp-dir\") pod \"sub-test-model-8bbcca-kserve-bf68fd574-cg6sz\" (UID: \"568ba356-2a61-4f5f-bb23-4d261e3f9c42\") " pod="ai-tenant-e2e-shared-b-w3-8bbcca/sub-test-model-8bbcca-kserve-bf68fd574-cg6sz"
Sep 19 04:49:54.914123 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:54.913953    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/568ba356-2a61-4f5f-bb23-4d261e3f9c42-home\") pod \"sub-test-model-8bbcca-kserve-bf68fd574-cg6sz\" (UID: \"568ba356-2a61-4f5f-bb23-4d261e3f9c42\") " pod="ai-tenant-e2e-shared-b-w3-8bbcca/sub-test-model-8bbcca-kserve-bf68fd574-cg6sz"
Sep 19 04:49:54.914123 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:54.914073    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/568ba356-2a61-4f5f-bb23-4d261e3f9c42-model-cache\") pod \"sub-test-model-8bbcca-kserve-bf68fd574-cg6sz\" (UID: \"568ba356-2a61-4f5f-bb23-4d261e3f9c42\") " pod="ai-tenant-e2e-shared-b-w3-8bbcca/sub-test-model-8bbcca-kserve-bf68fd574-cg6sz"
Sep 19 04:49:54.916211 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:54.916185    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/568ba356-2a61-4f5f-bb23-4d261e3f9c42-dshm\") pod \"sub-test-model-8bbcca-kserve-bf68fd574-cg6sz\" (UID: \"568ba356-2a61-4f5f-bb23-4d261e3f9c42\") " pod="ai-tenant-e2e-shared-b-w3-8bbcca/sub-test-model-8bbcca-kserve-bf68fd574-cg6sz"
Sep 19 04:49:54.916438 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:54.916420    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/568ba356-2a61-4f5f-bb23-4d261e3f9c42-tls-certs\") pod \"sub-test-model-8bbcca-kserve-bf68fd574-cg6sz\" (UID: \"568ba356-2a61-4f5f-bb23-4d261e3f9c42\") " pod="ai-tenant-e2e-shared-b-w3-8bbcca/sub-test-model-8bbcca-kserve-bf68fd574-cg6sz"
Sep 19 04:49:54.924516 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:54.924494    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvgkj\" (UniqueName: \"kubernetes.io/projected/568ba356-2a61-4f5f-bb23-4d261e3f9c42-kube-api-access-pvgkj\") pod \"sub-test-model-8bbcca-kserve-bf68fd574-cg6sz\" (UID: \"568ba356-2a61-4f5f-bb23-4d261e3f9c42\") " pod="ai-tenant-e2e-shared-b-w3-8bbcca/sub-test-model-8bbcca-kserve-bf68fd574-cg6sz"
Sep 19 04:49:54.965083 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:54.965046    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-8bbcca/sub-test-model-8bbcca-kserve-bf68fd574-cg6sz"
Sep 19 04:49:55.251252 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:55.251215    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8bbcca/sub-test-model-8bbcca-kserve-bf68fd574-cg6sz"]
Sep 19 04:49:55.253537 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:49:55.253512    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod568ba356_2a61_4f5f_bb23_4d261e3f9c42.slice/crio-21e8ca59cffeff75fe06db7b8a7047b02963b797ae9f2042f7c1c687ad44e959 WatchSource:0}: Error finding container 21e8ca59cffeff75fe06db7b8a7047b02963b797ae9f2042f7c1c687ad44e959: Status 404 returned error can't find the container with id 21e8ca59cffeff75fe06db7b8a7047b02963b797ae9f2042f7c1c687ad44e959
Sep 19 04:49:55.837768 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:55.837728    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-8bbcca/sub-test-model-8bbcca-kserve-bf68fd574-cg6sz" event={"ID":"568ba356-2a61-4f5f-bb23-4d261e3f9c42","Type":"ContainerStarted","Data":"e6651953046c8da153b715663241f21957bcc0f70a3c6e9067267cbb8572f37c"}
Sep 19 04:49:55.837768 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:55.837774    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-8bbcca/sub-test-model-8bbcca-kserve-bf68fd574-cg6sz" event={"ID":"568ba356-2a61-4f5f-bb23-4d261e3f9c42","Type":"ContainerStarted","Data":"21e8ca59cffeff75fe06db7b8a7047b02963b797ae9f2042f7c1c687ad44e959"}
Sep 19 04:49:55.872569 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:55.872521    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-8bbcca/sub-test-model-8bbcca-kserve-bf68fd574-cg6sz" podStartSLOduration=1.872504237 podStartE2EDuration="1.872504237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 04:49:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:49:55.869133327 +0000 UTC m=+2175.052094475" watchObservedRunningTime="2026-09-19 04:49:55.872504237 +0000 UTC m=+2175.055465413"
Sep 19 04:49:55.965769 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:49:55.965726    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-8bbcca/sub-test-model-8bbcca-kserve-bf68fd574-cg6sz"
Sep 19 04:50:04.965512 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:04.965467    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-8bbcca/sub-test-model-8bbcca-kserve-bf68fd574-cg6sz"
Sep 19 04:50:04.977956 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:04.977932    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-8bbcca/sub-test-model-8bbcca-kserve-bf68fd574-cg6sz"
Sep 19 04:50:05.908466 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:05.908434    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-8bbcca/sub-test-model-8bbcca-kserve-bf68fd574-cg6sz"
Sep 19 04:50:36.588278 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:36.588240    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-10379-5dfc44"]
Sep 19 04:50:36.592450 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:36.592431    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10379-5dfc44"
Sep 19 04:50:36.597264 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:36.597237    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-6vmt9\""
Sep 19 04:50:36.602260 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:36.602235    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-10379-5dfc44"]
Sep 19 04:50:36.717180 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:36.717139    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvsf9\" (UniqueName: \"kubernetes.io/projected/ad13a461-fcf9-4347-aef3-d824098d2c6a-kube-api-access-jvsf9\") pod \"test-curl-10379-5dfc44\" (UID: \"ad13a461-fcf9-4347-aef3-d824098d2c6a\") " pod="opendatahub/test-curl-10379-5dfc44"
Sep 19 04:50:36.818482 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:36.818446    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvsf9\" (UniqueName: \"kubernetes.io/projected/ad13a461-fcf9-4347-aef3-d824098d2c6a-kube-api-access-jvsf9\") pod \"test-curl-10379-5dfc44\" (UID: \"ad13a461-fcf9-4347-aef3-d824098d2c6a\") " pod="opendatahub/test-curl-10379-5dfc44"
Sep 19 04:50:36.830023 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:36.829994    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvsf9\" (UniqueName: \"kubernetes.io/projected/ad13a461-fcf9-4347-aef3-d824098d2c6a-kube-api-access-jvsf9\") pod \"test-curl-10379-5dfc44\" (UID: \"ad13a461-fcf9-4347-aef3-d824098d2c6a\") " pod="opendatahub/test-curl-10379-5dfc44"
Sep 19 04:50:36.903963 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:36.903938    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10379-5dfc44"
Sep 19 04:50:37.192107 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:37.192078    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-10379-5dfc44"]
Sep 19 04:50:37.193768 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:50:37.193731    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podad13a461_fcf9_4347_aef3_d824098d2c6a.slice/crio-3bf6f9f621c9a9e8f2e08a4007e0a79ec5e29283c3dcabb72a46d7eef65b9ec2 WatchSource:0}: Error finding container 3bf6f9f621c9a9e8f2e08a4007e0a79ec5e29283c3dcabb72a46d7eef65b9ec2: Status 404 returned error can't find the container with id 3bf6f9f621c9a9e8f2e08a4007e0a79ec5e29283c3dcabb72a46d7eef65b9ec2
Sep 19 04:50:38.094281 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:38.094211    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-10379-5dfc44" event={"ID":"ad13a461-fcf9-4347-aef3-d824098d2c6a","Type":"ContainerStarted","Data":"2e784365760471ddac5e4efd13b847ded5500bfcf4bb4f993316f4d843ac8b21"}
Sep 19 04:50:38.094281 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:38.094255    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-10379-5dfc44" event={"ID":"ad13a461-fcf9-4347-aef3-d824098d2c6a","Type":"ContainerStarted","Data":"3bf6f9f621c9a9e8f2e08a4007e0a79ec5e29283c3dcabb72a46d7eef65b9ec2"}
Sep 19 04:50:38.117106 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:38.117042    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-10379-5dfc44" podStartSLOduration=2.117024832 podStartE2EDuration="2.117024832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 04:50:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:50:38.114204168 +0000 UTC m=+2217.297165317" watchObservedRunningTime="2026-09-19 04:50:38.117024832 +0000 UTC m=+2217.299985979"
Sep 19 04:50:38.883849 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:38.883763    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-10379-5dfc44"]
Sep 19 04:50:38.889214 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:38.889184    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-10379-5dfc44"]
Sep 19 04:50:39.100103 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:50:39.100073    2579 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-10379-5dfc44" container="test-curl-10379-5dfc44"
Sep 19 04:50:40.105591 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:40.105549    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-10379-5dfc44" podUID="ad13a461-fcf9-4347-aef3-d824098d2c6a" containerName="test-curl-10379-5dfc44" containerID="cri-o://2e784365760471ddac5e4efd13b847ded5500bfcf4bb4f993316f4d843ac8b21" gracePeriod=2
Sep 19 04:50:41.072636 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:41.072601    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-561bb1/sub-test-model-561bb1-kserve-755b9c9866-28mqt"]
Sep 19 04:50:42.299746 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:42.299719    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10379-5dfc44"
Sep 19 04:50:42.370145 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:42.370060    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jvsf9\" (UniqueName: \"kubernetes.io/projected/ad13a461-fcf9-4347-aef3-d824098d2c6a-kube-api-access-jvsf9\") pod \"ad13a461-fcf9-4347-aef3-d824098d2c6a\" (UID: \"ad13a461-fcf9-4347-aef3-d824098d2c6a\") "
Sep 19 04:50:42.372221 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:42.372199    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad13a461-fcf9-4347-aef3-d824098d2c6a-kube-api-access-jvsf9" (OuterVolumeSpecName: "kube-api-access-jvsf9") pod "ad13a461-fcf9-4347-aef3-d824098d2c6a" (UID: "ad13a461-fcf9-4347-aef3-d824098d2c6a"). InnerVolumeSpecName "kube-api-access-jvsf9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 04:50:42.470713 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:42.470672    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvsf9\" (UniqueName: \"kubernetes.io/projected/ad13a461-fcf9-4347-aef3-d824098d2c6a-kube-api-access-jvsf9\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:50:43.123616 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:43.123581    2579 generic.go:356] "Generic (PLEG): container finished" podID="ad13a461-fcf9-4347-aef3-d824098d2c6a" containerID="2e784365760471ddac5e4efd13b847ded5500bfcf4bb4f993316f4d843ac8b21" exitCode=137
Sep 19 04:50:43.123781 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:43.123622    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10379-5dfc44"
Sep 19 04:50:43.123781 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:43.123677    2579 scope.go:117] "RemoveContainer" containerID="2e784365760471ddac5e4efd13b847ded5500bfcf4bb4f993316f4d843ac8b21"
Sep 19 04:50:43.133128 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:43.133109    2579 scope.go:117] "RemoveContainer" containerID="2e784365760471ddac5e4efd13b847ded5500bfcf4bb4f993316f4d843ac8b21"
Sep 19 04:50:43.133416 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:50:43.133399    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e784365760471ddac5e4efd13b847ded5500bfcf4bb4f993316f4d843ac8b21\": container with ID starting with 2e784365760471ddac5e4efd13b847ded5500bfcf4bb4f993316f4d843ac8b21 not found: ID does not exist" containerID="2e784365760471ddac5e4efd13b847ded5500bfcf4bb4f993316f4d843ac8b21"
Sep 19 04:50:43.133490 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:43.133425    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e784365760471ddac5e4efd13b847ded5500bfcf4bb4f993316f4d843ac8b21"} err="failed to get container status \"2e784365760471ddac5e4efd13b847ded5500bfcf4bb4f993316f4d843ac8b21\": rpc error: code = NotFound desc = could not find container \"2e784365760471ddac5e4efd13b847ded5500bfcf4bb4f993316f4d843ac8b21\": container with ID starting with 2e784365760471ddac5e4efd13b847ded5500bfcf4bb4f993316f4d843ac8b21 not found: ID does not exist"
Sep 19 04:50:43.147151 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:50:43.147122    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-10379-5dfc44\" is forbidden: User \"system:node:ip-10-0-129-215.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-215.ec2.internal' and this object" podUID="ad13a461-fcf9-4347-aef3-d824098d2c6a" pod="opendatahub/test-curl-10379-5dfc44"
Sep 19 04:50:43.169857 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:43.169833    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8bbcca/sub-test-model-8bbcca-kserve-bf68fd574-cg6sz"]
Sep 19 04:50:43.214566 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:50:43.214510    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-10379-5dfc44\" is forbidden: User \"system:node:ip-10-0-129-215.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-215.ec2.internal' and this object" podUID="ad13a461-fcf9-4347-aef3-d824098d2c6a" pod="opendatahub/test-curl-10379-5dfc44"
Sep 19 04:50:43.351676 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:43.351637    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad13a461-fcf9-4347-aef3-d824098d2c6a" path="/var/lib/kubelet/pods/ad13a461-fcf9-4347-aef3-d824098d2c6a/volumes"
Sep 19 04:50:53.442269 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:53.442238    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-561bb1/rate-test-model-561bb1-kserve-58545c5c88-t5l49"]
Sep 19 04:50:53.442678 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:53.442610    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad13a461-fcf9-4347-aef3-d824098d2c6a" containerName="test-curl-10379-5dfc44"
Sep 19 04:50:53.442678 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:53.442620    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad13a461-fcf9-4347-aef3-d824098d2c6a" containerName="test-curl-10379-5dfc44"
Sep 19 04:50:53.442799 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:53.442690    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad13a461-fcf9-4347-aef3-d824098d2c6a" containerName="test-curl-10379-5dfc44"
Sep 19 04:50:53.445157 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:53.445140    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-561bb1/rate-test-model-561bb1-kserve-58545c5c88-t5l49"
Sep 19 04:50:53.451081 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:53.451056    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-561bb1\"/\"rate-test-model-561bb1-kserve-self-signed-certs\""
Sep 19 04:50:53.474609 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:53.474577    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55ace636-dbb2-4f5d-a26e-56b624107f1f-home\") pod \"rate-test-model-561bb1-kserve-58545c5c88-t5l49\" (UID: \"55ace636-dbb2-4f5d-a26e-56b624107f1f\") " pod="ai-tenant-e2e-shared-a-w3-561bb1/rate-test-model-561bb1-kserve-58545c5c88-t5l49"
Sep 19 04:50:53.474787 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:53.474646    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsmjc\" (UniqueName: \"kubernetes.io/projected/55ace636-dbb2-4f5d-a26e-56b624107f1f-kube-api-access-vsmjc\") pod \"rate-test-model-561bb1-kserve-58545c5c88-t5l49\" (UID: \"55ace636-dbb2-4f5d-a26e-56b624107f1f\") " pod="ai-tenant-e2e-shared-a-w3-561bb1/rate-test-model-561bb1-kserve-58545c5c88-t5l49"
Sep 19 04:50:53.474787 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:53.474756    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55ace636-dbb2-4f5d-a26e-56b624107f1f-model-cache\") pod \"rate-test-model-561bb1-kserve-58545c5c88-t5l49\" (UID: \"55ace636-dbb2-4f5d-a26e-56b624107f1f\") " pod="ai-tenant-e2e-shared-a-w3-561bb1/rate-test-model-561bb1-kserve-58545c5c88-t5l49"
Sep 19 04:50:53.474787 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:53.474785    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55ace636-dbb2-4f5d-a26e-56b624107f1f-tmp-dir\") pod \"rate-test-model-561bb1-kserve-58545c5c88-t5l49\" (UID: \"55ace636-dbb2-4f5d-a26e-56b624107f1f\") " pod="ai-tenant-e2e-shared-a-w3-561bb1/rate-test-model-561bb1-kserve-58545c5c88-t5l49"
Sep 19 04:50:53.474935 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:53.474817    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55ace636-dbb2-4f5d-a26e-56b624107f1f-tls-certs\") pod \"rate-test-model-561bb1-kserve-58545c5c88-t5l49\" (UID: \"55ace636-dbb2-4f5d-a26e-56b624107f1f\") " pod="ai-tenant-e2e-shared-a-w3-561bb1/rate-test-model-561bb1-kserve-58545c5c88-t5l49"
Sep 19 04:50:53.474971 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:53.474920    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55ace636-dbb2-4f5d-a26e-56b624107f1f-dshm\") pod \"rate-test-model-561bb1-kserve-58545c5c88-t5l49\" (UID: \"55ace636-dbb2-4f5d-a26e-56b624107f1f\") " pod="ai-tenant-e2e-shared-a-w3-561bb1/rate-test-model-561bb1-kserve-58545c5c88-t5l49"
Sep 19 04:50:53.476834 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:53.476807    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-561bb1/rate-test-model-561bb1-kserve-58545c5c88-t5l49"]
Sep 19 04:50:53.575460 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:53.575417    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55ace636-dbb2-4f5d-a26e-56b624107f1f-model-cache\") pod \"rate-test-model-561bb1-kserve-58545c5c88-t5l49\" (UID: \"55ace636-dbb2-4f5d-a26e-56b624107f1f\") " pod="ai-tenant-e2e-shared-a-w3-561bb1/rate-test-model-561bb1-kserve-58545c5c88-t5l49"
Sep 19 04:50:53.575460 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:53.575458    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55ace636-dbb2-4f5d-a26e-56b624107f1f-tmp-dir\") pod \"rate-test-model-561bb1-kserve-58545c5c88-t5l49\" (UID: \"55ace636-dbb2-4f5d-a26e-56b624107f1f\") " pod="ai-tenant-e2e-shared-a-w3-561bb1/rate-test-model-561bb1-kserve-58545c5c88-t5l49"
Sep 19 04:50:53.575766 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:53.575481    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55ace636-dbb2-4f5d-a26e-56b624107f1f-tls-certs\") pod \"rate-test-model-561bb1-kserve-58545c5c88-t5l49\" (UID: \"55ace636-dbb2-4f5d-a26e-56b624107f1f\") " pod="ai-tenant-e2e-shared-a-w3-561bb1/rate-test-model-561bb1-kserve-58545c5c88-t5l49"
Sep 19 04:50:53.575766 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:53.575512    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55ace636-dbb2-4f5d-a26e-56b624107f1f-dshm\") pod \"rate-test-model-561bb1-kserve-58545c5c88-t5l49\" (UID: \"55ace636-dbb2-4f5d-a26e-56b624107f1f\") " pod="ai-tenant-e2e-shared-a-w3-561bb1/rate-test-model-561bb1-kserve-58545c5c88-t5l49"
Sep 19 04:50:53.575766 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:53.575553    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55ace636-dbb2-4f5d-a26e-56b624107f1f-home\") pod \"rate-test-model-561bb1-kserve-58545c5c88-t5l49\" (UID: \"55ace636-dbb2-4f5d-a26e-56b624107f1f\") " pod="ai-tenant-e2e-shared-a-w3-561bb1/rate-test-model-561bb1-kserve-58545c5c88-t5l49"
Sep 19 04:50:53.575766 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:53.575585    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsmjc\" (UniqueName: \"kubernetes.io/projected/55ace636-dbb2-4f5d-a26e-56b624107f1f-kube-api-access-vsmjc\") pod \"rate-test-model-561bb1-kserve-58545c5c88-t5l49\" (UID: \"55ace636-dbb2-4f5d-a26e-56b624107f1f\") " pod="ai-tenant-e2e-shared-a-w3-561bb1/rate-test-model-561bb1-kserve-58545c5c88-t5l49"
Sep 19 04:50:53.575993 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:53.575911    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55ace636-dbb2-4f5d-a26e-56b624107f1f-model-cache\") pod \"rate-test-model-561bb1-kserve-58545c5c88-t5l49\" (UID: \"55ace636-dbb2-4f5d-a26e-56b624107f1f\") " pod="ai-tenant-e2e-shared-a-w3-561bb1/rate-test-model-561bb1-kserve-58545c5c88-t5l49"
Sep 19 04:50:53.575993 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:53.575914    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55ace636-dbb2-4f5d-a26e-56b624107f1f-tmp-dir\") pod \"rate-test-model-561bb1-kserve-58545c5c88-t5l49\" (UID: \"55ace636-dbb2-4f5d-a26e-56b624107f1f\") " pod="ai-tenant-e2e-shared-a-w3-561bb1/rate-test-model-561bb1-kserve-58545c5c88-t5l49"
Sep 19 04:50:53.576064 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:53.576002    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55ace636-dbb2-4f5d-a26e-56b624107f1f-home\") pod \"rate-test-model-561bb1-kserve-58545c5c88-t5l49\" (UID: \"55ace636-dbb2-4f5d-a26e-56b624107f1f\") " pod="ai-tenant-e2e-shared-a-w3-561bb1/rate-test-model-561bb1-kserve-58545c5c88-t5l49"
Sep 19 04:50:53.578170 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:53.578145    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55ace636-dbb2-4f5d-a26e-56b624107f1f-dshm\") pod \"rate-test-model-561bb1-kserve-58545c5c88-t5l49\" (UID: \"55ace636-dbb2-4f5d-a26e-56b624107f1f\") " pod="ai-tenant-e2e-shared-a-w3-561bb1/rate-test-model-561bb1-kserve-58545c5c88-t5l49"
Sep 19 04:50:53.578418 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:53.578395    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55ace636-dbb2-4f5d-a26e-56b624107f1f-tls-certs\") pod \"rate-test-model-561bb1-kserve-58545c5c88-t5l49\" (UID: \"55ace636-dbb2-4f5d-a26e-56b624107f1f\") " pod="ai-tenant-e2e-shared-a-w3-561bb1/rate-test-model-561bb1-kserve-58545c5c88-t5l49"
Sep 19 04:50:53.593919 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:53.593893    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsmjc\" (UniqueName: \"kubernetes.io/projected/55ace636-dbb2-4f5d-a26e-56b624107f1f-kube-api-access-vsmjc\") pod \"rate-test-model-561bb1-kserve-58545c5c88-t5l49\" (UID: \"55ace636-dbb2-4f5d-a26e-56b624107f1f\") " pod="ai-tenant-e2e-shared-a-w3-561bb1/rate-test-model-561bb1-kserve-58545c5c88-t5l49"
Sep 19 04:50:53.754329 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:53.754233    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-561bb1/rate-test-model-561bb1-kserve-58545c5c88-t5l49"
Sep 19 04:50:54.034669 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:54.034642    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-561bb1/rate-test-model-561bb1-kserve-58545c5c88-t5l49"]
Sep 19 04:50:54.036043 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:50:54.036016    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod55ace636_dbb2_4f5d_a26e_56b624107f1f.slice/crio-af4446172e39ac60f92aac6032ee00dd41ad108bc6fc03cb29f2cfb4635e8209 WatchSource:0}: Error finding container af4446172e39ac60f92aac6032ee00dd41ad108bc6fc03cb29f2cfb4635e8209: Status 404 returned error can't find the container with id af4446172e39ac60f92aac6032ee00dd41ad108bc6fc03cb29f2cfb4635e8209
Sep 19 04:50:54.189157 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:54.189123    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-561bb1/rate-test-model-561bb1-kserve-58545c5c88-t5l49" event={"ID":"55ace636-dbb2-4f5d-a26e-56b624107f1f","Type":"ContainerStarted","Data":"2fe9b12bd8cd53a9157b631bde35c766548711a854ac0df1c64622d4096187ee"}
Sep 19 04:50:54.189319 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:54.189173    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-561bb1/rate-test-model-561bb1-kserve-58545c5c88-t5l49" event={"ID":"55ace636-dbb2-4f5d-a26e-56b624107f1f","Type":"ContainerStarted","Data":"af4446172e39ac60f92aac6032ee00dd41ad108bc6fc03cb29f2cfb4635e8209"}
Sep 19 04:50:54.218423 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:54.218372    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-561bb1/rate-test-model-561bb1-kserve-58545c5c88-t5l49" podStartSLOduration=1.218357967 podStartE2EDuration="1.218357967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 04:50:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:50:54.214852275 +0000 UTC m=+2233.397813423" watchObservedRunningTime="2026-09-19 04:50:54.218357967 +0000 UTC m=+2233.401319116"
Sep 19 04:50:54.755208 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:54.755173    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-561bb1/rate-test-model-561bb1-kserve-58545c5c88-t5l49"
Sep 19 04:50:56.092437 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:56.092396    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-561bb1/sub-test-model-561bb1-kserve-755b9c9866-28mqt" podUID="188b3bbc-7ac0-4244-9ee1-87d90652f6ff" containerName="main" containerID="cri-o://e93e8f5a01fe82013923ca874db927baf013a7b5e3de760d282385b67a925e30" gracePeriod=45
Sep 19 04:50:56.349536 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:56.349480    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-561bb1/sub-test-model-561bb1-kserve-755b9c9866-28mqt"
Sep 19 04:50:56.401793 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:56.401755    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/188b3bbc-7ac0-4244-9ee1-87d90652f6ff-home\") pod \"188b3bbc-7ac0-4244-9ee1-87d90652f6ff\" (UID: \"188b3bbc-7ac0-4244-9ee1-87d90652f6ff\") "
Sep 19 04:50:56.401981 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:56.401842    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/188b3bbc-7ac0-4244-9ee1-87d90652f6ff-dshm\") pod \"188b3bbc-7ac0-4244-9ee1-87d90652f6ff\" (UID: \"188b3bbc-7ac0-4244-9ee1-87d90652f6ff\") "
Sep 19 04:50:56.401981 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:56.401877    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/188b3bbc-7ac0-4244-9ee1-87d90652f6ff-tls-certs\") pod \"188b3bbc-7ac0-4244-9ee1-87d90652f6ff\" (UID: \"188b3bbc-7ac0-4244-9ee1-87d90652f6ff\") "
Sep 19 04:50:56.401981 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:56.401903    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/188b3bbc-7ac0-4244-9ee1-87d90652f6ff-tmp-dir\") pod \"188b3bbc-7ac0-4244-9ee1-87d90652f6ff\" (UID: \"188b3bbc-7ac0-4244-9ee1-87d90652f6ff\") "
Sep 19 04:50:56.402512 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:56.402017    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/188b3bbc-7ac0-4244-9ee1-87d90652f6ff-model-cache\") pod \"188b3bbc-7ac0-4244-9ee1-87d90652f6ff\" (UID: \"188b3bbc-7ac0-4244-9ee1-87d90652f6ff\") "
Sep 19 04:50:56.402512 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:56.402044    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pwhpn\" (UniqueName: \"kubernetes.io/projected/188b3bbc-7ac0-4244-9ee1-87d90652f6ff-kube-api-access-pwhpn\") pod \"188b3bbc-7ac0-4244-9ee1-87d90652f6ff\" (UID: \"188b3bbc-7ac0-4244-9ee1-87d90652f6ff\") "
Sep 19 04:50:56.402512 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:56.402068    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/188b3bbc-7ac0-4244-9ee1-87d90652f6ff-home" (OuterVolumeSpecName: "home") pod "188b3bbc-7ac0-4244-9ee1-87d90652f6ff" (UID: "188b3bbc-7ac0-4244-9ee1-87d90652f6ff"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:50:56.402512 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:56.402152    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/188b3bbc-7ac0-4244-9ee1-87d90652f6ff-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "188b3bbc-7ac0-4244-9ee1-87d90652f6ff" (UID: "188b3bbc-7ac0-4244-9ee1-87d90652f6ff"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:50:56.402512 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:56.402253    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/188b3bbc-7ac0-4244-9ee1-87d90652f6ff-model-cache" (OuterVolumeSpecName: "model-cache") pod "188b3bbc-7ac0-4244-9ee1-87d90652f6ff" (UID: "188b3bbc-7ac0-4244-9ee1-87d90652f6ff"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:50:56.402512 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:56.402459    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/188b3bbc-7ac0-4244-9ee1-87d90652f6ff-model-cache\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:50:56.402512 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:56.402479    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/188b3bbc-7ac0-4244-9ee1-87d90652f6ff-home\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:50:56.402512 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:56.402494    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/188b3bbc-7ac0-4244-9ee1-87d90652f6ff-tmp-dir\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:50:56.404287 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:56.404256    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/188b3bbc-7ac0-4244-9ee1-87d90652f6ff-dshm" (OuterVolumeSpecName: "dshm") pod "188b3bbc-7ac0-4244-9ee1-87d90652f6ff" (UID: "188b3bbc-7ac0-4244-9ee1-87d90652f6ff"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:50:56.404287 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:56.404256    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/188b3bbc-7ac0-4244-9ee1-87d90652f6ff-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "188b3bbc-7ac0-4244-9ee1-87d90652f6ff" (UID: "188b3bbc-7ac0-4244-9ee1-87d90652f6ff"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 04:50:56.404439 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:56.404372    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/188b3bbc-7ac0-4244-9ee1-87d90652f6ff-kube-api-access-pwhpn" (OuterVolumeSpecName: "kube-api-access-pwhpn") pod "188b3bbc-7ac0-4244-9ee1-87d90652f6ff" (UID: "188b3bbc-7ac0-4244-9ee1-87d90652f6ff"). InnerVolumeSpecName "kube-api-access-pwhpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 04:50:56.503340 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:56.503304    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwhpn\" (UniqueName: \"kubernetes.io/projected/188b3bbc-7ac0-4244-9ee1-87d90652f6ff-kube-api-access-pwhpn\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:50:56.503340 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:56.503335    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/188b3bbc-7ac0-4244-9ee1-87d90652f6ff-dshm\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:50:56.503340 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:56.503347    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/188b3bbc-7ac0-4244-9ee1-87d90652f6ff-tls-certs\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:50:57.210713 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:57.210656    2579 generic.go:356] "Generic (PLEG): container finished" podID="188b3bbc-7ac0-4244-9ee1-87d90652f6ff" containerID="e93e8f5a01fe82013923ca874db927baf013a7b5e3de760d282385b67a925e30" exitCode=0
Sep 19 04:50:57.211177 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:57.210717    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-561bb1/sub-test-model-561bb1-kserve-755b9c9866-28mqt"
Sep 19 04:50:57.211177 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:57.210740    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-561bb1/sub-test-model-561bb1-kserve-755b9c9866-28mqt" event={"ID":"188b3bbc-7ac0-4244-9ee1-87d90652f6ff","Type":"ContainerDied","Data":"e93e8f5a01fe82013923ca874db927baf013a7b5e3de760d282385b67a925e30"}
Sep 19 04:50:57.211177 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:57.210790    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-561bb1/sub-test-model-561bb1-kserve-755b9c9866-28mqt" event={"ID":"188b3bbc-7ac0-4244-9ee1-87d90652f6ff","Type":"ContainerDied","Data":"f01cf38c76fa67dca17204b28812e9b542259ad23cd9783c440632fdcf5b684e"}
Sep 19 04:50:57.211177 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:57.210815    2579 scope.go:117] "RemoveContainer" containerID="e93e8f5a01fe82013923ca874db927baf013a7b5e3de760d282385b67a925e30"
Sep 19 04:50:57.221110 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:57.221091    2579 scope.go:117] "RemoveContainer" containerID="e93e8f5a01fe82013923ca874db927baf013a7b5e3de760d282385b67a925e30"
Sep 19 04:50:57.221384 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:50:57.221364    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e93e8f5a01fe82013923ca874db927baf013a7b5e3de760d282385b67a925e30\": container with ID starting with e93e8f5a01fe82013923ca874db927baf013a7b5e3de760d282385b67a925e30 not found: ID does not exist" containerID="e93e8f5a01fe82013923ca874db927baf013a7b5e3de760d282385b67a925e30"
Sep 19 04:50:57.221466 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:57.221394    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e93e8f5a01fe82013923ca874db927baf013a7b5e3de760d282385b67a925e30"} err="failed to get container status \"e93e8f5a01fe82013923ca874db927baf013a7b5e3de760d282385b67a925e30\": rpc error: code = NotFound desc = could not find container \"e93e8f5a01fe82013923ca874db927baf013a7b5e3de760d282385b67a925e30\": container with ID starting with e93e8f5a01fe82013923ca874db927baf013a7b5e3de760d282385b67a925e30 not found: ID does not exist"
Sep 19 04:50:57.247076 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:57.247042    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-561bb1/sub-test-model-561bb1-kserve-755b9c9866-28mqt"]
Sep 19 04:50:57.262299 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:57.262263    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-561bb1/sub-test-model-561bb1-kserve-755b9c9866-28mqt"]
Sep 19 04:50:57.352338 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:57.352302    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="188b3bbc-7ac0-4244-9ee1-87d90652f6ff" path="/var/lib/kubelet/pods/188b3bbc-7ac0-4244-9ee1-87d90652f6ff/volumes"
Sep 19 04:50:58.192552 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:58.192502    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-8bbcca/sub-test-model-8bbcca-kserve-bf68fd574-cg6sz" podUID="568ba356-2a61-4f5f-bb23-4d261e3f9c42" containerName="main" containerID="cri-o://e6651953046c8da153b715663241f21957bcc0f70a3c6e9067267cbb8572f37c" gracePeriod=45
Sep 19 04:50:58.444798 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:58.444737    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-8bbcca/sub-test-model-8bbcca-kserve-bf68fd574-cg6sz"
Sep 19 04:50:58.521183 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:58.521151    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pvgkj\" (UniqueName: \"kubernetes.io/projected/568ba356-2a61-4f5f-bb23-4d261e3f9c42-kube-api-access-pvgkj\") pod \"568ba356-2a61-4f5f-bb23-4d261e3f9c42\" (UID: \"568ba356-2a61-4f5f-bb23-4d261e3f9c42\") "
Sep 19 04:50:58.521343 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:58.521201    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/568ba356-2a61-4f5f-bb23-4d261e3f9c42-home\") pod \"568ba356-2a61-4f5f-bb23-4d261e3f9c42\" (UID: \"568ba356-2a61-4f5f-bb23-4d261e3f9c42\") "
Sep 19 04:50:58.521343 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:58.521235    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/568ba356-2a61-4f5f-bb23-4d261e3f9c42-tls-certs\") pod \"568ba356-2a61-4f5f-bb23-4d261e3f9c42\" (UID: \"568ba356-2a61-4f5f-bb23-4d261e3f9c42\") "
Sep 19 04:50:58.521463 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:58.521397    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/568ba356-2a61-4f5f-bb23-4d261e3f9c42-model-cache\") pod \"568ba356-2a61-4f5f-bb23-4d261e3f9c42\" (UID: \"568ba356-2a61-4f5f-bb23-4d261e3f9c42\") "
Sep 19 04:50:58.521524 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:58.521461    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/568ba356-2a61-4f5f-bb23-4d261e3f9c42-dshm\") pod \"568ba356-2a61-4f5f-bb23-4d261e3f9c42\" (UID: \"568ba356-2a61-4f5f-bb23-4d261e3f9c42\") "
Sep 19 04:50:58.521524 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:58.521502    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/568ba356-2a61-4f5f-bb23-4d261e3f9c42-home" (OuterVolumeSpecName: "home") pod "568ba356-2a61-4f5f-bb23-4d261e3f9c42" (UID: "568ba356-2a61-4f5f-bb23-4d261e3f9c42"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:50:58.521644 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:58.521552    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/568ba356-2a61-4f5f-bb23-4d261e3f9c42-tmp-dir\") pod \"568ba356-2a61-4f5f-bb23-4d261e3f9c42\" (UID: \"568ba356-2a61-4f5f-bb23-4d261e3f9c42\") "
Sep 19 04:50:58.521644 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:58.521617    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/568ba356-2a61-4f5f-bb23-4d261e3f9c42-model-cache" (OuterVolumeSpecName: "model-cache") pod "568ba356-2a61-4f5f-bb23-4d261e3f9c42" (UID: "568ba356-2a61-4f5f-bb23-4d261e3f9c42"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:50:58.521912 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:58.521879    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/568ba356-2a61-4f5f-bb23-4d261e3f9c42-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "568ba356-2a61-4f5f-bb23-4d261e3f9c42" (UID: "568ba356-2a61-4f5f-bb23-4d261e3f9c42"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:50:58.521912 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:58.521898    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/568ba356-2a61-4f5f-bb23-4d261e3f9c42-home\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:50:58.522055 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:58.521918    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/568ba356-2a61-4f5f-bb23-4d261e3f9c42-model-cache\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:50:58.523507 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:58.523485    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/568ba356-2a61-4f5f-bb23-4d261e3f9c42-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "568ba356-2a61-4f5f-bb23-4d261e3f9c42" (UID: "568ba356-2a61-4f5f-bb23-4d261e3f9c42"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 04:50:58.523861 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:58.523831    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/568ba356-2a61-4f5f-bb23-4d261e3f9c42-dshm" (OuterVolumeSpecName: "dshm") pod "568ba356-2a61-4f5f-bb23-4d261e3f9c42" (UID: "568ba356-2a61-4f5f-bb23-4d261e3f9c42"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:50:58.523965 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:58.523899    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/568ba356-2a61-4f5f-bb23-4d261e3f9c42-kube-api-access-pvgkj" (OuterVolumeSpecName: "kube-api-access-pvgkj") pod "568ba356-2a61-4f5f-bb23-4d261e3f9c42" (UID: "568ba356-2a61-4f5f-bb23-4d261e3f9c42"). InnerVolumeSpecName "kube-api-access-pvgkj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 04:50:58.623059 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:58.623023    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/568ba356-2a61-4f5f-bb23-4d261e3f9c42-tmp-dir\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:50:58.623059 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:58.623052    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvgkj\" (UniqueName: \"kubernetes.io/projected/568ba356-2a61-4f5f-bb23-4d261e3f9c42-kube-api-access-pvgkj\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:50:58.623059 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:58.623064    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/568ba356-2a61-4f5f-bb23-4d261e3f9c42-tls-certs\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:50:58.623278 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:58.623075    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/568ba356-2a61-4f5f-bb23-4d261e3f9c42-dshm\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:50:59.224161 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:59.224128    2579 generic.go:356] "Generic (PLEG): container finished" podID="568ba356-2a61-4f5f-bb23-4d261e3f9c42" containerID="e6651953046c8da153b715663241f21957bcc0f70a3c6e9067267cbb8572f37c" exitCode=0
Sep 19 04:50:59.224353 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:59.224174    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-8bbcca/sub-test-model-8bbcca-kserve-bf68fd574-cg6sz"
Sep 19 04:50:59.224353 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:59.224175    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-8bbcca/sub-test-model-8bbcca-kserve-bf68fd574-cg6sz" event={"ID":"568ba356-2a61-4f5f-bb23-4d261e3f9c42","Type":"ContainerDied","Data":"e6651953046c8da153b715663241f21957bcc0f70a3c6e9067267cbb8572f37c"}
Sep 19 04:50:59.224353 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:59.224223    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-8bbcca/sub-test-model-8bbcca-kserve-bf68fd574-cg6sz" event={"ID":"568ba356-2a61-4f5f-bb23-4d261e3f9c42","Type":"ContainerDied","Data":"21e8ca59cffeff75fe06db7b8a7047b02963b797ae9f2042f7c1c687ad44e959"}
Sep 19 04:50:59.224353 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:59.224240    2579 scope.go:117] "RemoveContainer" containerID="e6651953046c8da153b715663241f21957bcc0f70a3c6e9067267cbb8572f37c"
Sep 19 04:50:59.234101 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:59.234081    2579 scope.go:117] "RemoveContainer" containerID="e6651953046c8da153b715663241f21957bcc0f70a3c6e9067267cbb8572f37c"
Sep 19 04:50:59.234376 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:50:59.234358    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6651953046c8da153b715663241f21957bcc0f70a3c6e9067267cbb8572f37c\": container with ID starting with e6651953046c8da153b715663241f21957bcc0f70a3c6e9067267cbb8572f37c not found: ID does not exist" containerID="e6651953046c8da153b715663241f21957bcc0f70a3c6e9067267cbb8572f37c"
Sep 19 04:50:59.234442 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:59.234382    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6651953046c8da153b715663241f21957bcc0f70a3c6e9067267cbb8572f37c"} err="failed to get container status \"e6651953046c8da153b715663241f21957bcc0f70a3c6e9067267cbb8572f37c\": rpc error: code = NotFound desc = could not find container \"e6651953046c8da153b715663241f21957bcc0f70a3c6e9067267cbb8572f37c\": container with ID starting with e6651953046c8da153b715663241f21957bcc0f70a3c6e9067267cbb8572f37c not found: ID does not exist"
Sep 19 04:50:59.248844 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:59.248814    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8bbcca/sub-test-model-8bbcca-kserve-bf68fd574-cg6sz"]
Sep 19 04:50:59.254153 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:59.254131    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8bbcca/sub-test-model-8bbcca-kserve-bf68fd574-cg6sz"]
Sep 19 04:50:59.351946 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:50:59.351914    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="568ba356-2a61-4f5f-bb23-4d261e3f9c42" path="/var/lib/kubelet/pods/568ba356-2a61-4f5f-bb23-4d261e3f9c42/volumes"
Sep 19 04:51:03.754953 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:51:03.754842    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-561bb1/rate-test-model-561bb1-kserve-58545c5c88-t5l49"
Sep 19 04:51:03.767965 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:51:03.767938    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-561bb1/rate-test-model-561bb1-kserve-58545c5c88-t5l49"
Sep 19 04:51:04.275937 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:51:04.275904    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-561bb1/rate-test-model-561bb1-kserve-58545c5c88-t5l49"
Sep 19 04:51:33.180733 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:51:33.180686    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-8bbcca/rate-test-model-8bbcca-kserve-66cd6f5684-qh2n5"]
Sep 19 04:51:33.181176 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:51:33.181061    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="188b3bbc-7ac0-4244-9ee1-87d90652f6ff" containerName="main"
Sep 19 04:51:33.181176 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:51:33.181073    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="188b3bbc-7ac0-4244-9ee1-87d90652f6ff" containerName="main"
Sep 19 04:51:33.181176 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:51:33.181091    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="568ba356-2a61-4f5f-bb23-4d261e3f9c42" containerName="main"
Sep 19 04:51:33.181176 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:51:33.181096    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="568ba356-2a61-4f5f-bb23-4d261e3f9c42" containerName="main"
Sep 19 04:51:33.181176 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:51:33.181158    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="188b3bbc-7ac0-4244-9ee1-87d90652f6ff" containerName="main"
Sep 19 04:51:33.181176 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:51:33.181165    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="568ba356-2a61-4f5f-bb23-4d261e3f9c42" containerName="main"
Sep 19 04:51:33.184604 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:51:33.184580    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-8bbcca/rate-test-model-8bbcca-kserve-66cd6f5684-qh2n5"
Sep 19 04:51:33.197339 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:51:33.197316    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-8bbcca\"/\"kube-root-ca.crt\""
Sep 19 04:51:33.198451 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:51:33.198433    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-8bbcca\"/\"default-dockercfg-pv8fk\""
Sep 19 04:51:33.198689 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:51:33.198670    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-8bbcca\"/\"rate-test-model-8bbcca-kserve-self-signed-certs\""
Sep 19 04:51:33.202850 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:51:33.202833    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-8bbcca\"/\"openshift-service-ca.crt\""
Sep 19 04:51:33.218766 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:51:33.218736    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8bbcca/rate-test-model-8bbcca-kserve-66cd6f5684-qh2n5"]
Sep 19 04:51:33.249601 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:51:33.249574    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b88a601-ea99-4db5-b3a0-1fd1240f44a1-dshm\") pod \"rate-test-model-8bbcca-kserve-66cd6f5684-qh2n5\" (UID: \"6b88a601-ea99-4db5-b3a0-1fd1240f44a1\") " pod="ai-tenant-e2e-shared-b-w3-8bbcca/rate-test-model-8bbcca-kserve-66cd6f5684-qh2n5"
Sep 19 04:51:33.249601 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:51:33.249604    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b88a601-ea99-4db5-b3a0-1fd1240f44a1-home\") pod \"rate-test-model-8bbcca-kserve-66cd6f5684-qh2n5\" (UID: \"6b88a601-ea99-4db5-b3a0-1fd1240f44a1\") " pod="ai-tenant-e2e-shared-b-w3-8bbcca/rate-test-model-8bbcca-kserve-66cd6f5684-qh2n5"
Sep 19 04:51:33.249882 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:51:33.249735    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpzj4\" (UniqueName: \"kubernetes.io/projected/6b88a601-ea99-4db5-b3a0-1fd1240f44a1-kube-api-access-zpzj4\") pod \"rate-test-model-8bbcca-kserve-66cd6f5684-qh2n5\" (UID: \"6b88a601-ea99-4db5-b3a0-1fd1240f44a1\") " pod="ai-tenant-e2e-shared-b-w3-8bbcca/rate-test-model-8bbcca-kserve-66cd6f5684-qh2n5"
Sep 19 04:51:33.249882 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:51:33.249778    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b88a601-ea99-4db5-b3a0-1fd1240f44a1-model-cache\") pod \"rate-test-model-8bbcca-kserve-66cd6f5684-qh2n5\" (UID: \"6b88a601-ea99-4db5-b3a0-1fd1240f44a1\") " pod="ai-tenant-e2e-shared-b-w3-8bbcca/rate-test-model-8bbcca-kserve-66cd6f5684-qh2n5"
Sep 19 04:51:33.249882 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:51:33.249855    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b88a601-ea99-4db5-b3a0-1fd1240f44a1-tmp-dir\") pod \"rate-test-model-8bbcca-kserve-66cd6f5684-qh2n5\" (UID: \"6b88a601-ea99-4db5-b3a0-1fd1240f44a1\") " pod="ai-tenant-e2e-shared-b-w3-8bbcca/rate-test-model-8bbcca-kserve-66cd6f5684-qh2n5"
Sep 19 04:51:33.250030 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:51:33.249908    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b88a601-ea99-4db5-b3a0-1fd1240f44a1-tls-certs\") pod \"rate-test-model-8bbcca-kserve-66cd6f5684-qh2n5\" (UID: \"6b88a601-ea99-4db5-b3a0-1fd1240f44a1\") " pod="ai-tenant-e2e-shared-b-w3-8bbcca/rate-test-model-8bbcca-kserve-66cd6f5684-qh2n5"
Sep 19 04:51:33.350480 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:51:33.350445    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b88a601-ea99-4db5-b3a0-1fd1240f44a1-dshm\") pod \"rate-test-model-8bbcca-kserve-66cd6f5684-qh2n5\" (UID: \"6b88a601-ea99-4db5-b3a0-1fd1240f44a1\") " pod="ai-tenant-e2e-shared-b-w3-8bbcca/rate-test-model-8bbcca-kserve-66cd6f5684-qh2n5"
Sep 19 04:51:33.350480 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:51:33.350489    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b88a601-ea99-4db5-b3a0-1fd1240f44a1-home\") pod \"rate-test-model-8bbcca-kserve-66cd6f5684-qh2n5\" (UID: \"6b88a601-ea99-4db5-b3a0-1fd1240f44a1\") " pod="ai-tenant-e2e-shared-b-w3-8bbcca/rate-test-model-8bbcca-kserve-66cd6f5684-qh2n5"
Sep 19 04:51:33.350777 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:51:33.350553    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpzj4\" (UniqueName: \"kubernetes.io/projected/6b88a601-ea99-4db5-b3a0-1fd1240f44a1-kube-api-access-zpzj4\") pod \"rate-test-model-8bbcca-kserve-66cd6f5684-qh2n5\" (UID: \"6b88a601-ea99-4db5-b3a0-1fd1240f44a1\") " pod="ai-tenant-e2e-shared-b-w3-8bbcca/rate-test-model-8bbcca-kserve-66cd6f5684-qh2n5"
Sep 19 04:51:33.350777 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:51:33.350578    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b88a601-ea99-4db5-b3a0-1fd1240f44a1-model-cache\") pod \"rate-test-model-8bbcca-kserve-66cd6f5684-qh2n5\" (UID: \"6b88a601-ea99-4db5-b3a0-1fd1240f44a1\") " pod="ai-tenant-e2e-shared-b-w3-8bbcca/rate-test-model-8bbcca-kserve-66cd6f5684-qh2n5"
Sep 19 04:51:33.350777 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:51:33.350629    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b88a601-ea99-4db5-b3a0-1fd1240f44a1-tmp-dir\") pod \"rate-test-model-8bbcca-kserve-66cd6f5684-qh2n5\" (UID: \"6b88a601-ea99-4db5-b3a0-1fd1240f44a1\") " pod="ai-tenant-e2e-shared-b-w3-8bbcca/rate-test-model-8bbcca-kserve-66cd6f5684-qh2n5"
Sep 19 04:51:33.350777 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:51:33.350673    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b88a601-ea99-4db5-b3a0-1fd1240f44a1-tls-certs\") pod \"rate-test-model-8bbcca-kserve-66cd6f5684-qh2n5\" (UID: \"6b88a601-ea99-4db5-b3a0-1fd1240f44a1\") " pod="ai-tenant-e2e-shared-b-w3-8bbcca/rate-test-model-8bbcca-kserve-66cd6f5684-qh2n5"
Sep 19 04:51:33.351102 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:51:33.351075    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b88a601-ea99-4db5-b3a0-1fd1240f44a1-home\") pod \"rate-test-model-8bbcca-kserve-66cd6f5684-qh2n5\" (UID: \"6b88a601-ea99-4db5-b3a0-1fd1240f44a1\") " pod="ai-tenant-e2e-shared-b-w3-8bbcca/rate-test-model-8bbcca-kserve-66cd6f5684-qh2n5"
Sep 19 04:51:33.351265 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:51:33.351241    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b88a601-ea99-4db5-b3a0-1fd1240f44a1-model-cache\") pod \"rate-test-model-8bbcca-kserve-66cd6f5684-qh2n5\" (UID: \"6b88a601-ea99-4db5-b3a0-1fd1240f44a1\") " pod="ai-tenant-e2e-shared-b-w3-8bbcca/rate-test-model-8bbcca-kserve-66cd6f5684-qh2n5"
Sep 19 04:51:33.351667 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:51:33.351641    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b88a601-ea99-4db5-b3a0-1fd1240f44a1-tmp-dir\") pod \"rate-test-model-8bbcca-kserve-66cd6f5684-qh2n5\" (UID: \"6b88a601-ea99-4db5-b3a0-1fd1240f44a1\") " pod="ai-tenant-e2e-shared-b-w3-8bbcca/rate-test-model-8bbcca-kserve-66cd6f5684-qh2n5"
Sep 19 04:51:33.353313 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:51:33.353290    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b88a601-ea99-4db5-b3a0-1fd1240f44a1-dshm\") pod \"rate-test-model-8bbcca-kserve-66cd6f5684-qh2n5\" (UID: \"6b88a601-ea99-4db5-b3a0-1fd1240f44a1\") " pod="ai-tenant-e2e-shared-b-w3-8bbcca/rate-test-model-8bbcca-kserve-66cd6f5684-qh2n5"
Sep 19 04:51:33.353548 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:51:33.353530    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b88a601-ea99-4db5-b3a0-1fd1240f44a1-tls-certs\") pod \"rate-test-model-8bbcca-kserve-66cd6f5684-qh2n5\" (UID: \"6b88a601-ea99-4db5-b3a0-1fd1240f44a1\") " pod="ai-tenant-e2e-shared-b-w3-8bbcca/rate-test-model-8bbcca-kserve-66cd6f5684-qh2n5"
Sep 19 04:51:33.362329 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:51:33.362299    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpzj4\" (UniqueName: \"kubernetes.io/projected/6b88a601-ea99-4db5-b3a0-1fd1240f44a1-kube-api-access-zpzj4\") pod \"rate-test-model-8bbcca-kserve-66cd6f5684-qh2n5\" (UID: \"6b88a601-ea99-4db5-b3a0-1fd1240f44a1\") " pod="ai-tenant-e2e-shared-b-w3-8bbcca/rate-test-model-8bbcca-kserve-66cd6f5684-qh2n5"
Sep 19 04:51:33.498503 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:51:33.498296    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-8bbcca/rate-test-model-8bbcca-kserve-66cd6f5684-qh2n5"
Sep 19 04:51:33.788352 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:51:33.788316    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8bbcca/rate-test-model-8bbcca-kserve-66cd6f5684-qh2n5"]
Sep 19 04:51:33.789247 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:51:33.789219    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6b88a601_ea99_4db5_b3a0_1fd1240f44a1.slice/crio-cdcd541e099a3273b078f742902679bdd6bc42cde6a0381b7258c531db9bd92d WatchSource:0}: Error finding container cdcd541e099a3273b078f742902679bdd6bc42cde6a0381b7258c531db9bd92d: Status 404 returned error can't find the container with id cdcd541e099a3273b078f742902679bdd6bc42cde6a0381b7258c531db9bd92d
Sep 19 04:51:34.443393 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:51:34.443349    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-8bbcca/rate-test-model-8bbcca-kserve-66cd6f5684-qh2n5" event={"ID":"6b88a601-ea99-4db5-b3a0-1fd1240f44a1","Type":"ContainerStarted","Data":"c1446f9ae01e951314af86376cb9ec406e1dd4aab2eef9b63c8b905ec66e6bac"}
Sep 19 04:51:34.443393 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:51:34.443397    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-8bbcca/rate-test-model-8bbcca-kserve-66cd6f5684-qh2n5" event={"ID":"6b88a601-ea99-4db5-b3a0-1fd1240f44a1","Type":"ContainerStarted","Data":"cdcd541e099a3273b078f742902679bdd6bc42cde6a0381b7258c531db9bd92d"}
Sep 19 04:51:34.465865 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:51:34.465803    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-8bbcca/rate-test-model-8bbcca-kserve-66cd6f5684-qh2n5" podStartSLOduration=1.465782289 podStartE2EDuration="1.465782289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 04:51:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:51:34.465341373 +0000 UTC m=+2273.648302520" watchObservedRunningTime="2026-09-19 04:51:34.465782289 +0000 UTC m=+2273.648743437"
Sep 19 04:51:34.499231 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:51:34.499190    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-8bbcca/rate-test-model-8bbcca-kserve-66cd6f5684-qh2n5"
Sep 19 04:51:41.836525 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:51:41.836482    2579 scope.go:117] "RemoveContainer" containerID="0bbf97b0cbcbb518dc9f44fb032538b478d45d2a03d24fb4c7b9385d568d9012"
Sep 19 04:51:41.845778 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:51:41.845754    2579 scope.go:117] "RemoveContainer" containerID="802d025c0cca96069e1066cb07bccf2bd655ae20d719587dc0dc72262053bcbf"
Sep 19 04:51:43.498897 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:51:43.498851    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-8bbcca/rate-test-model-8bbcca-kserve-66cd6f5684-qh2n5"
Sep 19 04:51:43.511734 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:51:43.511683    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-8bbcca/rate-test-model-8bbcca-kserve-66cd6f5684-qh2n5"
Sep 19 04:51:44.510921 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:51:44.510889    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-8bbcca/rate-test-model-8bbcca-kserve-66cd6f5684-qh2n5"
Sep 19 04:52:21.520483 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:21.520441    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-561bb1/rate-test-model-561bb1-kserve-58545c5c88-t5l49"]
Sep 19 04:52:23.127829 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:23.127791    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8bbcca/rate-test-model-8bbcca-kserve-66cd6f5684-qh2n5"]
Sep 19 04:52:23.893080 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:23.893038    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-cacd12-openshift-default-7c68f65c4-gvfxl"]
Sep 19 04:52:23.904127 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:23.904098    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-cacd12-openshift-default-7c68f65c4-gvfxl"
Sep 19 04:52:23.908791 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:23.908767    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-cacd12-openshift-default-dockercfg-ldrjl\""
Sep 19 04:52:23.919015 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:23.918985    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-cacd12-openshift-default-7c68f65c4-gvfxl"]
Sep 19 04:52:24.027105 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:24.027066    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp27n\" (UniqueName: \"kubernetes.io/projected/7ebcd658-999c-447c-addb-bce60fea8200-kube-api-access-pp27n\") pod \"e2e-ipp-a-cacd12-openshift-default-7c68f65c4-gvfxl\" (UID: \"7ebcd658-999c-447c-addb-bce60fea8200\") " pod="openshift-ingress/e2e-ipp-a-cacd12-openshift-default-7c68f65c4-gvfxl"
Sep 19 04:52:24.027105 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:24.027108    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ebcd658-999c-447c-addb-bce60fea8200-istio-podinfo\") pod \"e2e-ipp-a-cacd12-openshift-default-7c68f65c4-gvfxl\" (UID: \"7ebcd658-999c-447c-addb-bce60fea8200\") " pod="openshift-ingress/e2e-ipp-a-cacd12-openshift-default-7c68f65c4-gvfxl"
Sep 19 04:52:24.027331 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:24.027130    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ebcd658-999c-447c-addb-bce60fea8200-istio-envoy\") pod \"e2e-ipp-a-cacd12-openshift-default-7c68f65c4-gvfxl\" (UID: \"7ebcd658-999c-447c-addb-bce60fea8200\") " pod="openshift-ingress/e2e-ipp-a-cacd12-openshift-default-7c68f65c4-gvfxl"
Sep 19 04:52:24.027331 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:24.027211    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ebcd658-999c-447c-addb-bce60fea8200-workload-certs\") pod \"e2e-ipp-a-cacd12-openshift-default-7c68f65c4-gvfxl\" (UID: \"7ebcd658-999c-447c-addb-bce60fea8200\") " pod="openshift-ingress/e2e-ipp-a-cacd12-openshift-default-7c68f65c4-gvfxl"
Sep 19 04:52:24.027331 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:24.027261    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ebcd658-999c-447c-addb-bce60fea8200-workload-socket\") pod \"e2e-ipp-a-cacd12-openshift-default-7c68f65c4-gvfxl\" (UID: \"7ebcd658-999c-447c-addb-bce60fea8200\") " pod="openshift-ingress/e2e-ipp-a-cacd12-openshift-default-7c68f65c4-gvfxl"
Sep 19 04:52:24.027331 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:24.027280    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ebcd658-999c-447c-addb-bce60fea8200-istio-data\") pod \"e2e-ipp-a-cacd12-openshift-default-7c68f65c4-gvfxl\" (UID: \"7ebcd658-999c-447c-addb-bce60fea8200\") " pod="openshift-ingress/e2e-ipp-a-cacd12-openshift-default-7c68f65c4-gvfxl"
Sep 19 04:52:24.027464 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:24.027355    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ebcd658-999c-447c-addb-bce60fea8200-istio-token\") pod \"e2e-ipp-a-cacd12-openshift-default-7c68f65c4-gvfxl\" (UID: \"7ebcd658-999c-447c-addb-bce60fea8200\") " pod="openshift-ingress/e2e-ipp-a-cacd12-openshift-default-7c68f65c4-gvfxl"
Sep 19 04:52:24.027464 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:24.027384    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ebcd658-999c-447c-addb-bce60fea8200-istiod-ca-cert\") pod \"e2e-ipp-a-cacd12-openshift-default-7c68f65c4-gvfxl\" (UID: \"7ebcd658-999c-447c-addb-bce60fea8200\") " pod="openshift-ingress/e2e-ipp-a-cacd12-openshift-default-7c68f65c4-gvfxl"
Sep 19 04:52:24.027464 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:24.027414    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ebcd658-999c-447c-addb-bce60fea8200-credential-socket\") pod \"e2e-ipp-a-cacd12-openshift-default-7c68f65c4-gvfxl\" (UID: \"7ebcd658-999c-447c-addb-bce60fea8200\") " pod="openshift-ingress/e2e-ipp-a-cacd12-openshift-default-7c68f65c4-gvfxl"
Sep 19 04:52:24.128400 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:24.128360    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ebcd658-999c-447c-addb-bce60fea8200-istio-envoy\") pod \"e2e-ipp-a-cacd12-openshift-default-7c68f65c4-gvfxl\" (UID: \"7ebcd658-999c-447c-addb-bce60fea8200\") " pod="openshift-ingress/e2e-ipp-a-cacd12-openshift-default-7c68f65c4-gvfxl"
Sep 19 04:52:24.128400 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:24.128404    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ebcd658-999c-447c-addb-bce60fea8200-workload-certs\") pod \"e2e-ipp-a-cacd12-openshift-default-7c68f65c4-gvfxl\" (UID: \"7ebcd658-999c-447c-addb-bce60fea8200\") " pod="openshift-ingress/e2e-ipp-a-cacd12-openshift-default-7c68f65c4-gvfxl"
Sep 19 04:52:24.128919 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:24.128438    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ebcd658-999c-447c-addb-bce60fea8200-workload-socket\") pod \"e2e-ipp-a-cacd12-openshift-default-7c68f65c4-gvfxl\" (UID: \"7ebcd658-999c-447c-addb-bce60fea8200\") " pod="openshift-ingress/e2e-ipp-a-cacd12-openshift-default-7c68f65c4-gvfxl"
Sep 19 04:52:24.128919 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:24.128465    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ebcd658-999c-447c-addb-bce60fea8200-istio-data\") pod \"e2e-ipp-a-cacd12-openshift-default-7c68f65c4-gvfxl\" (UID: \"7ebcd658-999c-447c-addb-bce60fea8200\") " pod="openshift-ingress/e2e-ipp-a-cacd12-openshift-default-7c68f65c4-gvfxl"
Sep 19 04:52:24.128919 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:24.128526    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ebcd658-999c-447c-addb-bce60fea8200-istio-token\") pod \"e2e-ipp-a-cacd12-openshift-default-7c68f65c4-gvfxl\" (UID: \"7ebcd658-999c-447c-addb-bce60fea8200\") " pod="openshift-ingress/e2e-ipp-a-cacd12-openshift-default-7c68f65c4-gvfxl"
Sep 19 04:52:24.128919 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:24.128556    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ebcd658-999c-447c-addb-bce60fea8200-istiod-ca-cert\") pod \"e2e-ipp-a-cacd12-openshift-default-7c68f65c4-gvfxl\" (UID: \"7ebcd658-999c-447c-addb-bce60fea8200\") " pod="openshift-ingress/e2e-ipp-a-cacd12-openshift-default-7c68f65c4-gvfxl"
Sep 19 04:52:24.128919 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:24.128584    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ebcd658-999c-447c-addb-bce60fea8200-credential-socket\") pod \"e2e-ipp-a-cacd12-openshift-default-7c68f65c4-gvfxl\" (UID: \"7ebcd658-999c-447c-addb-bce60fea8200\") " pod="openshift-ingress/e2e-ipp-a-cacd12-openshift-default-7c68f65c4-gvfxl"
Sep 19 04:52:24.128919 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:24.128649    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp27n\" (UniqueName: \"kubernetes.io/projected/7ebcd658-999c-447c-addb-bce60fea8200-kube-api-access-pp27n\") pod \"e2e-ipp-a-cacd12-openshift-default-7c68f65c4-gvfxl\" (UID: \"7ebcd658-999c-447c-addb-bce60fea8200\") " pod="openshift-ingress/e2e-ipp-a-cacd12-openshift-default-7c68f65c4-gvfxl"
Sep 19 04:52:24.128919 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:24.128674    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ebcd658-999c-447c-addb-bce60fea8200-istio-podinfo\") pod \"e2e-ipp-a-cacd12-openshift-default-7c68f65c4-gvfxl\" (UID: \"7ebcd658-999c-447c-addb-bce60fea8200\") " pod="openshift-ingress/e2e-ipp-a-cacd12-openshift-default-7c68f65c4-gvfxl"
Sep 19 04:52:24.129267 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:24.128951    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ebcd658-999c-447c-addb-bce60fea8200-workload-socket\") pod \"e2e-ipp-a-cacd12-openshift-default-7c68f65c4-gvfxl\" (UID: \"7ebcd658-999c-447c-addb-bce60fea8200\") " pod="openshift-ingress/e2e-ipp-a-cacd12-openshift-default-7c68f65c4-gvfxl"
Sep 19 04:52:24.129267 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:24.129111    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ebcd658-999c-447c-addb-bce60fea8200-credential-socket\") pod \"e2e-ipp-a-cacd12-openshift-default-7c68f65c4-gvfxl\" (UID: \"7ebcd658-999c-447c-addb-bce60fea8200\") " pod="openshift-ingress/e2e-ipp-a-cacd12-openshift-default-7c68f65c4-gvfxl"
Sep 19 04:52:24.129267 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:24.129127    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ebcd658-999c-447c-addb-bce60fea8200-workload-certs\") pod \"e2e-ipp-a-cacd12-openshift-default-7c68f65c4-gvfxl\" (UID: \"7ebcd658-999c-447c-addb-bce60fea8200\") " pod="openshift-ingress/e2e-ipp-a-cacd12-openshift-default-7c68f65c4-gvfxl"
Sep 19 04:52:24.129267 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:24.129180    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ebcd658-999c-447c-addb-bce60fea8200-istio-data\") pod \"e2e-ipp-a-cacd12-openshift-default-7c68f65c4-gvfxl\" (UID: \"7ebcd658-999c-447c-addb-bce60fea8200\") " pod="openshift-ingress/e2e-ipp-a-cacd12-openshift-default-7c68f65c4-gvfxl"
Sep 19 04:52:24.129517 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:24.129498    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ebcd658-999c-447c-addb-bce60fea8200-istiod-ca-cert\") pod \"e2e-ipp-a-cacd12-openshift-default-7c68f65c4-gvfxl\" (UID: \"7ebcd658-999c-447c-addb-bce60fea8200\") " pod="openshift-ingress/e2e-ipp-a-cacd12-openshift-default-7c68f65c4-gvfxl"
Sep 19 04:52:24.130912 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:24.130890    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ebcd658-999c-447c-addb-bce60fea8200-istio-envoy\") pod \"e2e-ipp-a-cacd12-openshift-default-7c68f65c4-gvfxl\" (UID: \"7ebcd658-999c-447c-addb-bce60fea8200\") " pod="openshift-ingress/e2e-ipp-a-cacd12-openshift-default-7c68f65c4-gvfxl"
Sep 19 04:52:24.131232 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:24.131213    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ebcd658-999c-447c-addb-bce60fea8200-istio-podinfo\") pod \"e2e-ipp-a-cacd12-openshift-default-7c68f65c4-gvfxl\" (UID: \"7ebcd658-999c-447c-addb-bce60fea8200\") " pod="openshift-ingress/e2e-ipp-a-cacd12-openshift-default-7c68f65c4-gvfxl"
Sep 19 04:52:24.143357 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:24.143294    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp27n\" (UniqueName: \"kubernetes.io/projected/7ebcd658-999c-447c-addb-bce60fea8200-kube-api-access-pp27n\") pod \"e2e-ipp-a-cacd12-openshift-default-7c68f65c4-gvfxl\" (UID: \"7ebcd658-999c-447c-addb-bce60fea8200\") " pod="openshift-ingress/e2e-ipp-a-cacd12-openshift-default-7c68f65c4-gvfxl"
Sep 19 04:52:24.151034 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:24.151012    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ebcd658-999c-447c-addb-bce60fea8200-istio-token\") pod \"e2e-ipp-a-cacd12-openshift-default-7c68f65c4-gvfxl\" (UID: \"7ebcd658-999c-447c-addb-bce60fea8200\") " pod="openshift-ingress/e2e-ipp-a-cacd12-openshift-default-7c68f65c4-gvfxl"
Sep 19 04:52:24.214640 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:24.214608    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-cacd12-openshift-default-7c68f65c4-gvfxl"
Sep 19 04:52:24.503605 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:24.503570    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-cacd12-openshift-default-7c68f65c4-gvfxl"]
Sep 19 04:52:24.507988 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:52:24.507961    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ebcd658_999c_447c_addb_bce60fea8200.slice/crio-cfc753617aa0f538aa7a56900bd6edc958940803bfbc74cf383cad7e147a8521 WatchSource:0}: Error finding container cfc753617aa0f538aa7a56900bd6edc958940803bfbc74cf383cad7e147a8521: Status 404 returned error can't find the container with id cfc753617aa0f538aa7a56900bd6edc958940803bfbc74cf383cad7e147a8521
Sep 19 04:52:24.510235 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:24.510173    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 04:52:24.510329 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:24.510272    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 04:52:24.510329 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:24.510299    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 04:52:24.739688 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:24.739652    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-cacd12-openshift-default-7c68f65c4-gvfxl" event={"ID":"7ebcd658-999c-447c-addb-bce60fea8200","Type":"ContainerStarted","Data":"2620574af0c0f280cda3d2e0a331c09a963684e73184b4e73f1e7849877eb7f5"}
Sep 19 04:52:24.739897 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:24.739739    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-cacd12-openshift-default-7c68f65c4-gvfxl" event={"ID":"7ebcd658-999c-447c-addb-bce60fea8200","Type":"ContainerStarted","Data":"cfc753617aa0f538aa7a56900bd6edc958940803bfbc74cf383cad7e147a8521"}
Sep 19 04:52:24.798592 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:24.798473    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-cacd12-openshift-default-7c68f65c4-gvfxl" podStartSLOduration=1.79845058 podStartE2EDuration="1.79845058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 04:52:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:52:24.797358789 +0000 UTC m=+2323.980319940" watchObservedRunningTime="2026-09-19 04:52:24.79845058 +0000 UTC m=+2323.981411729"
Sep 19 04:52:25.215589 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:25.215542    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-cacd12-openshift-default-7c68f65c4-gvfxl"
Sep 19 04:52:25.221562 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:25.221532    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-cacd12-openshift-default-7c68f65c4-gvfxl"
Sep 19 04:52:25.746096 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:25.746061    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-cacd12-openshift-default-7c68f65c4-gvfxl"
Sep 19 04:52:25.747218 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:25.747197    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-cacd12-openshift-default-7c68f65c4-gvfxl"
Sep 19 04:52:29.532003 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:29.531962    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-cacd12-779b59d64b-dqdcv"]
Sep 19 04:52:29.535773 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:29.535753    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cacd12-779b59d64b-dqdcv"
Sep 19 04:52:29.538950 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:29.538926    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-cacd12\""
Sep 19 04:52:29.539088 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:29.538963    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-cacd12-dockercfg-l5pc2\""
Sep 19 04:52:29.551555 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:29.551527    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-cacd12-779b59d64b-dqdcv"]
Sep 19 04:52:29.685224 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:29.685173    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cctsv\" (UniqueName: \"kubernetes.io/projected/669638a2-9a0d-40b3-bb81-558a6f82cb2b-kube-api-access-cctsv\") pod \"payload-pre-processing-e2e-ipp-a-cacd12-779b59d64b-dqdcv\" (UID: \"669638a2-9a0d-40b3-bb81-558a6f82cb2b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cacd12-779b59d64b-dqdcv"
Sep 19 04:52:29.685419 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:29.685282    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/669638a2-9a0d-40b3-bb81-558a6f82cb2b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-cacd12-779b59d64b-dqdcv\" (UID: \"669638a2-9a0d-40b3-bb81-558a6f82cb2b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cacd12-779b59d64b-dqdcv"
Sep 19 04:52:29.786519 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:29.786415    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cctsv\" (UniqueName: \"kubernetes.io/projected/669638a2-9a0d-40b3-bb81-558a6f82cb2b-kube-api-access-cctsv\") pod \"payload-pre-processing-e2e-ipp-a-cacd12-779b59d64b-dqdcv\" (UID: \"669638a2-9a0d-40b3-bb81-558a6f82cb2b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cacd12-779b59d64b-dqdcv"
Sep 19 04:52:29.786519 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:29.786508    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/669638a2-9a0d-40b3-bb81-558a6f82cb2b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-cacd12-779b59d64b-dqdcv\" (UID: \"669638a2-9a0d-40b3-bb81-558a6f82cb2b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cacd12-779b59d64b-dqdcv"
Sep 19 04:52:29.787284 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:29.787260    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/669638a2-9a0d-40b3-bb81-558a6f82cb2b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-cacd12-779b59d64b-dqdcv\" (UID: \"669638a2-9a0d-40b3-bb81-558a6f82cb2b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cacd12-779b59d64b-dqdcv"
Sep 19 04:52:29.799099 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:29.799071    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cctsv\" (UniqueName: \"kubernetes.io/projected/669638a2-9a0d-40b3-bb81-558a6f82cb2b-kube-api-access-cctsv\") pod \"payload-pre-processing-e2e-ipp-a-cacd12-779b59d64b-dqdcv\" (UID: \"669638a2-9a0d-40b3-bb81-558a6f82cb2b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cacd12-779b59d64b-dqdcv"
Sep 19 04:52:29.846934 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:29.846899    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cacd12-779b59d64b-dqdcv"
Sep 19 04:52:30.137868 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:30.137836    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-cacd12-779b59d64b-dqdcv"]
Sep 19 04:52:30.139169 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:52:30.139137    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod669638a2_9a0d_40b3_bb81_558a6f82cb2b.slice/crio-2daa730a680c706cac6ee13a315d402227a1e926ab3a16ae58a21dc10b59a87d WatchSource:0}: Error finding container 2daa730a680c706cac6ee13a315d402227a1e926ab3a16ae58a21dc10b59a87d: Status 404 returned error can't find the container with id 2daa730a680c706cac6ee13a315d402227a1e926ab3a16ae58a21dc10b59a87d
Sep 19 04:52:30.778591 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:30.778555    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cacd12-779b59d64b-dqdcv" event={"ID":"669638a2-9a0d-40b3-bb81-558a6f82cb2b","Type":"ContainerStarted","Data":"e4300702d2382a4c84a145b095afc1a8decc81c545b69d7694c2693ebb5857ca"}
Sep 19 04:52:30.779093 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:30.778621    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cacd12-779b59d64b-dqdcv"
Sep 19 04:52:30.779093 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:30.778646    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cacd12-779b59d64b-dqdcv" event={"ID":"669638a2-9a0d-40b3-bb81-558a6f82cb2b","Type":"ContainerStarted","Data":"2daa730a680c706cac6ee13a315d402227a1e926ab3a16ae58a21dc10b59a87d"}
Sep 19 04:52:30.800151 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:30.800087    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cacd12-779b59d64b-dqdcv" podStartSLOduration=1.800062456 podStartE2EDuration="1.800062456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 04:52:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:52:30.798007268 +0000 UTC m=+2329.980968419" watchObservedRunningTime="2026-09-19 04:52:30.800062456 +0000 UTC m=+2329.983023606"
Sep 19 04:52:36.540573 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:36.540461    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-561bb1/rate-test-model-561bb1-kserve-58545c5c88-t5l49" podUID="55ace636-dbb2-4f5d-a26e-56b624107f1f" containerName="main" containerID="cri-o://2fe9b12bd8cd53a9157b631bde35c766548711a854ac0df1c64622d4096187ee" gracePeriod=45
Sep 19 04:52:36.802736 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:36.802647    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-561bb1/rate-test-model-561bb1-kserve-58545c5c88-t5l49"
Sep 19 04:52:36.822177 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:36.822137    2579 generic.go:356] "Generic (PLEG): container finished" podID="55ace636-dbb2-4f5d-a26e-56b624107f1f" containerID="2fe9b12bd8cd53a9157b631bde35c766548711a854ac0df1c64622d4096187ee" exitCode=0
Sep 19 04:52:36.822177 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:36.822181    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-561bb1/rate-test-model-561bb1-kserve-58545c5c88-t5l49"
Sep 19 04:52:36.822442 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:36.822220    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-561bb1/rate-test-model-561bb1-kserve-58545c5c88-t5l49" event={"ID":"55ace636-dbb2-4f5d-a26e-56b624107f1f","Type":"ContainerDied","Data":"2fe9b12bd8cd53a9157b631bde35c766548711a854ac0df1c64622d4096187ee"}
Sep 19 04:52:36.822442 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:36.822278    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-561bb1/rate-test-model-561bb1-kserve-58545c5c88-t5l49" event={"ID":"55ace636-dbb2-4f5d-a26e-56b624107f1f","Type":"ContainerDied","Data":"af4446172e39ac60f92aac6032ee00dd41ad108bc6fc03cb29f2cfb4635e8209"}
Sep 19 04:52:36.822442 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:36.822302    2579 scope.go:117] "RemoveContainer" containerID="2fe9b12bd8cd53a9157b631bde35c766548711a854ac0df1c64622d4096187ee"
Sep 19 04:52:36.833127 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:36.833102    2579 scope.go:117] "RemoveContainer" containerID="2fe9b12bd8cd53a9157b631bde35c766548711a854ac0df1c64622d4096187ee"
Sep 19 04:52:36.833455 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:52:36.833433    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fe9b12bd8cd53a9157b631bde35c766548711a854ac0df1c64622d4096187ee\": container with ID starting with 2fe9b12bd8cd53a9157b631bde35c766548711a854ac0df1c64622d4096187ee not found: ID does not exist" containerID="2fe9b12bd8cd53a9157b631bde35c766548711a854ac0df1c64622d4096187ee"
Sep 19 04:52:36.833556 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:36.833465    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fe9b12bd8cd53a9157b631bde35c766548711a854ac0df1c64622d4096187ee"} err="failed to get container status \"2fe9b12bd8cd53a9157b631bde35c766548711a854ac0df1c64622d4096187ee\": rpc error: code = NotFound desc = could not find container \"2fe9b12bd8cd53a9157b631bde35c766548711a854ac0df1c64622d4096187ee\": container with ID starting with 2fe9b12bd8cd53a9157b631bde35c766548711a854ac0df1c64622d4096187ee not found: ID does not exist"
Sep 19 04:52:36.957289 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:36.957258    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55ace636-dbb2-4f5d-a26e-56b624107f1f-home\") pod \"55ace636-dbb2-4f5d-a26e-56b624107f1f\" (UID: \"55ace636-dbb2-4f5d-a26e-56b624107f1f\") "
Sep 19 04:52:36.957459 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:36.957354    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55ace636-dbb2-4f5d-a26e-56b624107f1f-tmp-dir\") pod \"55ace636-dbb2-4f5d-a26e-56b624107f1f\" (UID: \"55ace636-dbb2-4f5d-a26e-56b624107f1f\") "
Sep 19 04:52:36.957459 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:36.957376    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55ace636-dbb2-4f5d-a26e-56b624107f1f-tls-certs\") pod \"55ace636-dbb2-4f5d-a26e-56b624107f1f\" (UID: \"55ace636-dbb2-4f5d-a26e-56b624107f1f\") "
Sep 19 04:52:36.957459 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:36.957390    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55ace636-dbb2-4f5d-a26e-56b624107f1f-dshm\") pod \"55ace636-dbb2-4f5d-a26e-56b624107f1f\" (UID: \"55ace636-dbb2-4f5d-a26e-56b624107f1f\") "
Sep 19 04:52:36.957459 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:36.957413    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vsmjc\" (UniqueName: \"kubernetes.io/projected/55ace636-dbb2-4f5d-a26e-56b624107f1f-kube-api-access-vsmjc\") pod \"55ace636-dbb2-4f5d-a26e-56b624107f1f\" (UID: \"55ace636-dbb2-4f5d-a26e-56b624107f1f\") "
Sep 19 04:52:36.957459 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:36.957439    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55ace636-dbb2-4f5d-a26e-56b624107f1f-model-cache\") pod \"55ace636-dbb2-4f5d-a26e-56b624107f1f\" (UID: \"55ace636-dbb2-4f5d-a26e-56b624107f1f\") "
Sep 19 04:52:36.957754 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:36.957496    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55ace636-dbb2-4f5d-a26e-56b624107f1f-home" (OuterVolumeSpecName: "home") pod "55ace636-dbb2-4f5d-a26e-56b624107f1f" (UID: "55ace636-dbb2-4f5d-a26e-56b624107f1f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:52:36.957754 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:36.957651    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55ace636-dbb2-4f5d-a26e-56b624107f1f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "55ace636-dbb2-4f5d-a26e-56b624107f1f" (UID: "55ace636-dbb2-4f5d-a26e-56b624107f1f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:52:36.957882 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:36.957802    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55ace636-dbb2-4f5d-a26e-56b624107f1f-model-cache" (OuterVolumeSpecName: "model-cache") pod "55ace636-dbb2-4f5d-a26e-56b624107f1f" (UID: "55ace636-dbb2-4f5d-a26e-56b624107f1f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:52:36.957882 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:36.957858    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55ace636-dbb2-4f5d-a26e-56b624107f1f-tmp-dir\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:52:36.957882 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:36.957871    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55ace636-dbb2-4f5d-a26e-56b624107f1f-home\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:52:36.959679 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:36.959651    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55ace636-dbb2-4f5d-a26e-56b624107f1f-kube-api-access-vsmjc" (OuterVolumeSpecName: "kube-api-access-vsmjc") pod "55ace636-dbb2-4f5d-a26e-56b624107f1f" (UID: "55ace636-dbb2-4f5d-a26e-56b624107f1f"). InnerVolumeSpecName "kube-api-access-vsmjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 04:52:36.959799 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:36.959681    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55ace636-dbb2-4f5d-a26e-56b624107f1f-dshm" (OuterVolumeSpecName: "dshm") pod "55ace636-dbb2-4f5d-a26e-56b624107f1f" (UID: "55ace636-dbb2-4f5d-a26e-56b624107f1f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:52:36.959799 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:36.959681    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55ace636-dbb2-4f5d-a26e-56b624107f1f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "55ace636-dbb2-4f5d-a26e-56b624107f1f" (UID: "55ace636-dbb2-4f5d-a26e-56b624107f1f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 04:52:37.058535 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:37.058452    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55ace636-dbb2-4f5d-a26e-56b624107f1f-tls-certs\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:52:37.058535 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:37.058480    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55ace636-dbb2-4f5d-a26e-56b624107f1f-dshm\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:52:37.058535 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:37.058492    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsmjc\" (UniqueName: \"kubernetes.io/projected/55ace636-dbb2-4f5d-a26e-56b624107f1f-kube-api-access-vsmjc\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:52:37.058535 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:37.058506    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55ace636-dbb2-4f5d-a26e-56b624107f1f-model-cache\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:52:37.156972 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:37.156931    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-561bb1/rate-test-model-561bb1-kserve-58545c5c88-t5l49"]
Sep 19 04:52:37.161956 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:37.161930    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-561bb1/rate-test-model-561bb1-kserve-58545c5c88-t5l49"]
Sep 19 04:52:37.351518 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:37.351442    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55ace636-dbb2-4f5d-a26e-56b624107f1f" path="/var/lib/kubelet/pods/55ace636-dbb2-4f5d-a26e-56b624107f1f/volumes"
Sep 19 04:52:38.147289 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:38.147246    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-8bbcca/rate-test-model-8bbcca-kserve-66cd6f5684-qh2n5" podUID="6b88a601-ea99-4db5-b3a0-1fd1240f44a1" containerName="main" containerID="cri-o://c1446f9ae01e951314af86376cb9ec406e1dd4aab2eef9b63c8b905ec66e6bac" gracePeriod=45
Sep 19 04:52:38.445004 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:38.444977    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-8bbcca/rate-test-model-8bbcca-kserve-66cd6f5684-qh2n5"
Sep 19 04:52:38.571811 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:38.571775    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b88a601-ea99-4db5-b3a0-1fd1240f44a1-home\") pod \"6b88a601-ea99-4db5-b3a0-1fd1240f44a1\" (UID: \"6b88a601-ea99-4db5-b3a0-1fd1240f44a1\") "
Sep 19 04:52:38.572006 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:38.571825    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b88a601-ea99-4db5-b3a0-1fd1240f44a1-tls-certs\") pod \"6b88a601-ea99-4db5-b3a0-1fd1240f44a1\" (UID: \"6b88a601-ea99-4db5-b3a0-1fd1240f44a1\") "
Sep 19 04:52:38.572006 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:38.571866    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zpzj4\" (UniqueName: \"kubernetes.io/projected/6b88a601-ea99-4db5-b3a0-1fd1240f44a1-kube-api-access-zpzj4\") pod \"6b88a601-ea99-4db5-b3a0-1fd1240f44a1\" (UID: \"6b88a601-ea99-4db5-b3a0-1fd1240f44a1\") "
Sep 19 04:52:38.572006 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:38.571886    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b88a601-ea99-4db5-b3a0-1fd1240f44a1-model-cache\") pod \"6b88a601-ea99-4db5-b3a0-1fd1240f44a1\" (UID: \"6b88a601-ea99-4db5-b3a0-1fd1240f44a1\") "
Sep 19 04:52:38.572006 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:38.571941    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b88a601-ea99-4db5-b3a0-1fd1240f44a1-dshm\") pod \"6b88a601-ea99-4db5-b3a0-1fd1240f44a1\" (UID: \"6b88a601-ea99-4db5-b3a0-1fd1240f44a1\") "
Sep 19 04:52:38.572006 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:38.571988    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b88a601-ea99-4db5-b3a0-1fd1240f44a1-tmp-dir\") pod \"6b88a601-ea99-4db5-b3a0-1fd1240f44a1\" (UID: \"6b88a601-ea99-4db5-b3a0-1fd1240f44a1\") "
Sep 19 04:52:38.572268 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:38.572051    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b88a601-ea99-4db5-b3a0-1fd1240f44a1-home" (OuterVolumeSpecName: "home") pod "6b88a601-ea99-4db5-b3a0-1fd1240f44a1" (UID: "6b88a601-ea99-4db5-b3a0-1fd1240f44a1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:52:38.572268 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:38.572176    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b88a601-ea99-4db5-b3a0-1fd1240f44a1-model-cache" (OuterVolumeSpecName: "model-cache") pod "6b88a601-ea99-4db5-b3a0-1fd1240f44a1" (UID: "6b88a601-ea99-4db5-b3a0-1fd1240f44a1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:52:38.572268 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:38.572233    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b88a601-ea99-4db5-b3a0-1fd1240f44a1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6b88a601-ea99-4db5-b3a0-1fd1240f44a1" (UID: "6b88a601-ea99-4db5-b3a0-1fd1240f44a1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:52:38.572392 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:38.572322    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b88a601-ea99-4db5-b3a0-1fd1240f44a1-tmp-dir\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:52:38.572392 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:38.572334    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b88a601-ea99-4db5-b3a0-1fd1240f44a1-home\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:52:38.572392 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:38.572342    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b88a601-ea99-4db5-b3a0-1fd1240f44a1-model-cache\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:52:38.574298 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:38.574267    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b88a601-ea99-4db5-b3a0-1fd1240f44a1-dshm" (OuterVolumeSpecName: "dshm") pod "6b88a601-ea99-4db5-b3a0-1fd1240f44a1" (UID: "6b88a601-ea99-4db5-b3a0-1fd1240f44a1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:52:38.574603 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:38.574575    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b88a601-ea99-4db5-b3a0-1fd1240f44a1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6b88a601-ea99-4db5-b3a0-1fd1240f44a1" (UID: "6b88a601-ea99-4db5-b3a0-1fd1240f44a1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 04:52:38.574603 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:38.574578    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b88a601-ea99-4db5-b3a0-1fd1240f44a1-kube-api-access-zpzj4" (OuterVolumeSpecName: "kube-api-access-zpzj4") pod "6b88a601-ea99-4db5-b3a0-1fd1240f44a1" (UID: "6b88a601-ea99-4db5-b3a0-1fd1240f44a1"). InnerVolumeSpecName "kube-api-access-zpzj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 04:52:38.673870 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:38.673832    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b88a601-ea99-4db5-b3a0-1fd1240f44a1-tls-certs\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:52:38.673870 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:38.673863    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpzj4\" (UniqueName: \"kubernetes.io/projected/6b88a601-ea99-4db5-b3a0-1fd1240f44a1-kube-api-access-zpzj4\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:52:38.673870 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:38.673872    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b88a601-ea99-4db5-b3a0-1fd1240f44a1-dshm\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:52:38.839182 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:38.839087    2579 generic.go:356] "Generic (PLEG): container finished" podID="6b88a601-ea99-4db5-b3a0-1fd1240f44a1" containerID="c1446f9ae01e951314af86376cb9ec406e1dd4aab2eef9b63c8b905ec66e6bac" exitCode=0
Sep 19 04:52:38.839182 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:38.839132    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-8bbcca/rate-test-model-8bbcca-kserve-66cd6f5684-qh2n5"
Sep 19 04:52:38.839182 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:38.839167    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-8bbcca/rate-test-model-8bbcca-kserve-66cd6f5684-qh2n5" event={"ID":"6b88a601-ea99-4db5-b3a0-1fd1240f44a1","Type":"ContainerDied","Data":"c1446f9ae01e951314af86376cb9ec406e1dd4aab2eef9b63c8b905ec66e6bac"}
Sep 19 04:52:38.839413 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:38.839207    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-8bbcca/rate-test-model-8bbcca-kserve-66cd6f5684-qh2n5" event={"ID":"6b88a601-ea99-4db5-b3a0-1fd1240f44a1","Type":"ContainerDied","Data":"cdcd541e099a3273b078f742902679bdd6bc42cde6a0381b7258c531db9bd92d"}
Sep 19 04:52:38.839413 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:38.839225    2579 scope.go:117] "RemoveContainer" containerID="c1446f9ae01e951314af86376cb9ec406e1dd4aab2eef9b63c8b905ec66e6bac"
Sep 19 04:52:38.849869 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:38.849854    2579 scope.go:117] "RemoveContainer" containerID="c1446f9ae01e951314af86376cb9ec406e1dd4aab2eef9b63c8b905ec66e6bac"
Sep 19 04:52:38.850115 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:52:38.850099    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1446f9ae01e951314af86376cb9ec406e1dd4aab2eef9b63c8b905ec66e6bac\": container with ID starting with c1446f9ae01e951314af86376cb9ec406e1dd4aab2eef9b63c8b905ec66e6bac not found: ID does not exist" containerID="c1446f9ae01e951314af86376cb9ec406e1dd4aab2eef9b63c8b905ec66e6bac"
Sep 19 04:52:38.850164 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:38.850122    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1446f9ae01e951314af86376cb9ec406e1dd4aab2eef9b63c8b905ec66e6bac"} err="failed to get container status \"c1446f9ae01e951314af86376cb9ec406e1dd4aab2eef9b63c8b905ec66e6bac\": rpc error: code = NotFound desc = could not find container \"c1446f9ae01e951314af86376cb9ec406e1dd4aab2eef9b63c8b905ec66e6bac\": container with ID starting with c1446f9ae01e951314af86376cb9ec406e1dd4aab2eef9b63c8b905ec66e6bac not found: ID does not exist"
Sep 19 04:52:38.862677 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:38.862647    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8bbcca/rate-test-model-8bbcca-kserve-66cd6f5684-qh2n5"]
Sep 19 04:52:38.866970 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:38.866948    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8bbcca/rate-test-model-8bbcca-kserve-66cd6f5684-qh2n5"]
Sep 19 04:52:39.351539 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:39.351500    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b88a601-ea99-4db5-b3a0-1fd1240f44a1" path="/var/lib/kubelet/pods/6b88a601-ea99-4db5-b3a0-1fd1240f44a1/volumes"
Sep 19 04:52:41.789931 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:41.789891    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cacd12-779b59d64b-dqdcv"
Sep 19 04:52:46.887662 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:46.887629    2579 generic.go:356] "Generic (PLEG): container finished" podID="7ebcd658-999c-447c-addb-bce60fea8200" containerID="2620574af0c0f280cda3d2e0a331c09a963684e73184b4e73f1e7849877eb7f5" exitCode=0
Sep 19 04:52:46.887662 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:46.887657    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-cacd12-openshift-default-7c68f65c4-gvfxl" event={"ID":"7ebcd658-999c-447c-addb-bce60fea8200","Type":"ContainerDied","Data":"2620574af0c0f280cda3d2e0a331c09a963684e73184b4e73f1e7849877eb7f5"}
Sep 19 04:52:46.888224 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:46.888103    2579 scope.go:117] "RemoveContainer" containerID="2620574af0c0f280cda3d2e0a331c09a963684e73184b4e73f1e7849877eb7f5"
Sep 19 04:52:46.889253 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:46.889225    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 04:52:46.889355 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:46.889309    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 04:52:46.889422 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:46.889360    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 04:52:47.899933 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:47.899897    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-cacd12-openshift-default-7c68f65c4-gvfxl" event={"ID":"7ebcd658-999c-447c-addb-bce60fea8200","Type":"ContainerStarted","Data":"464ebedead1ee3481dd28315845b8b6cc4a73bf963aad730aa7e9196b7b7ea7f"}
Sep 19 04:52:48.215505 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:48.215469    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-cacd12-openshift-default-7c68f65c4-gvfxl"
Sep 19 04:52:48.215750 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:48.215731    2579 patch_prober.go:28] interesting pod/e2e-ipp-a-cacd12-openshift-default-7c68f65c4-gvfxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Sep 19 04:52:48.215819 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:48.215802    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-cacd12-openshift-default-7c68f65c4-gvfxl" podUID="7ebcd658-999c-447c-addb-bce60fea8200" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Sep 19 04:52:49.215961 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:49.215924    2579 patch_prober.go:28] interesting pod/e2e-ipp-a-cacd12-openshift-default-7c68f65c4-gvfxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Sep 19 04:52:49.216365 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:49.215986    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-cacd12-openshift-default-7c68f65c4-gvfxl" podUID="7ebcd658-999c-447c-addb-bce60fea8200" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Sep 19 04:52:50.215350 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:50.215308    2579 patch_prober.go:28] interesting pod/e2e-ipp-a-cacd12-openshift-default-7c68f65c4-gvfxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Sep 19 04:52:50.215551 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:50.215374    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-cacd12-openshift-default-7c68f65c4-gvfxl" podUID="7ebcd658-999c-447c-addb-bce60fea8200" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Sep 19 04:52:51.216068 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:51.216032    2579 patch_prober.go:28] interesting pod/e2e-ipp-a-cacd12-openshift-default-7c68f65c4-gvfxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Sep 19 04:52:51.216535 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:51.216090    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-cacd12-openshift-default-7c68f65c4-gvfxl" podUID="7ebcd658-999c-447c-addb-bce60fea8200" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Sep 19 04:52:52.215126 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:52.215090    2579 patch_prober.go:28] interesting pod/e2e-ipp-a-cacd12-openshift-default-7c68f65c4-gvfxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Sep 19 04:52:52.215315 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:52.215153    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-cacd12-openshift-default-7c68f65c4-gvfxl" podUID="7ebcd658-999c-447c-addb-bce60fea8200" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Sep 19 04:52:52.898441 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:52.898396    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-618e62-openshift-default-59dbc788bd-9qtw7"]
Sep 19 04:52:52.898955 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:52.898841    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b88a601-ea99-4db5-b3a0-1fd1240f44a1" containerName="main"
Sep 19 04:52:52.898955 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:52.898854    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b88a601-ea99-4db5-b3a0-1fd1240f44a1" containerName="main"
Sep 19 04:52:52.898955 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:52.898878    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55ace636-dbb2-4f5d-a26e-56b624107f1f" containerName="main"
Sep 19 04:52:52.898955 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:52.898883    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="55ace636-dbb2-4f5d-a26e-56b624107f1f" containerName="main"
Sep 19 04:52:52.898955 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:52.898949    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b88a601-ea99-4db5-b3a0-1fd1240f44a1" containerName="main"
Sep 19 04:52:52.898955 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:52.898960    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="55ace636-dbb2-4f5d-a26e-56b624107f1f" containerName="main"
Sep 19 04:52:52.902295 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:52.902273    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-618e62-openshift-default-59dbc788bd-9qtw7"
Sep 19 04:52:52.905417 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:52.905393    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-618e62-openshift-default-dockercfg-nmvr6\""
Sep 19 04:52:52.926639 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:52.926342    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-618e62-openshift-default-59dbc788bd-9qtw7"]
Sep 19 04:52:52.927335 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:52.927309    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdd4n\" (UniqueName: \"kubernetes.io/projected/c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7-kube-api-access-bdd4n\") pod \"e2e-ipp-b-618e62-openshift-default-59dbc788bd-9qtw7\" (UID: \"c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7\") " pod="openshift-ingress/e2e-ipp-b-618e62-openshift-default-59dbc788bd-9qtw7"
Sep 19 04:52:52.927523 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:52.927505    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7-istio-token\") pod \"e2e-ipp-b-618e62-openshift-default-59dbc788bd-9qtw7\" (UID: \"c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7\") " pod="openshift-ingress/e2e-ipp-b-618e62-openshift-default-59dbc788bd-9qtw7"
Sep 19 04:52:52.927670 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:52.927653    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7-workload-certs\") pod \"e2e-ipp-b-618e62-openshift-default-59dbc788bd-9qtw7\" (UID: \"c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7\") " pod="openshift-ingress/e2e-ipp-b-618e62-openshift-default-59dbc788bd-9qtw7"
Sep 19 04:52:52.927811 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:52.927795    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7-istio-envoy\") pod \"e2e-ipp-b-618e62-openshift-default-59dbc788bd-9qtw7\" (UID: \"c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7\") " pod="openshift-ingress/e2e-ipp-b-618e62-openshift-default-59dbc788bd-9qtw7"
Sep 19 04:52:52.927930 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:52.927916    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7-istio-podinfo\") pod \"e2e-ipp-b-618e62-openshift-default-59dbc788bd-9qtw7\" (UID: \"c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7\") " pod="openshift-ingress/e2e-ipp-b-618e62-openshift-default-59dbc788bd-9qtw7"
Sep 19 04:52:52.928033 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:52.928020    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7-istio-data\") pod \"e2e-ipp-b-618e62-openshift-default-59dbc788bd-9qtw7\" (UID: \"c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7\") " pod="openshift-ingress/e2e-ipp-b-618e62-openshift-default-59dbc788bd-9qtw7"
Sep 19 04:52:52.928149 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:52.928137    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7-istiod-ca-cert\") pod \"e2e-ipp-b-618e62-openshift-default-59dbc788bd-9qtw7\" (UID: \"c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7\") " pod="openshift-ingress/e2e-ipp-b-618e62-openshift-default-59dbc788bd-9qtw7"
Sep 19 04:52:52.931872 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:52.931844    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7-credential-socket\") pod \"e2e-ipp-b-618e62-openshift-default-59dbc788bd-9qtw7\" (UID: \"c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7\") " pod="openshift-ingress/e2e-ipp-b-618e62-openshift-default-59dbc788bd-9qtw7"
Sep 19 04:52:52.932112 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:52.932091    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7-workload-socket\") pod \"e2e-ipp-b-618e62-openshift-default-59dbc788bd-9qtw7\" (UID: \"c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7\") " pod="openshift-ingress/e2e-ipp-b-618e62-openshift-default-59dbc788bd-9qtw7"
Sep 19 04:52:53.033235 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:53.033192    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7-istio-data\") pod \"e2e-ipp-b-618e62-openshift-default-59dbc788bd-9qtw7\" (UID: \"c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7\") " pod="openshift-ingress/e2e-ipp-b-618e62-openshift-default-59dbc788bd-9qtw7"
Sep 19 04:52:53.033235 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:53.033236    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7-istiod-ca-cert\") pod \"e2e-ipp-b-618e62-openshift-default-59dbc788bd-9qtw7\" (UID: \"c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7\") " pod="openshift-ingress/e2e-ipp-b-618e62-openshift-default-59dbc788bd-9qtw7"
Sep 19 04:52:53.033506 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:53.033259    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7-credential-socket\") pod \"e2e-ipp-b-618e62-openshift-default-59dbc788bd-9qtw7\" (UID: \"c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7\") " pod="openshift-ingress/e2e-ipp-b-618e62-openshift-default-59dbc788bd-9qtw7"
Sep 19 04:52:53.033506 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:53.033327    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7-workload-socket\") pod \"e2e-ipp-b-618e62-openshift-default-59dbc788bd-9qtw7\" (UID: \"c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7\") " pod="openshift-ingress/e2e-ipp-b-618e62-openshift-default-59dbc788bd-9qtw7"
Sep 19 04:52:53.033506 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:53.033369    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdd4n\" (UniqueName: \"kubernetes.io/projected/c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7-kube-api-access-bdd4n\") pod \"e2e-ipp-b-618e62-openshift-default-59dbc788bd-9qtw7\" (UID: \"c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7\") " pod="openshift-ingress/e2e-ipp-b-618e62-openshift-default-59dbc788bd-9qtw7"
Sep 19 04:52:53.033506 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:53.033401    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7-istio-token\") pod \"e2e-ipp-b-618e62-openshift-default-59dbc788bd-9qtw7\" (UID: \"c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7\") " pod="openshift-ingress/e2e-ipp-b-618e62-openshift-default-59dbc788bd-9qtw7"
Sep 19 04:52:53.033506 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:53.033441    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7-workload-certs\") pod \"e2e-ipp-b-618e62-openshift-default-59dbc788bd-9qtw7\" (UID: \"c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7\") " pod="openshift-ingress/e2e-ipp-b-618e62-openshift-default-59dbc788bd-9qtw7"
Sep 19 04:52:53.033506 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:53.033463    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7-istio-envoy\") pod \"e2e-ipp-b-618e62-openshift-default-59dbc788bd-9qtw7\" (UID: \"c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7\") " pod="openshift-ingress/e2e-ipp-b-618e62-openshift-default-59dbc788bd-9qtw7"
Sep 19 04:52:53.033506 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:53.033488    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7-istio-podinfo\") pod \"e2e-ipp-b-618e62-openshift-default-59dbc788bd-9qtw7\" (UID: \"c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7\") " pod="openshift-ingress/e2e-ipp-b-618e62-openshift-default-59dbc788bd-9qtw7"
Sep 19 04:52:53.034109 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:53.033675    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7-istio-data\") pod \"e2e-ipp-b-618e62-openshift-default-59dbc788bd-9qtw7\" (UID: \"c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7\") " pod="openshift-ingress/e2e-ipp-b-618e62-openshift-default-59dbc788bd-9qtw7"
Sep 19 04:52:53.034109 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:53.033985    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7-credential-socket\") pod \"e2e-ipp-b-618e62-openshift-default-59dbc788bd-9qtw7\" (UID: \"c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7\") " pod="openshift-ingress/e2e-ipp-b-618e62-openshift-default-59dbc788bd-9qtw7"
Sep 19 04:52:53.034109 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:53.034004    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7-workload-certs\") pod \"e2e-ipp-b-618e62-openshift-default-59dbc788bd-9qtw7\" (UID: \"c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7\") " pod="openshift-ingress/e2e-ipp-b-618e62-openshift-default-59dbc788bd-9qtw7"
Sep 19 04:52:53.034109 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:53.034093    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7-istiod-ca-cert\") pod \"e2e-ipp-b-618e62-openshift-default-59dbc788bd-9qtw7\" (UID: \"c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7\") " pod="openshift-ingress/e2e-ipp-b-618e62-openshift-default-59dbc788bd-9qtw7"
Sep 19 04:52:53.034398 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:53.034378    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7-workload-socket\") pod \"e2e-ipp-b-618e62-openshift-default-59dbc788bd-9qtw7\" (UID: \"c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7\") " pod="openshift-ingress/e2e-ipp-b-618e62-openshift-default-59dbc788bd-9qtw7"
Sep 19 04:52:53.036759 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:53.036734    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7-istio-envoy\") pod \"e2e-ipp-b-618e62-openshift-default-59dbc788bd-9qtw7\" (UID: \"c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7\") " pod="openshift-ingress/e2e-ipp-b-618e62-openshift-default-59dbc788bd-9qtw7"
Sep 19 04:52:53.036875 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:53.036855    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7-istio-podinfo\") pod \"e2e-ipp-b-618e62-openshift-default-59dbc788bd-9qtw7\" (UID: \"c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7\") " pod="openshift-ingress/e2e-ipp-b-618e62-openshift-default-59dbc788bd-9qtw7"
Sep 19 04:52:53.043230 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:53.043199    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdd4n\" (UniqueName: \"kubernetes.io/projected/c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7-kube-api-access-bdd4n\") pod \"e2e-ipp-b-618e62-openshift-default-59dbc788bd-9qtw7\" (UID: \"c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7\") " pod="openshift-ingress/e2e-ipp-b-618e62-openshift-default-59dbc788bd-9qtw7"
Sep 19 04:52:53.044105 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:53.044085    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7-istio-token\") pod \"e2e-ipp-b-618e62-openshift-default-59dbc788bd-9qtw7\" (UID: \"c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7\") " pod="openshift-ingress/e2e-ipp-b-618e62-openshift-default-59dbc788bd-9qtw7"
Sep 19 04:52:53.215941 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:53.215892    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-618e62-openshift-default-59dbc788bd-9qtw7"
Sep 19 04:52:53.258940 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:53.258903    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-cacd12-openshift-default-7c68f65c4-gvfxl"
Sep 19 04:52:53.259506 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:53.259486    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-cacd12-openshift-default-7c68f65c4-gvfxl"
Sep 19 04:52:53.260574 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:53.260555    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-cacd12-openshift-default-7c68f65c4-gvfxl"
Sep 19 04:52:53.515556 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:53.515478    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-618e62-openshift-default-59dbc788bd-9qtw7"]
Sep 19 04:52:53.519142 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:52:53.519115    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc19fb42b_e0e4_4122_ae6d_40d5b8bdf7d7.slice/crio-967639db3aa53ba056cad8336ebab0c38bb0b25ab40834ffb9c48663d747701b WatchSource:0}: Error finding container 967639db3aa53ba056cad8336ebab0c38bb0b25ab40834ffb9c48663d747701b: Status 404 returned error can't find the container with id 967639db3aa53ba056cad8336ebab0c38bb0b25ab40834ffb9c48663d747701b
Sep 19 04:52:53.521592 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:53.521557    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 04:52:53.521665 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:53.521644    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 04:52:53.521762 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:53.521689    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 04:52:53.946093 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:53.946058    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-618e62-openshift-default-59dbc788bd-9qtw7" event={"ID":"c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7","Type":"ContainerStarted","Data":"24211ea7d0e0bb58d4d45a6f0919f1ed36f4f9bc2cfeda13aba22b1d9ac089dc"}
Sep 19 04:52:53.946499 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:53.946113    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-618e62-openshift-default-59dbc788bd-9qtw7" event={"ID":"c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7","Type":"ContainerStarted","Data":"967639db3aa53ba056cad8336ebab0c38bb0b25ab40834ffb9c48663d747701b"}
Sep 19 04:52:53.975293 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:53.975244    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-618e62-openshift-default-59dbc788bd-9qtw7" podStartSLOduration=1.9752299089999998 podStartE2EDuration="1.975229909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 04:52:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:52:53.972363848 +0000 UTC m=+2353.155324977" watchObservedRunningTime="2026-09-19 04:52:53.975229909 +0000 UTC m=+2353.158191056"
Sep 19 04:52:54.217090 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:54.217000    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-618e62-openshift-default-59dbc788bd-9qtw7"
Sep 19 04:52:54.222410 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:54.222384    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-618e62-openshift-default-59dbc788bd-9qtw7"
Sep 19 04:52:54.952903 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:54.952864    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-618e62-openshift-default-59dbc788bd-9qtw7"
Sep 19 04:52:54.953664 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:54.953644    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-618e62-openshift-default-59dbc788bd-9qtw7"
Sep 19 04:52:59.764586 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:59.764549    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-618e62-5ddf9bc6f5-pzsrq"]
Sep 19 04:52:59.769245 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:59.769227    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-618e62-5ddf9bc6f5-pzsrq"
Sep 19 04:52:59.772511 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:59.772489    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-618e62\""
Sep 19 04:52:59.781459 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:59.781429    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-618e62-5ddf9bc6f5-pzsrq"]
Sep 19 04:52:59.906393 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:59.906314    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5772b706-6b5d-42cf-9f27-3d3cc187548c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-618e62-5ddf9bc6f5-pzsrq\" (UID: \"5772b706-6b5d-42cf-9f27-3d3cc187548c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-618e62-5ddf9bc6f5-pzsrq"
Sep 19 04:52:59.906594 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:52:59.906492    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjht5\" (UniqueName: \"kubernetes.io/projected/5772b706-6b5d-42cf-9f27-3d3cc187548c-kube-api-access-hjht5\") pod \"payload-processing-e2e-ipp-b-618e62-5ddf9bc6f5-pzsrq\" (UID: \"5772b706-6b5d-42cf-9f27-3d3cc187548c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-618e62-5ddf9bc6f5-pzsrq"
Sep 19 04:53:00.008040 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:53:00.007986    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5772b706-6b5d-42cf-9f27-3d3cc187548c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-618e62-5ddf9bc6f5-pzsrq\" (UID: \"5772b706-6b5d-42cf-9f27-3d3cc187548c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-618e62-5ddf9bc6f5-pzsrq"
Sep 19 04:53:00.008225 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:53:00.008069    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjht5\" (UniqueName: \"kubernetes.io/projected/5772b706-6b5d-42cf-9f27-3d3cc187548c-kube-api-access-hjht5\") pod \"payload-processing-e2e-ipp-b-618e62-5ddf9bc6f5-pzsrq\" (UID: \"5772b706-6b5d-42cf-9f27-3d3cc187548c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-618e62-5ddf9bc6f5-pzsrq"
Sep 19 04:53:00.008798 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:53:00.008779    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5772b706-6b5d-42cf-9f27-3d3cc187548c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-618e62-5ddf9bc6f5-pzsrq\" (UID: \"5772b706-6b5d-42cf-9f27-3d3cc187548c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-618e62-5ddf9bc6f5-pzsrq"
Sep 19 04:53:00.017997 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:53:00.017925    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjht5\" (UniqueName: \"kubernetes.io/projected/5772b706-6b5d-42cf-9f27-3d3cc187548c-kube-api-access-hjht5\") pod \"payload-processing-e2e-ipp-b-618e62-5ddf9bc6f5-pzsrq\" (UID: \"5772b706-6b5d-42cf-9f27-3d3cc187548c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-618e62-5ddf9bc6f5-pzsrq"
Sep 19 04:53:00.081777 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:53:00.081687    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-618e62-5ddf9bc6f5-pzsrq"
Sep 19 04:53:00.363172 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:53:00.363149    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-618e62-5ddf9bc6f5-pzsrq"]
Sep 19 04:53:00.364967 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:53:00.364933    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5772b706_6b5d_42cf_9f27_3d3cc187548c.slice/crio-db0d8bc57d9d996e4d57b404c6bdcec4bc2dc7850558f855dbe547948dc1e073 WatchSource:0}: Error finding container db0d8bc57d9d996e4d57b404c6bdcec4bc2dc7850558f855dbe547948dc1e073: Status 404 returned error can't find the container with id db0d8bc57d9d996e4d57b404c6bdcec4bc2dc7850558f855dbe547948dc1e073
Sep 19 04:53:00.993569 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:53:00.993531    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-618e62-5ddf9bc6f5-pzsrq" event={"ID":"5772b706-6b5d-42cf-9f27-3d3cc187548c","Type":"ContainerStarted","Data":"994979cf7eefe7fad60d776d5530af176817b256ff91ed5e132fcad8c2ac15ff"}
Sep 19 04:53:00.994087 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:53:00.993591    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-618e62-5ddf9bc6f5-pzsrq" event={"ID":"5772b706-6b5d-42cf-9f27-3d3cc187548c","Type":"ContainerStarted","Data":"db0d8bc57d9d996e4d57b404c6bdcec4bc2dc7850558f855dbe547948dc1e073"}
Sep 19 04:53:00.994087 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:53:00.993719    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-618e62-5ddf9bc6f5-pzsrq"
Sep 19 04:53:01.016429 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:53:01.016371    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-618e62-5ddf9bc6f5-pzsrq" podStartSLOduration=2.016355927 podStartE2EDuration="2.016355927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 04:52:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:53:01.013002441 +0000 UTC m=+2360.195963588" watchObservedRunningTime="2026-09-19 04:53:01.016355927 +0000 UTC m=+2360.199317052"
Sep 19 04:53:12.005271 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:53:12.005226    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-618e62-5ddf9bc6f5-pzsrq"
Sep 19 04:53:17.095362 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:53:17.095325    2579 generic.go:356] "Generic (PLEG): container finished" podID="c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7" containerID="24211ea7d0e0bb58d4d45a6f0919f1ed36f4f9bc2cfeda13aba22b1d9ac089dc" exitCode=0
Sep 19 04:53:17.095865 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:53:17.095404    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-618e62-openshift-default-59dbc788bd-9qtw7" event={"ID":"c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7","Type":"ContainerDied","Data":"24211ea7d0e0bb58d4d45a6f0919f1ed36f4f9bc2cfeda13aba22b1d9ac089dc"}
Sep 19 04:53:17.095939 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:53:17.095871    2579 scope.go:117] "RemoveContainer" containerID="24211ea7d0e0bb58d4d45a6f0919f1ed36f4f9bc2cfeda13aba22b1d9ac089dc"
Sep 19 04:53:17.097045 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:53:17.097009    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 04:53:17.097144 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:53:17.097077    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 04:53:17.097144 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:53:17.097109    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 04:53:18.108195 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:53:18.108150    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-618e62-openshift-default-59dbc788bd-9qtw7" event={"ID":"c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7","Type":"ContainerStarted","Data":"ee2012ca95eeca8aaa72f8916039481920b489062c463a9adc43aa54378834cf"}
Sep 19 04:53:18.216385 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:53:18.216338    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-618e62-openshift-default-59dbc788bd-9qtw7"
Sep 19 04:53:18.216637 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:53:18.216613    2579 patch_prober.go:28] interesting pod/e2e-ipp-b-618e62-openshift-default-59dbc788bd-9qtw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Sep 19 04:53:18.216777 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:53:18.216683    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-618e62-openshift-default-59dbc788bd-9qtw7" podUID="c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Sep 19 04:53:19.216523 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:53:19.216421    2579 patch_prober.go:28] interesting pod/e2e-ipp-b-618e62-openshift-default-59dbc788bd-9qtw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Sep 19 04:53:19.216523 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:53:19.216493    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-618e62-openshift-default-59dbc788bd-9qtw7" podUID="c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Sep 19 04:53:20.216654 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:53:20.216609    2579 patch_prober.go:28] interesting pod/e2e-ipp-b-618e62-openshift-default-59dbc788bd-9qtw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Sep 19 04:53:20.217094 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:53:20.216688    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-618e62-openshift-default-59dbc788bd-9qtw7" podUID="c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Sep 19 04:53:21.217298 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:53:21.217257    2579 patch_prober.go:28] interesting pod/e2e-ipp-b-618e62-openshift-default-59dbc788bd-9qtw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Sep 19 04:53:21.217686 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:53:21.217317    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-618e62-openshift-default-59dbc788bd-9qtw7" podUID="c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Sep 19 04:53:22.217194 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:53:22.217143    2579 patch_prober.go:28] interesting pod/e2e-ipp-b-618e62-openshift-default-59dbc788bd-9qtw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Sep 19 04:53:22.217416 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:53:22.217212    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-618e62-openshift-default-59dbc788bd-9qtw7" podUID="c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Sep 19 04:53:23.310534 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:53:23.310489    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-618e62-openshift-default-59dbc788bd-9qtw7"
Sep 19 04:53:23.311040 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:53:23.310875    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-618e62-openshift-default-59dbc788bd-9qtw7"
Sep 19 04:53:23.311927 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:53:23.311903    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-618e62-openshift-default-59dbc788bd-9qtw7"
Sep 19 04:53:31.331412 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:53:31.331380    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-sjhch_fa5e4819-0ed6-44af-a94a-c3529a77ef4b/payload-processing/0.log"
Sep 19 04:53:31.441897 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:53:31.441859    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-618e62-5ddf9bc6f5-pzsrq_5772b706-6b5d-42cf-9f27-3d3cc187548c/payload-processing/0.log"
Sep 19 04:53:41.909088 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:53:41.909055    2579 scope.go:117] "RemoveContainer" containerID="55a70d83a3aa2ad1efd9296fd646d7d6a513e3e447eb62b7cc17e26fec4522e8"
Sep 19 04:53:41.918056 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:53:41.918037    2579 scope.go:117] "RemoveContainer" containerID="ce7f46575784b9c36c1f08357a6407c1be806833cc39098d3ea57426520c6887"
Sep 19 04:53:43.544303 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:53:43.544269    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-cacd12/ipp-route-cacd12-kserve-7f8864dcd4-ghfwf"]
Sep 19 04:53:43.549668 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:53:43.549645    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-cacd12/ipp-route-cacd12-kserve-7f8864dcd4-ghfwf"
Sep 19 04:53:43.553439 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:53:43.553416    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-cacd12\"/\"kube-root-ca.crt\""
Sep 19 04:53:43.553666 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:53:43.553651    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-cacd12\"/\"openshift-service-ca.crt\""
Sep 19 04:53:43.554410 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:53:43.554375    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-cacd12\"/\"default-dockercfg-7wkzt\""
Sep 19 04:53:43.554410 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:53:43.554405    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-cacd12\"/\"ipp-route-cacd12-kserve-self-signed-certs\""
Sep 19 04:53:43.562467 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:53:43.562448    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-cacd12/ipp-route-cacd12-kserve-7f8864dcd4-ghfwf"]
Sep 19 04:53:43.653093 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:53:43.653057    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ef7f683-8b60-4ce8-98fd-aa30062387d0-tls-certs\") pod \"ipp-route-cacd12-kserve-7f8864dcd4-ghfwf\" (UID: \"7ef7f683-8b60-4ce8-98fd-aa30062387d0\") " pod="ai-tenant-e2e-ipp-a-cacd12/ipp-route-cacd12-kserve-7f8864dcd4-ghfwf"
Sep 19 04:53:43.653269 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:53:43.653104    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ef7f683-8b60-4ce8-98fd-aa30062387d0-home\") pod \"ipp-route-cacd12-kserve-7f8864dcd4-ghfwf\" (UID: \"7ef7f683-8b60-4ce8-98fd-aa30062387d0\") " pod="ai-tenant-e2e-ipp-a-cacd12/ipp-route-cacd12-kserve-7f8864dcd4-ghfwf"
Sep 19 04:53:43.653269 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:53:43.653143    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ef7f683-8b60-4ce8-98fd-aa30062387d0-tmp-dir\") pod \"ipp-route-cacd12-kserve-7f8864dcd4-ghfwf\" (UID: \"7ef7f683-8b60-4ce8-98fd-aa30062387d0\") " pod="ai-tenant-e2e-ipp-a-cacd12/ipp-route-cacd12-kserve-7f8864dcd4-ghfwf"
Sep 19 04:53:43.653269 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:53:43.653170    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ef7f683-8b60-4ce8-98fd-aa30062387d0-model-cache\") pod \"ipp-route-cacd12-kserve-7f8864dcd4-ghfwf\" (UID: \"7ef7f683-8b60-4ce8-98fd-aa30062387d0\") " pod="ai-tenant-e2e-ipp-a-cacd12/ipp-route-cacd12-kserve-7f8864dcd4-ghfwf"
Sep 19 04:53:43.653269 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:53:43.653212    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hdt9\" (UniqueName: \"kubernetes.io/projected/7ef7f683-8b60-4ce8-98fd-aa30062387d0-kube-api-access-6hdt9\") pod \"ipp-route-cacd12-kserve-7f8864dcd4-ghfwf\" (UID: \"7ef7f683-8b60-4ce8-98fd-aa30062387d0\") " pod="ai-tenant-e2e-ipp-a-cacd12/ipp-route-cacd12-kserve-7f8864dcd4-ghfwf"
Sep 19 04:53:43.653415 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:53:43.653304    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ef7f683-8b60-4ce8-98fd-aa30062387d0-dshm\") pod \"ipp-route-cacd12-kserve-7f8864dcd4-ghfwf\" (UID: \"7ef7f683-8b60-4ce8-98fd-aa30062387d0\") " pod="ai-tenant-e2e-ipp-a-cacd12/ipp-route-cacd12-kserve-7f8864dcd4-ghfwf"
Sep 19 04:53:43.754195 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:53:43.754167    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ef7f683-8b60-4ce8-98fd-aa30062387d0-dshm\") pod \"ipp-route-cacd12-kserve-7f8864dcd4-ghfwf\" (UID: \"7ef7f683-8b60-4ce8-98fd-aa30062387d0\") " pod="ai-tenant-e2e-ipp-a-cacd12/ipp-route-cacd12-kserve-7f8864dcd4-ghfwf"
Sep 19 04:53:43.754375 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:53:43.754212    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ef7f683-8b60-4ce8-98fd-aa30062387d0-tls-certs\") pod \"ipp-route-cacd12-kserve-7f8864dcd4-ghfwf\" (UID: \"7ef7f683-8b60-4ce8-98fd-aa30062387d0\") " pod="ai-tenant-e2e-ipp-a-cacd12/ipp-route-cacd12-kserve-7f8864dcd4-ghfwf"
Sep 19 04:53:43.754375 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:53:43.754244    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ef7f683-8b60-4ce8-98fd-aa30062387d0-home\") pod \"ipp-route-cacd12-kserve-7f8864dcd4-ghfwf\" (UID: \"7ef7f683-8b60-4ce8-98fd-aa30062387d0\") " pod="ai-tenant-e2e-ipp-a-cacd12/ipp-route-cacd12-kserve-7f8864dcd4-ghfwf"
Sep 19 04:53:43.754375 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:53:43.754281    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ef7f683-8b60-4ce8-98fd-aa30062387d0-tmp-dir\") pod \"ipp-route-cacd12-kserve-7f8864dcd4-ghfwf\" (UID: \"7ef7f683-8b60-4ce8-98fd-aa30062387d0\") " pod="ai-tenant-e2e-ipp-a-cacd12/ipp-route-cacd12-kserve-7f8864dcd4-ghfwf"
Sep 19 04:53:43.754375 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:53:43.754312    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ef7f683-8b60-4ce8-98fd-aa30062387d0-model-cache\") pod \"ipp-route-cacd12-kserve-7f8864dcd4-ghfwf\" (UID: \"7ef7f683-8b60-4ce8-98fd-aa30062387d0\") " pod="ai-tenant-e2e-ipp-a-cacd12/ipp-route-cacd12-kserve-7f8864dcd4-ghfwf"
Sep 19 04:53:43.754375 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:53:43.754351    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hdt9\" (UniqueName: \"kubernetes.io/projected/7ef7f683-8b60-4ce8-98fd-aa30062387d0-kube-api-access-6hdt9\") pod \"ipp-route-cacd12-kserve-7f8864dcd4-ghfwf\" (UID: \"7ef7f683-8b60-4ce8-98fd-aa30062387d0\") " pod="ai-tenant-e2e-ipp-a-cacd12/ipp-route-cacd12-kserve-7f8864dcd4-ghfwf"
Sep 19 04:53:43.754774 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:53:43.754737    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ef7f683-8b60-4ce8-98fd-aa30062387d0-home\") pod \"ipp-route-cacd12-kserve-7f8864dcd4-ghfwf\" (UID: \"7ef7f683-8b60-4ce8-98fd-aa30062387d0\") " pod="ai-tenant-e2e-ipp-a-cacd12/ipp-route-cacd12-kserve-7f8864dcd4-ghfwf"
Sep 19 04:53:43.754906 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:53:43.754783    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ef7f683-8b60-4ce8-98fd-aa30062387d0-tmp-dir\") pod \"ipp-route-cacd12-kserve-7f8864dcd4-ghfwf\" (UID: \"7ef7f683-8b60-4ce8-98fd-aa30062387d0\") " pod="ai-tenant-e2e-ipp-a-cacd12/ipp-route-cacd12-kserve-7f8864dcd4-ghfwf"
Sep 19 04:53:43.754906 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:53:43.754785    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ef7f683-8b60-4ce8-98fd-aa30062387d0-model-cache\") pod \"ipp-route-cacd12-kserve-7f8864dcd4-ghfwf\" (UID: \"7ef7f683-8b60-4ce8-98fd-aa30062387d0\") " pod="ai-tenant-e2e-ipp-a-cacd12/ipp-route-cacd12-kserve-7f8864dcd4-ghfwf"
Sep 19 04:53:43.756675 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:53:43.756647    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ef7f683-8b60-4ce8-98fd-aa30062387d0-dshm\") pod \"ipp-route-cacd12-kserve-7f8864dcd4-ghfwf\" (UID: \"7ef7f683-8b60-4ce8-98fd-aa30062387d0\") " pod="ai-tenant-e2e-ipp-a-cacd12/ipp-route-cacd12-kserve-7f8864dcd4-ghfwf"
Sep 19 04:53:43.757131 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:53:43.757115    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ef7f683-8b60-4ce8-98fd-aa30062387d0-tls-certs\") pod \"ipp-route-cacd12-kserve-7f8864dcd4-ghfwf\" (UID: \"7ef7f683-8b60-4ce8-98fd-aa30062387d0\") " pod="ai-tenant-e2e-ipp-a-cacd12/ipp-route-cacd12-kserve-7f8864dcd4-ghfwf"
Sep 19 04:53:43.763792 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:53:43.763682    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hdt9\" (UniqueName: \"kubernetes.io/projected/7ef7f683-8b60-4ce8-98fd-aa30062387d0-kube-api-access-6hdt9\") pod \"ipp-route-cacd12-kserve-7f8864dcd4-ghfwf\" (UID: \"7ef7f683-8b60-4ce8-98fd-aa30062387d0\") " pod="ai-tenant-e2e-ipp-a-cacd12/ipp-route-cacd12-kserve-7f8864dcd4-ghfwf"
Sep 19 04:53:43.859117 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:53:43.859025    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-cacd12/ipp-route-cacd12-kserve-7f8864dcd4-ghfwf"
Sep 19 04:53:44.147403 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:53:44.147373    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-cacd12/ipp-route-cacd12-kserve-7f8864dcd4-ghfwf"]
Sep 19 04:53:44.148793 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:53:44.148762    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7ef7f683_8b60_4ce8_98fd_aa30062387d0.slice/crio-d1ba7e87943374fbcf559f9a32d627e0bf09beebfe726137f079f94ea1b06352 WatchSource:0}: Error finding container d1ba7e87943374fbcf559f9a32d627e0bf09beebfe726137f079f94ea1b06352: Status 404 returned error can't find the container with id d1ba7e87943374fbcf559f9a32d627e0bf09beebfe726137f079f94ea1b06352
Sep 19 04:53:44.260894 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:53:44.260854    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-cacd12/ipp-route-cacd12-kserve-7f8864dcd4-ghfwf" event={"ID":"7ef7f683-8b60-4ce8-98fd-aa30062387d0","Type":"ContainerStarted","Data":"ff11072bbe9ea46136d335ee9b45c9d435efee3a30c0040570b62a22ec3018cc"}
Sep 19 04:53:44.261060 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:53:44.260909    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-cacd12/ipp-route-cacd12-kserve-7f8864dcd4-ghfwf" event={"ID":"7ef7f683-8b60-4ce8-98fd-aa30062387d0","Type":"ContainerStarted","Data":"d1ba7e87943374fbcf559f9a32d627e0bf09beebfe726137f079f94ea1b06352"}
Sep 19 04:53:44.279911 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:53:44.279855    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-cacd12/ipp-route-cacd12-kserve-7f8864dcd4-ghfwf" podStartSLOduration=1.279837355 podStartE2EDuration="1.279837355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 04:53:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:53:44.278487583 +0000 UTC m=+2403.461448744" watchObservedRunningTime="2026-09-19 04:53:44.279837355 +0000 UTC m=+2403.462798565"
Sep 19 04:53:44.859870 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:53:44.859839    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-cacd12/ipp-route-cacd12-kserve-7f8864dcd4-ghfwf"
Sep 19 04:53:53.859480 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:53:53.859439    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-cacd12/ipp-route-cacd12-kserve-7f8864dcd4-ghfwf"
Sep 19 04:53:53.871882 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:53:53.871858    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-cacd12/ipp-route-cacd12-kserve-7f8864dcd4-ghfwf"
Sep 19 04:53:54.332818 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:53:54.332791    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-cacd12/ipp-route-cacd12-kserve-7f8864dcd4-ghfwf"
Sep 19 04:54:20.128550 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:54:20.128514    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-sjhch_fa5e4819-0ed6-44af-a94a-c3529a77ef4b/payload-processing/0.log"
Sep 19 04:54:20.323807 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:54:20.323763    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-618e62-5ddf9bc6f5-pzsrq_5772b706-6b5d-42cf-9f27-3d3cc187548c/payload-processing/0.log"
Sep 19 04:54:20.453220 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:54:20.453191    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-618e62-5ddf9bc6f5-pzsrq_5772b706-6b5d-42cf-9f27-3d3cc187548c/payload-processing/0.log"
Sep 19 04:54:28.934205 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:54:28.934175    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-022574-7b6c48f5-wclxh"]
Sep 19 04:54:28.937957 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:54:28.937940    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-022574-7b6c48f5-wclxh"
Sep 19 04:54:28.942470 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:54:28.942448    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-022574-dockercfg-j426z\""
Sep 19 04:54:28.942470 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:54:28.942462    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-022574\""
Sep 19 04:54:28.949170 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:54:28.949143    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-022574-7b6c48f5-wclxh"]
Sep 19 04:54:29.014297 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:54:29.014256    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zk5h\" (UniqueName: \"kubernetes.io/projected/95e2dcd4-a330-4dbc-9067-d646690b98ce-kube-api-access-6zk5h\") pod \"payload-processing-e2e-ipp-cleanup-022574-7b6c48f5-wclxh\" (UID: \"95e2dcd4-a330-4dbc-9067-d646690b98ce\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-022574-7b6c48f5-wclxh"
Sep 19 04:54:29.014659 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:54:29.014622    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95e2dcd4-a330-4dbc-9067-d646690b98ce-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-022574-7b6c48f5-wclxh\" (UID: \"95e2dcd4-a330-4dbc-9067-d646690b98ce\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-022574-7b6c48f5-wclxh"
Sep 19 04:54:29.116349 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:54:29.116309    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zk5h\" (UniqueName: \"kubernetes.io/projected/95e2dcd4-a330-4dbc-9067-d646690b98ce-kube-api-access-6zk5h\") pod \"payload-processing-e2e-ipp-cleanup-022574-7b6c48f5-wclxh\" (UID: \"95e2dcd4-a330-4dbc-9067-d646690b98ce\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-022574-7b6c48f5-wclxh"
Sep 19 04:54:29.116534 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:54:29.116392    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95e2dcd4-a330-4dbc-9067-d646690b98ce-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-022574-7b6c48f5-wclxh\" (UID: \"95e2dcd4-a330-4dbc-9067-d646690b98ce\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-022574-7b6c48f5-wclxh"
Sep 19 04:54:29.117077 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:54:29.117057    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95e2dcd4-a330-4dbc-9067-d646690b98ce-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-022574-7b6c48f5-wclxh\" (UID: \"95e2dcd4-a330-4dbc-9067-d646690b98ce\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-022574-7b6c48f5-wclxh"
Sep 19 04:54:29.130495 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:54:29.130466    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zk5h\" (UniqueName: \"kubernetes.io/projected/95e2dcd4-a330-4dbc-9067-d646690b98ce-kube-api-access-6zk5h\") pod \"payload-processing-e2e-ipp-cleanup-022574-7b6c48f5-wclxh\" (UID: \"95e2dcd4-a330-4dbc-9067-d646690b98ce\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-022574-7b6c48f5-wclxh"
Sep 19 04:54:29.249389 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:54:29.249308    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-022574-7b6c48f5-wclxh"
Sep 19 04:54:29.527328 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:54:29.527297    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-022574-7b6c48f5-wclxh"]
Sep 19 04:54:29.527769 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:54:29.527745    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95e2dcd4_a330_4dbc_9067_d646690b98ce.slice/crio-974eea950261ef07cb143d01fed3bc44d3cb7547c98b612874bebfd0848de165 WatchSource:0}: Error finding container 974eea950261ef07cb143d01fed3bc44d3cb7547c98b612874bebfd0848de165: Status 404 returned error can't find the container with id 974eea950261ef07cb143d01fed3bc44d3cb7547c98b612874bebfd0848de165
Sep 19 04:54:29.552370 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:54:29.552335    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-022574-7b6c48f5-wclxh" event={"ID":"95e2dcd4-a330-4dbc-9067-d646690b98ce","Type":"ContainerStarted","Data":"974eea950261ef07cb143d01fed3bc44d3cb7547c98b612874bebfd0848de165"}
Sep 19 04:54:30.559568 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:54:30.559524    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-022574-7b6c48f5-wclxh" event={"ID":"95e2dcd4-a330-4dbc-9067-d646690b98ce","Type":"ContainerStarted","Data":"7a82f6d26d156c4f373120446fc880662d7eb4f22ab3d9516b4c1084f6814dda"}
Sep 19 04:54:30.560001 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:54:30.559594    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-022574-7b6c48f5-wclxh"
Sep 19 04:54:30.579564 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:54:30.579509    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-022574-7b6c48f5-wclxh" podStartSLOduration=2.579494726 podStartE2EDuration="2.579494726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 04:54:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:54:30.577135456 +0000 UTC m=+2449.760096606" watchObservedRunningTime="2026-09-19 04:54:30.579494726 +0000 UTC m=+2449.762455874"
Sep 19 04:54:41.567477 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:54:41.567447    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-022574-7b6c48f5-wclxh"
Sep 19 04:54:41.942343 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:54:41.942311    2579 scope.go:117] "RemoveContainer" containerID="2c82ecd5f8dd296bc3c3fa18135fcbeffab68e6185d8db45b2182ee7705cd814"
Sep 19 04:54:41.951505 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:54:41.951485    2579 scope.go:117] "RemoveContainer" containerID="93caab62c6bdd7a122f129c7ab9a4119aca982ba4299fda90afefff2f4da6acd"
Sep 19 04:54:41.959685 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:54:41.959666    2579 scope.go:117] "RemoveContainer" containerID="5044c1dce773da7a478137025418addec88a0508702bdac710d91d1f0e021211"
Sep 19 04:54:54.982836 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:54:54.982794    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-022574-kptg5"]
Sep 19 04:54:54.987159 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:54:54.987131    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-022574-kptg5"
Sep 19 04:54:54.990883 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:54:54.990843    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 19 04:54:54.991041 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:54:54.990919    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s6xbq\""
Sep 19 04:54:54.998277 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:54:54.998247    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-022574-kptg5"]
Sep 19 04:54:55.084187 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:54:55.084150    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/259da0be-c9ba-4b66-8406-2137dfbdbfe3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-022574-kptg5\" (UID: \"259da0be-c9ba-4b66-8406-2137dfbdbfe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-022574-kptg5"
Sep 19 04:54:55.185749 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:54:55.185689    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/259da0be-c9ba-4b66-8406-2137dfbdbfe3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-022574-kptg5\" (UID: \"259da0be-c9ba-4b66-8406-2137dfbdbfe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-022574-kptg5"
Sep 19 04:54:55.186366 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:54:55.186344    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/259da0be-c9ba-4b66-8406-2137dfbdbfe3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-022574-kptg5\" (UID: \"259da0be-c9ba-4b66-8406-2137dfbdbfe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-022574-kptg5"
Sep 19 04:54:55.301337 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:54:55.301250    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-022574-kptg5"
Sep 19 04:54:55.585399 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:54:55.585370    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-022574-kptg5"]
Sep 19 04:54:55.588301 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:54:55.588272    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod259da0be_c9ba_4b66_8406_2137dfbdbfe3.slice/crio-ca999b19c740f8adf5c1a4977a22c77da44e49b371c97e467502566dfa02fdc1 WatchSource:0}: Error finding container ca999b19c740f8adf5c1a4977a22c77da44e49b371c97e467502566dfa02fdc1: Status 404 returned error can't find the container with id ca999b19c740f8adf5c1a4977a22c77da44e49b371c97e467502566dfa02fdc1
Sep 19 04:54:55.718645 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:54:55.718606    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-022574-kptg5" event={"ID":"259da0be-c9ba-4b66-8406-2137dfbdbfe3","Type":"ContainerStarted","Data":"ca999b19c740f8adf5c1a4977a22c77da44e49b371c97e467502566dfa02fdc1"}
Sep 19 04:54:56.726161 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:54:56.726128    2579 generic.go:356] "Generic (PLEG): container finished" podID="259da0be-c9ba-4b66-8406-2137dfbdbfe3" containerID="d5efbea920b31b8efcf063e8e34c9ca29ccf1016310f9411666f228a37b106da" exitCode=0
Sep 19 04:54:56.726609 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:54:56.726192    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-022574-kptg5" event={"ID":"259da0be-c9ba-4b66-8406-2137dfbdbfe3","Type":"ContainerDied","Data":"d5efbea920b31b8efcf063e8e34c9ca29ccf1016310f9411666f228a37b106da"}
Sep 19 04:54:57.874085 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:54:57.874060    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-022574-kptg5"
Sep 19 04:54:58.013139 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:54:58.013046    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/259da0be-c9ba-4b66-8406-2137dfbdbfe3-maas-api-service-ca\") pod \"259da0be-c9ba-4b66-8406-2137dfbdbfe3\" (UID: \"259da0be-c9ba-4b66-8406-2137dfbdbfe3\") "
Sep 19 04:54:58.013368 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:54:58.013344    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/259da0be-c9ba-4b66-8406-2137dfbdbfe3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "259da0be-c9ba-4b66-8406-2137dfbdbfe3" (UID: "259da0be-c9ba-4b66-8406-2137dfbdbfe3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 04:54:58.114757 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:54:58.114685    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/259da0be-c9ba-4b66-8406-2137dfbdbfe3-maas-api-service-ca\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:54:58.739504 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:54:58.739476    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-022574-kptg5"
Sep 19 04:54:58.739717 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:54:58.739525    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-022574-kptg5" event={"ID":"259da0be-c9ba-4b66-8406-2137dfbdbfe3","Type":"ContainerDied","Data":"ca999b19c740f8adf5c1a4977a22c77da44e49b371c97e467502566dfa02fdc1"}
Sep 19 04:54:58.739717 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:54:58.739584    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca999b19c740f8adf5c1a4977a22c77da44e49b371c97e467502566dfa02fdc1"
Sep 19 04:55:06.180207 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:06.180167    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-022574-7b6c48f5-wclxh"]
Sep 19 04:55:06.180829 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:06.180473    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-022574-7b6c48f5-wclxh" podUID="95e2dcd4-a330-4dbc-9067-d646690b98ce" containerName="payload-processing" containerID="cri-o://7a82f6d26d156c4f373120446fc880662d7eb4f22ab3d9516b4c1084f6814dda" gracePeriod=30
Sep 19 04:55:06.401508 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:06.401473    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-022574-kptg5"]
Sep 19 04:55:06.408108 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:06.408060    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-022574-kptg5"]
Sep 19 04:55:06.452875 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:06.452844    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-022574-7b6c48f5-wclxh"
Sep 19 04:55:06.496451 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:06.496416    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6zk5h\" (UniqueName: \"kubernetes.io/projected/95e2dcd4-a330-4dbc-9067-d646690b98ce-kube-api-access-6zk5h\") pod \"95e2dcd4-a330-4dbc-9067-d646690b98ce\" (UID: \"95e2dcd4-a330-4dbc-9067-d646690b98ce\") "
Sep 19 04:55:06.496648 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:06.496486    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95e2dcd4-a330-4dbc-9067-d646690b98ce-plugins-config-volume\") pod \"95e2dcd4-a330-4dbc-9067-d646690b98ce\" (UID: \"95e2dcd4-a330-4dbc-9067-d646690b98ce\") "
Sep 19 04:55:06.496881 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:06.496852    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95e2dcd4-a330-4dbc-9067-d646690b98ce-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "95e2dcd4-a330-4dbc-9067-d646690b98ce" (UID: "95e2dcd4-a330-4dbc-9067-d646690b98ce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 04:55:06.499052 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:06.499002    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95e2dcd4-a330-4dbc-9067-d646690b98ce-kube-api-access-6zk5h" (OuterVolumeSpecName: "kube-api-access-6zk5h") pod "95e2dcd4-a330-4dbc-9067-d646690b98ce" (UID: "95e2dcd4-a330-4dbc-9067-d646690b98ce"). InnerVolumeSpecName "kube-api-access-6zk5h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 04:55:06.597226 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:06.597183    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zk5h\" (UniqueName: \"kubernetes.io/projected/95e2dcd4-a330-4dbc-9067-d646690b98ce-kube-api-access-6zk5h\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:55:06.597226 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:06.597221    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95e2dcd4-a330-4dbc-9067-d646690b98ce-plugins-config-volume\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:55:06.787537 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:06.787435    2579 generic.go:356] "Generic (PLEG): container finished" podID="95e2dcd4-a330-4dbc-9067-d646690b98ce" containerID="7a82f6d26d156c4f373120446fc880662d7eb4f22ab3d9516b4c1084f6814dda" exitCode=0
Sep 19 04:55:06.787537 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:06.787483    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-022574-7b6c48f5-wclxh"
Sep 19 04:55:06.787537 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:06.787523    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-022574-7b6c48f5-wclxh" event={"ID":"95e2dcd4-a330-4dbc-9067-d646690b98ce","Type":"ContainerDied","Data":"7a82f6d26d156c4f373120446fc880662d7eb4f22ab3d9516b4c1084f6814dda"}
Sep 19 04:55:06.787859 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:06.787578    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-022574-7b6c48f5-wclxh" event={"ID":"95e2dcd4-a330-4dbc-9067-d646690b98ce","Type":"ContainerDied","Data":"974eea950261ef07cb143d01fed3bc44d3cb7547c98b612874bebfd0848de165"}
Sep 19 04:55:06.787859 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:06.787601    2579 scope.go:117] "RemoveContainer" containerID="7a82f6d26d156c4f373120446fc880662d7eb4f22ab3d9516b4c1084f6814dda"
Sep 19 04:55:06.798316 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:06.798286    2579 scope.go:117] "RemoveContainer" containerID="7a82f6d26d156c4f373120446fc880662d7eb4f22ab3d9516b4c1084f6814dda"
Sep 19 04:55:06.798775 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:55:06.798747    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a82f6d26d156c4f373120446fc880662d7eb4f22ab3d9516b4c1084f6814dda\": container with ID starting with 7a82f6d26d156c4f373120446fc880662d7eb4f22ab3d9516b4c1084f6814dda not found: ID does not exist" containerID="7a82f6d26d156c4f373120446fc880662d7eb4f22ab3d9516b4c1084f6814dda"
Sep 19 04:55:06.798902 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:06.798787    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a82f6d26d156c4f373120446fc880662d7eb4f22ab3d9516b4c1084f6814dda"} err="failed to get container status \"7a82f6d26d156c4f373120446fc880662d7eb4f22ab3d9516b4c1084f6814dda\": rpc error: code = NotFound desc = could not find container \"7a82f6d26d156c4f373120446fc880662d7eb4f22ab3d9516b4c1084f6814dda\": container with ID starting with 7a82f6d26d156c4f373120446fc880662d7eb4f22ab3d9516b4c1084f6814dda not found: ID does not exist"
Sep 19 04:55:06.818323 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:06.818279    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-022574-7b6c48f5-wclxh"]
Sep 19 04:55:06.828852 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:06.828820    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-022574-7b6c48f5-wclxh"]
Sep 19 04:55:07.352026 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:07.351991    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="259da0be-c9ba-4b66-8406-2137dfbdbfe3" path="/var/lib/kubelet/pods/259da0be-c9ba-4b66-8406-2137dfbdbfe3/volumes"
Sep 19 04:55:07.352515 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:07.352494    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95e2dcd4-a330-4dbc-9067-d646690b98ce" path="/var/lib/kubelet/pods/95e2dcd4-a330-4dbc-9067-d646690b98ce/volumes"
Sep 19 04:55:17.804974 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:17.804942    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cacd12-8427l"]
Sep 19 04:55:17.805358 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:17.805347    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95e2dcd4-a330-4dbc-9067-d646690b98ce" containerName="payload-processing"
Sep 19 04:55:17.805400 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:17.805362    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="95e2dcd4-a330-4dbc-9067-d646690b98ce" containerName="payload-processing"
Sep 19 04:55:17.805400 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:17.805386    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="259da0be-c9ba-4b66-8406-2137dfbdbfe3" containerName="revoke-keys"
Sep 19 04:55:17.805400 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:17.805392    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="259da0be-c9ba-4b66-8406-2137dfbdbfe3" containerName="revoke-keys"
Sep 19 04:55:17.805509 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:17.805461    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="95e2dcd4-a330-4dbc-9067-d646690b98ce" containerName="payload-processing"
Sep 19 04:55:17.805509 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:17.805474    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="259da0be-c9ba-4b66-8406-2137dfbdbfe3" containerName="revoke-keys"
Sep 19 04:55:17.808617 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:17.808597    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cacd12-8427l"
Sep 19 04:55:17.812247 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:17.812224    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 19 04:55:17.812383 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:17.812229    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s6xbq\""
Sep 19 04:55:17.818373 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:17.818351    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cacd12-8427l"]
Sep 19 04:55:17.911080 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:17.911044    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7eeea1fc-67ce-4941-ad3d-06c39758867b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-cacd12-8427l\" (UID: \"7eeea1fc-67ce-4941-ad3d-06c39758867b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cacd12-8427l"
Sep 19 04:55:18.012134 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:18.012099    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7eeea1fc-67ce-4941-ad3d-06c39758867b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-cacd12-8427l\" (UID: \"7eeea1fc-67ce-4941-ad3d-06c39758867b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cacd12-8427l"
Sep 19 04:55:18.012741 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:18.012724    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7eeea1fc-67ce-4941-ad3d-06c39758867b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-cacd12-8427l\" (UID: \"7eeea1fc-67ce-4941-ad3d-06c39758867b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cacd12-8427l"
Sep 19 04:55:18.121219 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:18.121136    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cacd12-8427l"
Sep 19 04:55:18.397506 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:18.397478    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cacd12-8427l"]
Sep 19 04:55:18.398795 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:55:18.398769    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eeea1fc_67ce_4941_ad3d_06c39758867b.slice/crio-7e496e632000acb33983659bfda95ded432d56c8157a677579b93ee5b53f2961 WatchSource:0}: Error finding container 7e496e632000acb33983659bfda95ded432d56c8157a677579b93ee5b53f2961: Status 404 returned error can't find the container with id 7e496e632000acb33983659bfda95ded432d56c8157a677579b93ee5b53f2961
Sep 19 04:55:18.863052 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:18.863010    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cacd12-8427l" event={"ID":"7eeea1fc-67ce-4941-ad3d-06c39758867b","Type":"ContainerStarted","Data":"da14780c5020afc6c734717f8ffada8af51d7e4a9ceb4a9a77025f89fc4a3e5a"}
Sep 19 04:55:18.863449 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:18.863111    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cacd12-8427l" event={"ID":"7eeea1fc-67ce-4941-ad3d-06c39758867b","Type":"ContainerStarted","Data":"7e496e632000acb33983659bfda95ded432d56c8157a677579b93ee5b53f2961"}
Sep 19 04:55:18.890313 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:18.890254    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cacd12-8427l" podStartSLOduration=1.890238351 podStartE2EDuration="1.890238351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 04:55:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:55:18.888137234 +0000 UTC m=+2498.071098383" watchObservedRunningTime="2026-09-19 04:55:18.890238351 +0000 UTC m=+2498.073199499"
Sep 19 04:55:19.873540 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:19.873448    2579 generic.go:356] "Generic (PLEG): container finished" podID="7eeea1fc-67ce-4941-ad3d-06c39758867b" containerID="da14780c5020afc6c734717f8ffada8af51d7e4a9ceb4a9a77025f89fc4a3e5a" exitCode=0
Sep 19 04:55:19.873540 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:19.873492    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cacd12-8427l" event={"ID":"7eeea1fc-67ce-4941-ad3d-06c39758867b","Type":"ContainerDied","Data":"da14780c5020afc6c734717f8ffada8af51d7e4a9ceb4a9a77025f89fc4a3e5a"}
Sep 19 04:55:21.026062 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:21.026034    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cacd12-8427l"
Sep 19 04:55:21.139017 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:21.138931    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7eeea1fc-67ce-4941-ad3d-06c39758867b-maas-api-service-ca\") pod \"7eeea1fc-67ce-4941-ad3d-06c39758867b\" (UID: \"7eeea1fc-67ce-4941-ad3d-06c39758867b\") "
Sep 19 04:55:21.139288 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:21.139264    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7eeea1fc-67ce-4941-ad3d-06c39758867b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7eeea1fc-67ce-4941-ad3d-06c39758867b" (UID: "7eeea1fc-67ce-4941-ad3d-06c39758867b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 04:55:21.240060 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:21.240018    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7eeea1fc-67ce-4941-ad3d-06c39758867b-maas-api-service-ca\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:55:21.886739 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:21.886681    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cacd12-8427l" event={"ID":"7eeea1fc-67ce-4941-ad3d-06c39758867b","Type":"ContainerDied","Data":"7e496e632000acb33983659bfda95ded432d56c8157a677579b93ee5b53f2961"}
Sep 19 04:55:21.886921 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:21.886745    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cacd12-8427l"
Sep 19 04:55:21.886921 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:21.886756    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e496e632000acb33983659bfda95ded432d56c8157a677579b93ee5b53f2961"
Sep 19 04:55:33.428566 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:33.428478    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-cacd12-779b59d64b-dqdcv"]
Sep 19 04:55:33.429130 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:33.428745    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cacd12-779b59d64b-dqdcv" podUID="669638a2-9a0d-40b3-bb81-558a6f82cb2b" containerName="payload-pre-processing" containerID="cri-o://e4300702d2382a4c84a145b095afc1a8decc81c545b69d7694c2693ebb5857ca" gracePeriod=30
Sep 19 04:55:33.699851 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:33.699823    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cacd12-8427l"]
Sep 19 04:55:33.703446 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:33.703420    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cacd12-8427l"]
Sep 19 04:55:33.708683 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:33.708663    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cacd12-779b59d64b-dqdcv"
Sep 19 04:55:33.872688 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:33.872658    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/669638a2-9a0d-40b3-bb81-558a6f82cb2b-plugins-config-volume\") pod \"669638a2-9a0d-40b3-bb81-558a6f82cb2b\" (UID: \"669638a2-9a0d-40b3-bb81-558a6f82cb2b\") "
Sep 19 04:55:33.872885 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:33.872729    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cctsv\" (UniqueName: \"kubernetes.io/projected/669638a2-9a0d-40b3-bb81-558a6f82cb2b-kube-api-access-cctsv\") pod \"669638a2-9a0d-40b3-bb81-558a6f82cb2b\" (UID: \"669638a2-9a0d-40b3-bb81-558a6f82cb2b\") "
Sep 19 04:55:33.873038 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:33.873018    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/669638a2-9a0d-40b3-bb81-558a6f82cb2b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "669638a2-9a0d-40b3-bb81-558a6f82cb2b" (UID: "669638a2-9a0d-40b3-bb81-558a6f82cb2b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 04:55:33.875035 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:33.875015    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/669638a2-9a0d-40b3-bb81-558a6f82cb2b-kube-api-access-cctsv" (OuterVolumeSpecName: "kube-api-access-cctsv") pod "669638a2-9a0d-40b3-bb81-558a6f82cb2b" (UID: "669638a2-9a0d-40b3-bb81-558a6f82cb2b"). InnerVolumeSpecName "kube-api-access-cctsv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 04:55:33.961425 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:33.961341    2579 generic.go:356] "Generic (PLEG): container finished" podID="669638a2-9a0d-40b3-bb81-558a6f82cb2b" containerID="e4300702d2382a4c84a145b095afc1a8decc81c545b69d7694c2693ebb5857ca" exitCode=0
Sep 19 04:55:33.961425 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:33.961392    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cacd12-779b59d64b-dqdcv"
Sep 19 04:55:33.961628 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:33.961430    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cacd12-779b59d64b-dqdcv" event={"ID":"669638a2-9a0d-40b3-bb81-558a6f82cb2b","Type":"ContainerDied","Data":"e4300702d2382a4c84a145b095afc1a8decc81c545b69d7694c2693ebb5857ca"}
Sep 19 04:55:33.961628 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:33.961493    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cacd12-779b59d64b-dqdcv" event={"ID":"669638a2-9a0d-40b3-bb81-558a6f82cb2b","Type":"ContainerDied","Data":"2daa730a680c706cac6ee13a315d402227a1e926ab3a16ae58a21dc10b59a87d"}
Sep 19 04:55:33.961628 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:33.961525    2579 scope.go:117] "RemoveContainer" containerID="e4300702d2382a4c84a145b095afc1a8decc81c545b69d7694c2693ebb5857ca"
Sep 19 04:55:33.971193 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:33.971177    2579 scope.go:117] "RemoveContainer" containerID="e4300702d2382a4c84a145b095afc1a8decc81c545b69d7694c2693ebb5857ca"
Sep 19 04:55:33.971449 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:55:33.971430    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4300702d2382a4c84a145b095afc1a8decc81c545b69d7694c2693ebb5857ca\": container with ID starting with e4300702d2382a4c84a145b095afc1a8decc81c545b69d7694c2693ebb5857ca not found: ID does not exist" containerID="e4300702d2382a4c84a145b095afc1a8decc81c545b69d7694c2693ebb5857ca"
Sep 19 04:55:33.971524 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:33.971458    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4300702d2382a4c84a145b095afc1a8decc81c545b69d7694c2693ebb5857ca"} err="failed to get container status \"e4300702d2382a4c84a145b095afc1a8decc81c545b69d7694c2693ebb5857ca\": rpc error: code = NotFound desc = could not find container \"e4300702d2382a4c84a145b095afc1a8decc81c545b69d7694c2693ebb5857ca\": container with ID starting with e4300702d2382a4c84a145b095afc1a8decc81c545b69d7694c2693ebb5857ca not found: ID does not exist"
Sep 19 04:55:33.973936 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:33.973920    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cctsv\" (UniqueName: \"kubernetes.io/projected/669638a2-9a0d-40b3-bb81-558a6f82cb2b-kube-api-access-cctsv\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:55:33.973992 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:33.973939    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/669638a2-9a0d-40b3-bb81-558a6f82cb2b-plugins-config-volume\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:55:33.987641 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:33.987617    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-cacd12-779b59d64b-dqdcv"]
Sep 19 04:55:33.997937 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:33.997910    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-cacd12-779b59d64b-dqdcv"]
Sep 19 04:55:35.351570 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:35.351535    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="669638a2-9a0d-40b3-bb81-558a6f82cb2b" path="/var/lib/kubelet/pods/669638a2-9a0d-40b3-bb81-558a6f82cb2b/volumes"
Sep 19 04:55:35.351966 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:35.351900    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7eeea1fc-67ce-4941-ad3d-06c39758867b" path="/var/lib/kubelet/pods/7eeea1fc-67ce-4941-ad3d-06c39758867b/volumes"
Sep 19 04:55:39.089176 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:39.089139    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-cacd12/ipp-route-cacd12-kserve-7f8864dcd4-ghfwf"]
Sep 19 04:55:54.109028 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:54.108961    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-cacd12/ipp-route-cacd12-kserve-7f8864dcd4-ghfwf" podUID="7ef7f683-8b60-4ce8-98fd-aa30062387d0" containerName="main" containerID="cri-o://ff11072bbe9ea46136d335ee9b45c9d435efee3a30c0040570b62a22ec3018cc" gracePeriod=45
Sep 19 04:55:54.367415 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:54.367351    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-cacd12/ipp-route-cacd12-kserve-7f8864dcd4-ghfwf"
Sep 19 04:55:54.480011 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:54.479973    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ef7f683-8b60-4ce8-98fd-aa30062387d0-tmp-dir\") pod \"7ef7f683-8b60-4ce8-98fd-aa30062387d0\" (UID: \"7ef7f683-8b60-4ce8-98fd-aa30062387d0\") "
Sep 19 04:55:54.480204 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:54.480021    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ef7f683-8b60-4ce8-98fd-aa30062387d0-home\") pod \"7ef7f683-8b60-4ce8-98fd-aa30062387d0\" (UID: \"7ef7f683-8b60-4ce8-98fd-aa30062387d0\") "
Sep 19 04:55:54.480204 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:54.480068    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ef7f683-8b60-4ce8-98fd-aa30062387d0-tls-certs\") pod \"7ef7f683-8b60-4ce8-98fd-aa30062387d0\" (UID: \"7ef7f683-8b60-4ce8-98fd-aa30062387d0\") "
Sep 19 04:55:54.480204 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:54.480140    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ef7f683-8b60-4ce8-98fd-aa30062387d0-model-cache\") pod \"7ef7f683-8b60-4ce8-98fd-aa30062387d0\" (UID: \"7ef7f683-8b60-4ce8-98fd-aa30062387d0\") "
Sep 19 04:55:54.480204 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:54.480156    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6hdt9\" (UniqueName: \"kubernetes.io/projected/7ef7f683-8b60-4ce8-98fd-aa30062387d0-kube-api-access-6hdt9\") pod \"7ef7f683-8b60-4ce8-98fd-aa30062387d0\" (UID: \"7ef7f683-8b60-4ce8-98fd-aa30062387d0\") "
Sep 19 04:55:54.480204 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:54.480178    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ef7f683-8b60-4ce8-98fd-aa30062387d0-dshm\") pod \"7ef7f683-8b60-4ce8-98fd-aa30062387d0\" (UID: \"7ef7f683-8b60-4ce8-98fd-aa30062387d0\") "
Sep 19 04:55:54.480479 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:54.480320    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ef7f683-8b60-4ce8-98fd-aa30062387d0-home" (OuterVolumeSpecName: "home") pod "7ef7f683-8b60-4ce8-98fd-aa30062387d0" (UID: "7ef7f683-8b60-4ce8-98fd-aa30062387d0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:55:54.480479 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:54.480332    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ef7f683-8b60-4ce8-98fd-aa30062387d0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7ef7f683-8b60-4ce8-98fd-aa30062387d0" (UID: "7ef7f683-8b60-4ce8-98fd-aa30062387d0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:55:54.480479 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:54.480435    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ef7f683-8b60-4ce8-98fd-aa30062387d0-model-cache" (OuterVolumeSpecName: "model-cache") pod "7ef7f683-8b60-4ce8-98fd-aa30062387d0" (UID: "7ef7f683-8b60-4ce8-98fd-aa30062387d0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:55:54.480635 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:54.480514    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ef7f683-8b60-4ce8-98fd-aa30062387d0-tmp-dir\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:55:54.480635 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:54.480535    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ef7f683-8b60-4ce8-98fd-aa30062387d0-home\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:55:54.482615 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:54.482573    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ef7f683-8b60-4ce8-98fd-aa30062387d0-kube-api-access-6hdt9" (OuterVolumeSpecName: "kube-api-access-6hdt9") pod "7ef7f683-8b60-4ce8-98fd-aa30062387d0" (UID: "7ef7f683-8b60-4ce8-98fd-aa30062387d0"). InnerVolumeSpecName "kube-api-access-6hdt9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 04:55:54.482772 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:54.482626    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ef7f683-8b60-4ce8-98fd-aa30062387d0-dshm" (OuterVolumeSpecName: "dshm") pod "7ef7f683-8b60-4ce8-98fd-aa30062387d0" (UID: "7ef7f683-8b60-4ce8-98fd-aa30062387d0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:55:54.482772 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:54.482664    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ef7f683-8b60-4ce8-98fd-aa30062387d0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7ef7f683-8b60-4ce8-98fd-aa30062387d0" (UID: "7ef7f683-8b60-4ce8-98fd-aa30062387d0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 04:55:54.582010 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:54.581977    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ef7f683-8b60-4ce8-98fd-aa30062387d0-tls-certs\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:55:54.582010 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:54.582007    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ef7f683-8b60-4ce8-98fd-aa30062387d0-model-cache\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:55:54.582010 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:54.582016    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hdt9\" (UniqueName: \"kubernetes.io/projected/7ef7f683-8b60-4ce8-98fd-aa30062387d0-kube-api-access-6hdt9\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:55:54.582319 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:54.582026    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ef7f683-8b60-4ce8-98fd-aa30062387d0-dshm\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:55:55.086861 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:55.086820    2579 generic.go:356] "Generic (PLEG): container finished" podID="7ef7f683-8b60-4ce8-98fd-aa30062387d0" containerID="ff11072bbe9ea46136d335ee9b45c9d435efee3a30c0040570b62a22ec3018cc" exitCode=0
Sep 19 04:55:55.086861 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:55.086869    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-cacd12/ipp-route-cacd12-kserve-7f8864dcd4-ghfwf"
Sep 19 04:55:55.087086 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:55.086903    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-cacd12/ipp-route-cacd12-kserve-7f8864dcd4-ghfwf" event={"ID":"7ef7f683-8b60-4ce8-98fd-aa30062387d0","Type":"ContainerDied","Data":"ff11072bbe9ea46136d335ee9b45c9d435efee3a30c0040570b62a22ec3018cc"}
Sep 19 04:55:55.087086 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:55.086955    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-cacd12/ipp-route-cacd12-kserve-7f8864dcd4-ghfwf" event={"ID":"7ef7f683-8b60-4ce8-98fd-aa30062387d0","Type":"ContainerDied","Data":"d1ba7e87943374fbcf559f9a32d627e0bf09beebfe726137f079f94ea1b06352"}
Sep 19 04:55:55.087086 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:55.086975    2579 scope.go:117] "RemoveContainer" containerID="ff11072bbe9ea46136d335ee9b45c9d435efee3a30c0040570b62a22ec3018cc"
Sep 19 04:55:55.096933 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:55.096916    2579 scope.go:117] "RemoveContainer" containerID="ff11072bbe9ea46136d335ee9b45c9d435efee3a30c0040570b62a22ec3018cc"
Sep 19 04:55:55.097225 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:55:55.097206    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff11072bbe9ea46136d335ee9b45c9d435efee3a30c0040570b62a22ec3018cc\": container with ID starting with ff11072bbe9ea46136d335ee9b45c9d435efee3a30c0040570b62a22ec3018cc not found: ID does not exist" containerID="ff11072bbe9ea46136d335ee9b45c9d435efee3a30c0040570b62a22ec3018cc"
Sep 19 04:55:55.097304 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:55.097238    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff11072bbe9ea46136d335ee9b45c9d435efee3a30c0040570b62a22ec3018cc"} err="failed to get container status \"ff11072bbe9ea46136d335ee9b45c9d435efee3a30c0040570b62a22ec3018cc\": rpc error: code = NotFound desc = could not find container \"ff11072bbe9ea46136d335ee9b45c9d435efee3a30c0040570b62a22ec3018cc\": container with ID starting with ff11072bbe9ea46136d335ee9b45c9d435efee3a30c0040570b62a22ec3018cc not found: ID does not exist"
Sep 19 04:55:55.111434 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:55.111414    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-cacd12/ipp-route-cacd12-kserve-7f8864dcd4-ghfwf"]
Sep 19 04:55:55.114936 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:55.114915    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-cacd12/ipp-route-cacd12-kserve-7f8864dcd4-ghfwf"]
Sep 19 04:55:55.322869 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:55.322824    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-cacd12/ipp-route-cacd12-kserve-7f8864dcd4-ghfwf" podUID="7ef7f683-8b60-4ce8-98fd-aa30062387d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.104:8000/ready\": context deadline exceeded"
Sep 19 04:55:55.351759 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:55:55.351657    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ef7f683-8b60-4ce8-98fd-aa30062387d0" path="/var/lib/kubelet/pods/7ef7f683-8b60-4ce8-98fd-aa30062387d0/volumes"
Sep 19 04:56:24.719576 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:24.719545    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-cacd12-openshift-default-7c68f65c4-gvfxl"]
Sep 19 04:56:24.720097 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:24.719788    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-cacd12-openshift-default-7c68f65c4-gvfxl" podUID="7ebcd658-999c-447c-addb-bce60fea8200" containerName="istio-proxy" containerID="cri-o://464ebedead1ee3481dd28315845b8b6cc4a73bf963aad730aa7e9196b7b7ea7f" gracePeriod=30
Sep 19 04:56:25.103267 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:25.103178    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-618e62-qfqbj"]
Sep 19 04:56:25.103600 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:25.103586    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ef7f683-8b60-4ce8-98fd-aa30062387d0" containerName="main"
Sep 19 04:56:25.103600 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:25.103601    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ef7f683-8b60-4ce8-98fd-aa30062387d0" containerName="main"
Sep 19 04:56:25.103714 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:25.103611    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="669638a2-9a0d-40b3-bb81-558a6f82cb2b" containerName="payload-pre-processing"
Sep 19 04:56:25.103714 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:25.103616    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="669638a2-9a0d-40b3-bb81-558a6f82cb2b" containerName="payload-pre-processing"
Sep 19 04:56:25.103714 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:25.103623    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7eeea1fc-67ce-4941-ad3d-06c39758867b" containerName="revoke-keys"
Sep 19 04:56:25.103714 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:25.103629    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7eeea1fc-67ce-4941-ad3d-06c39758867b" containerName="revoke-keys"
Sep 19 04:56:25.103714 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:25.103713    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ef7f683-8b60-4ce8-98fd-aa30062387d0" containerName="main"
Sep 19 04:56:25.103899 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:25.103722    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7eeea1fc-67ce-4941-ad3d-06c39758867b" containerName="revoke-keys"
Sep 19 04:56:25.103899 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:25.103729    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="669638a2-9a0d-40b3-bb81-558a6f82cb2b" containerName="payload-pre-processing"
Sep 19 04:56:25.106830 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:25.106813    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-618e62-qfqbj"
Sep 19 04:56:25.109431 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:25.109402    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s6xbq\""
Sep 19 04:56:25.109543 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:25.109498    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 19 04:56:25.119268 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:25.119239    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-618e62-qfqbj"]
Sep 19 04:56:25.176526 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:25.176481    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9513ab86-f9fe-4be9-b571-5d4916d699c6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-618e62-qfqbj\" (UID: \"9513ab86-f9fe-4be9-b571-5d4916d699c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-618e62-qfqbj"
Sep 19 04:56:25.277654 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:25.277624    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9513ab86-f9fe-4be9-b571-5d4916d699c6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-618e62-qfqbj\" (UID: \"9513ab86-f9fe-4be9-b571-5d4916d699c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-618e62-qfqbj"
Sep 19 04:56:25.278239 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:25.278221    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9513ab86-f9fe-4be9-b571-5d4916d699c6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-618e62-qfqbj\" (UID: \"9513ab86-f9fe-4be9-b571-5d4916d699c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-618e62-qfqbj"
Sep 19 04:56:25.419062 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:25.419017    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-618e62-qfqbj"
Sep 19 04:56:25.706544 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:25.706515    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-618e62-qfqbj"]
Sep 19 04:56:25.707448 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:56:25.707419    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9513ab86_f9fe_4be9_b571_5d4916d699c6.slice/crio-a0cd4eb4f698660ccf7b6766fd6f6ff64b93e94f7a89970de74167f0b8ce502c WatchSource:0}: Error finding container a0cd4eb4f698660ccf7b6766fd6f6ff64b93e94f7a89970de74167f0b8ce502c: Status 404 returned error can't find the container with id a0cd4eb4f698660ccf7b6766fd6f6ff64b93e94f7a89970de74167f0b8ce502c
Sep 19 04:56:26.270246 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:26.270153    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-618e62-qfqbj" event={"ID":"9513ab86-f9fe-4be9-b571-5d4916d699c6","Type":"ContainerStarted","Data":"2fb0860b54cbfaabac7d83ae13e52ed875e038c8063184fc8cf7b018175ffcfb"}
Sep 19 04:56:26.270246 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:26.270210    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-618e62-qfqbj" event={"ID":"9513ab86-f9fe-4be9-b571-5d4916d699c6","Type":"ContainerStarted","Data":"a0cd4eb4f698660ccf7b6766fd6f6ff64b93e94f7a89970de74167f0b8ce502c"}
Sep 19 04:56:27.284508 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:27.284467    2579 generic.go:356] "Generic (PLEG): container finished" podID="9513ab86-f9fe-4be9-b571-5d4916d699c6" containerID="2fb0860b54cbfaabac7d83ae13e52ed875e038c8063184fc8cf7b018175ffcfb" exitCode=0
Sep 19 04:56:27.284964 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:27.284568    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-618e62-qfqbj" event={"ID":"9513ab86-f9fe-4be9-b571-5d4916d699c6","Type":"ContainerDied","Data":"2fb0860b54cbfaabac7d83ae13e52ed875e038c8063184fc8cf7b018175ffcfb"}
Sep 19 04:56:28.431063 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:28.431040    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-618e62-qfqbj"
Sep 19 04:56:28.507857 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:28.507824    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9513ab86-f9fe-4be9-b571-5d4916d699c6-maas-api-service-ca\") pod \"9513ab86-f9fe-4be9-b571-5d4916d699c6\" (UID: \"9513ab86-f9fe-4be9-b571-5d4916d699c6\") "
Sep 19 04:56:28.508237 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:28.508211    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9513ab86-f9fe-4be9-b571-5d4916d699c6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9513ab86-f9fe-4be9-b571-5d4916d699c6" (UID: "9513ab86-f9fe-4be9-b571-5d4916d699c6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 04:56:28.609347 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:28.609258    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9513ab86-f9fe-4be9-b571-5d4916d699c6-maas-api-service-ca\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:56:29.296979 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:29.296940    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-618e62-qfqbj" event={"ID":"9513ab86-f9fe-4be9-b571-5d4916d699c6","Type":"ContainerDied","Data":"a0cd4eb4f698660ccf7b6766fd6f6ff64b93e94f7a89970de74167f0b8ce502c"}
Sep 19 04:56:29.297183 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:29.296998    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0cd4eb4f698660ccf7b6766fd6f6ff64b93e94f7a89970de74167f0b8ce502c"
Sep 19 04:56:29.297183 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:29.296947    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-618e62-qfqbj"
Sep 19 04:56:29.975038 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:29.975014    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-cacd12-openshift-default-7c68f65c4-gvfxl"
Sep 19 04:56:30.020785 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:30.020672    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ebcd658-999c-447c-addb-bce60fea8200-workload-socket\") pod \"7ebcd658-999c-447c-addb-bce60fea8200\" (UID: \"7ebcd658-999c-447c-addb-bce60fea8200\") "
Sep 19 04:56:30.020785 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:30.020762    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pp27n\" (UniqueName: \"kubernetes.io/projected/7ebcd658-999c-447c-addb-bce60fea8200-kube-api-access-pp27n\") pod \"7ebcd658-999c-447c-addb-bce60fea8200\" (UID: \"7ebcd658-999c-447c-addb-bce60fea8200\") "
Sep 19 04:56:30.021027 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:30.020790    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ebcd658-999c-447c-addb-bce60fea8200-istio-token\") pod \"7ebcd658-999c-447c-addb-bce60fea8200\" (UID: \"7ebcd658-999c-447c-addb-bce60fea8200\") "
Sep 19 04:56:30.021027 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:30.020806    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ebcd658-999c-447c-addb-bce60fea8200-istiod-ca-cert\") pod \"7ebcd658-999c-447c-addb-bce60fea8200\" (UID: \"7ebcd658-999c-447c-addb-bce60fea8200\") "
Sep 19 04:56:30.021027 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:30.020983    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ebcd658-999c-447c-addb-bce60fea8200-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7ebcd658-999c-447c-addb-bce60fea8200" (UID: "7ebcd658-999c-447c-addb-bce60fea8200"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:56:30.021265 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:30.021241    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ebcd658-999c-447c-addb-bce60fea8200-istio-podinfo\") pod \"7ebcd658-999c-447c-addb-bce60fea8200\" (UID: \"7ebcd658-999c-447c-addb-bce60fea8200\") "
Sep 19 04:56:30.021340 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:30.021315    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ebcd658-999c-447c-addb-bce60fea8200-istio-data\") pod \"7ebcd658-999c-447c-addb-bce60fea8200\" (UID: \"7ebcd658-999c-447c-addb-bce60fea8200\") "
Sep 19 04:56:30.021340 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:30.021321    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ebcd658-999c-447c-addb-bce60fea8200-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7ebcd658-999c-447c-addb-bce60fea8200" (UID: "7ebcd658-999c-447c-addb-bce60fea8200"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 04:56:30.021769 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:30.021743    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ebcd658-999c-447c-addb-bce60fea8200-istio-data" (OuterVolumeSpecName: "istio-data") pod "7ebcd658-999c-447c-addb-bce60fea8200" (UID: "7ebcd658-999c-447c-addb-bce60fea8200"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:56:30.021888 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:30.021851    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ebcd658-999c-447c-addb-bce60fea8200-credential-socket\") pod \"7ebcd658-999c-447c-addb-bce60fea8200\" (UID: \"7ebcd658-999c-447c-addb-bce60fea8200\") "
Sep 19 04:56:30.021956 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:30.021884    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ebcd658-999c-447c-addb-bce60fea8200-istio-envoy\") pod \"7ebcd658-999c-447c-addb-bce60fea8200\" (UID: \"7ebcd658-999c-447c-addb-bce60fea8200\") "
Sep 19 04:56:30.022017 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:30.021990    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ebcd658-999c-447c-addb-bce60fea8200-workload-certs\") pod \"7ebcd658-999c-447c-addb-bce60fea8200\" (UID: \"7ebcd658-999c-447c-addb-bce60fea8200\") "
Sep 19 04:56:30.022336 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:30.022305    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ebcd658-999c-447c-addb-bce60fea8200-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7ebcd658-999c-447c-addb-bce60fea8200" (UID: "7ebcd658-999c-447c-addb-bce60fea8200"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:56:30.022445 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:30.022395    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ebcd658-999c-447c-addb-bce60fea8200-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7ebcd658-999c-447c-addb-bce60fea8200" (UID: "7ebcd658-999c-447c-addb-bce60fea8200"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:56:30.022624 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:30.022604    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ebcd658-999c-447c-addb-bce60fea8200-workload-socket\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:56:30.022903 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:30.022627    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ebcd658-999c-447c-addb-bce60fea8200-istiod-ca-cert\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:56:30.022903 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:30.022641    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ebcd658-999c-447c-addb-bce60fea8200-istio-data\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:56:30.022903 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:30.022653    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ebcd658-999c-447c-addb-bce60fea8200-credential-socket\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:56:30.022903 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:30.022666    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ebcd658-999c-447c-addb-bce60fea8200-workload-certs\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:56:30.023469 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:30.023444    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ebcd658-999c-447c-addb-bce60fea8200-kube-api-access-pp27n" (OuterVolumeSpecName: "kube-api-access-pp27n") pod "7ebcd658-999c-447c-addb-bce60fea8200" (UID: "7ebcd658-999c-447c-addb-bce60fea8200"). InnerVolumeSpecName "kube-api-access-pp27n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 04:56:30.023469 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:30.023449    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ebcd658-999c-447c-addb-bce60fea8200-istio-token" (OuterVolumeSpecName: "istio-token") pod "7ebcd658-999c-447c-addb-bce60fea8200" (UID: "7ebcd658-999c-447c-addb-bce60fea8200"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 04:56:30.023755 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:30.023735    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ebcd658-999c-447c-addb-bce60fea8200-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7ebcd658-999c-447c-addb-bce60fea8200" (UID: "7ebcd658-999c-447c-addb-bce60fea8200"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 19 04:56:30.024277 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:30.024259    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ebcd658-999c-447c-addb-bce60fea8200-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7ebcd658-999c-447c-addb-bce60fea8200" (UID: "7ebcd658-999c-447c-addb-bce60fea8200"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:56:30.123882 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:30.123848    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pp27n\" (UniqueName: \"kubernetes.io/projected/7ebcd658-999c-447c-addb-bce60fea8200-kube-api-access-pp27n\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:56:30.123882 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:30.123878    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ebcd658-999c-447c-addb-bce60fea8200-istio-token\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:56:30.123882 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:30.123888    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ebcd658-999c-447c-addb-bce60fea8200-istio-podinfo\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:56:30.124108 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:30.123896    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ebcd658-999c-447c-addb-bce60fea8200-istio-envoy\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:56:30.304763 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:30.304666    2579 generic.go:356] "Generic (PLEG): container finished" podID="7ebcd658-999c-447c-addb-bce60fea8200" containerID="464ebedead1ee3481dd28315845b8b6cc4a73bf963aad730aa7e9196b7b7ea7f" exitCode=0
Sep 19 04:56:30.304763 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:30.304748    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-cacd12-openshift-default-7c68f65c4-gvfxl"
Sep 19 04:56:30.304763 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:30.304749    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-cacd12-openshift-default-7c68f65c4-gvfxl" event={"ID":"7ebcd658-999c-447c-addb-bce60fea8200","Type":"ContainerDied","Data":"464ebedead1ee3481dd28315845b8b6cc4a73bf963aad730aa7e9196b7b7ea7f"}
Sep 19 04:56:30.305042 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:30.304805    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-cacd12-openshift-default-7c68f65c4-gvfxl" event={"ID":"7ebcd658-999c-447c-addb-bce60fea8200","Type":"ContainerDied","Data":"cfc753617aa0f538aa7a56900bd6edc958940803bfbc74cf383cad7e147a8521"}
Sep 19 04:56:30.305042 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:30.304827    2579 scope.go:117] "RemoveContainer" containerID="464ebedead1ee3481dd28315845b8b6cc4a73bf963aad730aa7e9196b7b7ea7f"
Sep 19 04:56:30.314913 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:30.314894    2579 scope.go:117] "RemoveContainer" containerID="2620574af0c0f280cda3d2e0a331c09a963684e73184b4e73f1e7849877eb7f5"
Sep 19 04:56:30.323315 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:30.323294    2579 scope.go:117] "RemoveContainer" containerID="464ebedead1ee3481dd28315845b8b6cc4a73bf963aad730aa7e9196b7b7ea7f"
Sep 19 04:56:30.323602 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:56:30.323582    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"464ebedead1ee3481dd28315845b8b6cc4a73bf963aad730aa7e9196b7b7ea7f\": container with ID starting with 464ebedead1ee3481dd28315845b8b6cc4a73bf963aad730aa7e9196b7b7ea7f not found: ID does not exist" containerID="464ebedead1ee3481dd28315845b8b6cc4a73bf963aad730aa7e9196b7b7ea7f"
Sep 19 04:56:30.323678 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:30.323616    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"464ebedead1ee3481dd28315845b8b6cc4a73bf963aad730aa7e9196b7b7ea7f"} err="failed to get container status \"464ebedead1ee3481dd28315845b8b6cc4a73bf963aad730aa7e9196b7b7ea7f\": rpc error: code = NotFound desc = could not find container \"464ebedead1ee3481dd28315845b8b6cc4a73bf963aad730aa7e9196b7b7ea7f\": container with ID starting with 464ebedead1ee3481dd28315845b8b6cc4a73bf963aad730aa7e9196b7b7ea7f not found: ID does not exist"
Sep 19 04:56:30.323678 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:30.323642    2579 scope.go:117] "RemoveContainer" containerID="2620574af0c0f280cda3d2e0a331c09a963684e73184b4e73f1e7849877eb7f5"
Sep 19 04:56:30.323952 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:56:30.323933    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2620574af0c0f280cda3d2e0a331c09a963684e73184b4e73f1e7849877eb7f5\": container with ID starting with 2620574af0c0f280cda3d2e0a331c09a963684e73184b4e73f1e7849877eb7f5 not found: ID does not exist" containerID="2620574af0c0f280cda3d2e0a331c09a963684e73184b4e73f1e7849877eb7f5"
Sep 19 04:56:30.323993 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:30.323960    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2620574af0c0f280cda3d2e0a331c09a963684e73184b4e73f1e7849877eb7f5"} err="failed to get container status \"2620574af0c0f280cda3d2e0a331c09a963684e73184b4e73f1e7849877eb7f5\": rpc error: code = NotFound desc = could not find container \"2620574af0c0f280cda3d2e0a331c09a963684e73184b4e73f1e7849877eb7f5\": container with ID starting with 2620574af0c0f280cda3d2e0a331c09a963684e73184b4e73f1e7849877eb7f5 not found: ID does not exist"
Sep 19 04:56:30.328661 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:30.328640    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-cacd12-openshift-default-7c68f65c4-gvfxl"]
Sep 19 04:56:30.332263 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:30.332240    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-cacd12-openshift-default-7c68f65c4-gvfxl"]
Sep 19 04:56:31.351902 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:31.351874    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ebcd658-999c-447c-addb-bce60fea8200" path="/var/lib/kubelet/pods/7ebcd658-999c-447c-addb-bce60fea8200/volumes"
Sep 19 04:56:35.882569 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:35.882528    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-618e62-5ddf9bc6f5-pzsrq"]
Sep 19 04:56:35.883044 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:35.882784    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-618e62-5ddf9bc6f5-pzsrq" podUID="5772b706-6b5d-42cf-9f27-3d3cc187548c" containerName="payload-processing" containerID="cri-o://994979cf7eefe7fad60d776d5530af176817b256ff91ed5e132fcad8c2ac15ff" gracePeriod=30
Sep 19 04:56:36.152224 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:36.152199    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-618e62-5ddf9bc6f5-pzsrq"
Sep 19 04:56:36.233831 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:36.233794    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-618e62-qfqbj"]
Sep 19 04:56:36.246435 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:36.246403    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-618e62-qfqbj"]
Sep 19 04:56:36.282895 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:36.282862    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hjht5\" (UniqueName: \"kubernetes.io/projected/5772b706-6b5d-42cf-9f27-3d3cc187548c-kube-api-access-hjht5\") pod \"5772b706-6b5d-42cf-9f27-3d3cc187548c\" (UID: \"5772b706-6b5d-42cf-9f27-3d3cc187548c\") "
Sep 19 04:56:36.283075 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:36.282919    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5772b706-6b5d-42cf-9f27-3d3cc187548c-plugins-config-volume\") pod \"5772b706-6b5d-42cf-9f27-3d3cc187548c\" (UID: \"5772b706-6b5d-42cf-9f27-3d3cc187548c\") "
Sep 19 04:56:36.283338 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:36.283314    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5772b706-6b5d-42cf-9f27-3d3cc187548c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5772b706-6b5d-42cf-9f27-3d3cc187548c" (UID: "5772b706-6b5d-42cf-9f27-3d3cc187548c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 04:56:36.285355 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:36.285327    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5772b706-6b5d-42cf-9f27-3d3cc187548c-kube-api-access-hjht5" (OuterVolumeSpecName: "kube-api-access-hjht5") pod "5772b706-6b5d-42cf-9f27-3d3cc187548c" (UID: "5772b706-6b5d-42cf-9f27-3d3cc187548c"). InnerVolumeSpecName "kube-api-access-hjht5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 04:56:36.340091 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:36.340058    2579 generic.go:356] "Generic (PLEG): container finished" podID="5772b706-6b5d-42cf-9f27-3d3cc187548c" containerID="994979cf7eefe7fad60d776d5530af176817b256ff91ed5e132fcad8c2ac15ff" exitCode=0
Sep 19 04:56:36.340255 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:36.340101    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-618e62-5ddf9bc6f5-pzsrq"
Sep 19 04:56:36.340255 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:36.340140    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-618e62-5ddf9bc6f5-pzsrq" event={"ID":"5772b706-6b5d-42cf-9f27-3d3cc187548c","Type":"ContainerDied","Data":"994979cf7eefe7fad60d776d5530af176817b256ff91ed5e132fcad8c2ac15ff"}
Sep 19 04:56:36.340255 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:36.340187    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-618e62-5ddf9bc6f5-pzsrq" event={"ID":"5772b706-6b5d-42cf-9f27-3d3cc187548c","Type":"ContainerDied","Data":"db0d8bc57d9d996e4d57b404c6bdcec4bc2dc7850558f855dbe547948dc1e073"}
Sep 19 04:56:36.340255 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:36.340209    2579 scope.go:117] "RemoveContainer" containerID="994979cf7eefe7fad60d776d5530af176817b256ff91ed5e132fcad8c2ac15ff"
Sep 19 04:56:36.350159 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:36.350140    2579 scope.go:117] "RemoveContainer" containerID="994979cf7eefe7fad60d776d5530af176817b256ff91ed5e132fcad8c2ac15ff"
Sep 19 04:56:36.350461 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:56:36.350441    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"994979cf7eefe7fad60d776d5530af176817b256ff91ed5e132fcad8c2ac15ff\": container with ID starting with 994979cf7eefe7fad60d776d5530af176817b256ff91ed5e132fcad8c2ac15ff not found: ID does not exist" containerID="994979cf7eefe7fad60d776d5530af176817b256ff91ed5e132fcad8c2ac15ff"
Sep 19 04:56:36.350553 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:36.350469    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"994979cf7eefe7fad60d776d5530af176817b256ff91ed5e132fcad8c2ac15ff"} err="failed to get container status \"994979cf7eefe7fad60d776d5530af176817b256ff91ed5e132fcad8c2ac15ff\": rpc error: code = NotFound desc = could not find container \"994979cf7eefe7fad60d776d5530af176817b256ff91ed5e132fcad8c2ac15ff\": container with ID starting with 994979cf7eefe7fad60d776d5530af176817b256ff91ed5e132fcad8c2ac15ff not found: ID does not exist"
Sep 19 04:56:36.368875 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:36.368851    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-618e62-5ddf9bc6f5-pzsrq"]
Sep 19 04:56:36.374049 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:36.374026    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-618e62-5ddf9bc6f5-pzsrq"]
Sep 19 04:56:36.384121 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:36.384092    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjht5\" (UniqueName: \"kubernetes.io/projected/5772b706-6b5d-42cf-9f27-3d3cc187548c-kube-api-access-hjht5\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:56:36.384218 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:36.384122    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5772b706-6b5d-42cf-9f27-3d3cc187548c-plugins-config-volume\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:56:37.350885 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:37.350856    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5772b706-6b5d-42cf-9f27-3d3cc187548c" path="/var/lib/kubelet/pods/5772b706-6b5d-42cf-9f27-3d3cc187548c/volumes"
Sep 19 04:56:37.351269 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:37.351234    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9513ab86-f9fe-4be9-b571-5d4916d699c6" path="/var/lib/kubelet/pods/9513ab86-f9fe-4be9-b571-5d4916d699c6/volumes"
Sep 19 04:56:43.277270 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:43.277236    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-618e62-openshift-default-59dbc788bd-9qtw7"]
Sep 19 04:56:43.277713 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:43.277449    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-618e62-openshift-default-59dbc788bd-9qtw7" podUID="c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7" containerName="istio-proxy" containerID="cri-o://ee2012ca95eeca8aaa72f8916039481920b489062c463a9adc43aa54378834cf" gracePeriod=30
Sep 19 04:56:43.648415 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:43.648330    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-561bb1-dh6cl"]
Sep 19 04:56:43.648750 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:43.648735    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9513ab86-f9fe-4be9-b571-5d4916d699c6" containerName="revoke-keys"
Sep 19 04:56:43.648791 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:43.648751    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9513ab86-f9fe-4be9-b571-5d4916d699c6" containerName="revoke-keys"
Sep 19 04:56:43.648791 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:43.648758    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ebcd658-999c-447c-addb-bce60fea8200" containerName="istio-proxy"
Sep 19 04:56:43.648791 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:43.648763    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ebcd658-999c-447c-addb-bce60fea8200" containerName="istio-proxy"
Sep 19 04:56:43.648791 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:43.648777    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ebcd658-999c-447c-addb-bce60fea8200" containerName="istio-proxy"
Sep 19 04:56:43.648791 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:43.648782    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ebcd658-999c-447c-addb-bce60fea8200" containerName="istio-proxy"
Sep 19 04:56:43.648791 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:43.648791    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5772b706-6b5d-42cf-9f27-3d3cc187548c" containerName="payload-processing"
Sep 19 04:56:43.648975 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:43.648797    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5772b706-6b5d-42cf-9f27-3d3cc187548c" containerName="payload-processing"
Sep 19 04:56:43.648975 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:43.648862    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ebcd658-999c-447c-addb-bce60fea8200" containerName="istio-proxy"
Sep 19 04:56:43.648975 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:43.648872    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9513ab86-f9fe-4be9-b571-5d4916d699c6" containerName="revoke-keys"
Sep 19 04:56:43.648975 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:43.648878    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5772b706-6b5d-42cf-9f27-3d3cc187548c" containerName="payload-processing"
Sep 19 04:56:43.653470 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:43.653448    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-561bb1-dh6cl"
Sep 19 04:56:43.656223 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:43.656204    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s6xbq\""
Sep 19 04:56:43.656336 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:43.656211    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 19 04:56:43.661042 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:43.660880    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-561bb1-dh6cl"]
Sep 19 04:56:43.753171 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:43.753138    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd9136e3-9291-49de-9796-972cc243072d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-561bb1-dh6cl\" (UID: \"fd9136e3-9291-49de-9796-972cc243072d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-561bb1-dh6cl"
Sep 19 04:56:43.853658 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:43.853617    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd9136e3-9291-49de-9796-972cc243072d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-561bb1-dh6cl\" (UID: \"fd9136e3-9291-49de-9796-972cc243072d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-561bb1-dh6cl"
Sep 19 04:56:43.854312 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:43.854292    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd9136e3-9291-49de-9796-972cc243072d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-561bb1-dh6cl\" (UID: \"fd9136e3-9291-49de-9796-972cc243072d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-561bb1-dh6cl"
Sep 19 04:56:43.964584 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:43.964550    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-561bb1-dh6cl"
Sep 19 04:56:44.240305 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:44.240279    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-561bb1-dh6cl"]
Sep 19 04:56:44.242112 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:56:44.242077    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd9136e3_9291_49de_9796_972cc243072d.slice/crio-3fee93ba8f7077703b583f381d2bb65389697494d34b1289d73d7274e3ec47d9 WatchSource:0}: Error finding container 3fee93ba8f7077703b583f381d2bb65389697494d34b1289d73d7274e3ec47d9: Status 404 returned error can't find the container with id 3fee93ba8f7077703b583f381d2bb65389697494d34b1289d73d7274e3ec47d9
Sep 19 04:56:44.388085 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:44.388046    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-561bb1-dh6cl" event={"ID":"fd9136e3-9291-49de-9796-972cc243072d","Type":"ContainerStarted","Data":"3fee93ba8f7077703b583f381d2bb65389697494d34b1289d73d7274e3ec47d9"}
Sep 19 04:56:45.396506 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:45.396413    2579 generic.go:356] "Generic (PLEG): container finished" podID="fd9136e3-9291-49de-9796-972cc243072d" containerID="b491fa3b9bbdfe53d240d6b13e18edb0584eb09bb252bf5a573283d8732af4bb" exitCode=0
Sep 19 04:56:45.396876 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:45.396495    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-561bb1-dh6cl" event={"ID":"fd9136e3-9291-49de-9796-972cc243072d","Type":"ContainerDied","Data":"b491fa3b9bbdfe53d240d6b13e18edb0584eb09bb252bf5a573283d8732af4bb"}
Sep 19 04:56:46.543426 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:46.543400    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-561bb1-dh6cl"
Sep 19 04:56:46.683348 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:46.683269    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd9136e3-9291-49de-9796-972cc243072d-maas-api-service-ca\") pod \"fd9136e3-9291-49de-9796-972cc243072d\" (UID: \"fd9136e3-9291-49de-9796-972cc243072d\") "
Sep 19 04:56:46.683625 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:46.683603    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd9136e3-9291-49de-9796-972cc243072d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fd9136e3-9291-49de-9796-972cc243072d" (UID: "fd9136e3-9291-49de-9796-972cc243072d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 04:56:46.784496 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:46.784456    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd9136e3-9291-49de-9796-972cc243072d-maas-api-service-ca\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:56:47.409818 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:47.409780    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-561bb1-dh6cl"
Sep 19 04:56:47.409999 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:47.409777    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-561bb1-dh6cl" event={"ID":"fd9136e3-9291-49de-9796-972cc243072d","Type":"ContainerDied","Data":"3fee93ba8f7077703b583f381d2bb65389697494d34b1289d73d7274e3ec47d9"}
Sep 19 04:56:47.409999 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:47.409922    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3fee93ba8f7077703b583f381d2bb65389697494d34b1289d73d7274e3ec47d9"
Sep 19 04:56:48.418170 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:48.418136    2579 generic.go:356] "Generic (PLEG): container finished" podID="c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7" containerID="ee2012ca95eeca8aaa72f8916039481920b489062c463a9adc43aa54378834cf" exitCode=0
Sep 19 04:56:48.418744 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:48.418182    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-618e62-openshift-default-59dbc788bd-9qtw7" event={"ID":"c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7","Type":"ContainerDied","Data":"ee2012ca95eeca8aaa72f8916039481920b489062c463a9adc43aa54378834cf"}
Sep 19 04:56:48.418744 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:48.418240    2579 scope.go:117] "RemoveContainer" containerID="24211ea7d0e0bb58d4d45a6f0919f1ed36f4f9bc2cfeda13aba22b1d9ac089dc"
Sep 19 04:56:48.525303 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:48.525276    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-618e62-openshift-default-59dbc788bd-9qtw7"
Sep 19 04:56:48.702509 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:48.702474    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7-istio-token\") pod \"c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7\" (UID: \"c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7\") "
Sep 19 04:56:48.702737 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:48.702556    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bdd4n\" (UniqueName: \"kubernetes.io/projected/c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7-kube-api-access-bdd4n\") pod \"c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7\" (UID: \"c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7\") "
Sep 19 04:56:48.702737 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:48.702577    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7-istio-data\") pod \"c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7\" (UID: \"c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7\") "
Sep 19 04:56:48.702737 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:48.702608    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7-istio-envoy\") pod \"c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7\" (UID: \"c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7\") "
Sep 19 04:56:48.702737 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:48.702627    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7-workload-certs\") pod \"c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7\" (UID: \"c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7\") "
Sep 19 04:56:48.702737 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:48.702647    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7-credential-socket\") pod \"c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7\" (UID: \"c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7\") "
Sep 19 04:56:48.702737 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:48.702683    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7-istio-podinfo\") pod \"c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7\" (UID: \"c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7\") "
Sep 19 04:56:48.703065 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:48.702743    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7-istiod-ca-cert\") pod \"c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7\" (UID: \"c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7\") "
Sep 19 04:56:48.703065 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:48.702765    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7-workload-socket\") pod \"c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7\" (UID: \"c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7\") "
Sep 19 04:56:48.703065 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:48.703030    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7" (UID: "c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:56:48.703065 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:48.703016    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7-istio-data" (OuterVolumeSpecName: "istio-data") pod "c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7" (UID: "c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:56:48.703244 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:48.703215    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7" (UID: "c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:56:48.703299 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:48.703240    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7" (UID: "c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:56:48.703358 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:48.703329    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7" (UID: "c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 04:56:48.705009 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:48.704978    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7" (UID: "c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 04:56:48.705104 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:48.705038    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7-istio-token" (OuterVolumeSpecName: "istio-token") pod "c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7" (UID: "c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 04:56:48.705837 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:48.705813    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7-kube-api-access-bdd4n" (OuterVolumeSpecName: "kube-api-access-bdd4n") pod "c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7" (UID: "c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7"). InnerVolumeSpecName "kube-api-access-bdd4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 04:56:48.705918 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:48.705811    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7" (UID: "c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 19 04:56:48.804035 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:48.804003    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7-istiod-ca-cert\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:56:48.804035 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:48.804029    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7-workload-socket\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:56:48.804035 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:48.804038    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7-istio-token\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:56:48.804265 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:48.804047    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdd4n\" (UniqueName: \"kubernetes.io/projected/c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7-kube-api-access-bdd4n\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:56:48.804265 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:48.804057    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7-istio-data\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:56:48.804265 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:48.804064    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7-istio-envoy\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:56:48.804265 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:48.804072    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7-workload-certs\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:56:48.804265 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:48.804080    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7-credential-socket\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:56:48.804265 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:48.804088    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7-istio-podinfo\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:56:49.425578 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:49.425535    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-618e62-openshift-default-59dbc788bd-9qtw7" event={"ID":"c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7","Type":"ContainerDied","Data":"967639db3aa53ba056cad8336ebab0c38bb0b25ab40834ffb9c48663d747701b"}
Sep 19 04:56:49.426088 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:49.425591    2579 scope.go:117] "RemoveContainer" containerID="ee2012ca95eeca8aaa72f8916039481920b489062c463a9adc43aa54378834cf"
Sep 19 04:56:49.426088 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:49.425548    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-618e62-openshift-default-59dbc788bd-9qtw7"
Sep 19 04:56:49.445925 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:49.445889    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-618e62-openshift-default-59dbc788bd-9qtw7"]
Sep 19 04:56:49.449741 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:49.449718    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-618e62-openshift-default-59dbc788bd-9qtw7"]
Sep 19 04:56:51.351334 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:51.351297    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7" path="/var/lib/kubelet/pods/c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7/volumes"
Sep 19 04:56:54.470224 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:54.470178    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-561bb1-dh6cl"]
Sep 19 04:56:54.472447 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:54.472426    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-561bb1-dh6cl"]
Sep 19 04:56:55.351300 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:56:55.351258    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd9136e3-9291-49de-9796-972cc243072d" path="/var/lib/kubelet/pods/fd9136e3-9291-49de-9796-972cc243072d/volumes"
Sep 19 04:57:01.843980 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:57:01.843949    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8bbcca-k4vrv"]
Sep 19 04:57:01.844377 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:57:01.844346    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7" containerName="istio-proxy"
Sep 19 04:57:01.844377 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:57:01.844367    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7" containerName="istio-proxy"
Sep 19 04:57:01.844377 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:57:01.844376    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd9136e3-9291-49de-9796-972cc243072d" containerName="revoke-keys"
Sep 19 04:57:01.844481 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:57:01.844382    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd9136e3-9291-49de-9796-972cc243072d" containerName="revoke-keys"
Sep 19 04:57:01.844481 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:57:01.844389    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7" containerName="istio-proxy"
Sep 19 04:57:01.844481 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:57:01.844395    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7" containerName="istio-proxy"
Sep 19 04:57:01.844481 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:57:01.844459    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7" containerName="istio-proxy"
Sep 19 04:57:01.844481 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:57:01.844467    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd9136e3-9291-49de-9796-972cc243072d" containerName="revoke-keys"
Sep 19 04:57:01.844481 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:57:01.844474    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ebcd658-999c-447c-addb-bce60fea8200" containerName="istio-proxy"
Sep 19 04:57:01.848938 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:57:01.848919    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8bbcca-k4vrv"
Sep 19 04:57:01.852317 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:57:01.852291    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s6xbq\""
Sep 19 04:57:01.852436 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:57:01.852358    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 19 04:57:01.858779 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:57:01.858755    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8bbcca-k4vrv"]
Sep 19 04:57:01.911932 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:57:01.911899    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a24cd94-c033-4020-8eba-97bba7ac4b15-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-8bbcca-k4vrv\" (UID: \"8a24cd94-c033-4020-8eba-97bba7ac4b15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8bbcca-k4vrv"
Sep 19 04:57:02.013102 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:57:02.013061    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a24cd94-c033-4020-8eba-97bba7ac4b15-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-8bbcca-k4vrv\" (UID: \"8a24cd94-c033-4020-8eba-97bba7ac4b15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8bbcca-k4vrv"
Sep 19 04:57:02.013743 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:57:02.013687    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a24cd94-c033-4020-8eba-97bba7ac4b15-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-8bbcca-k4vrv\" (UID: \"8a24cd94-c033-4020-8eba-97bba7ac4b15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8bbcca-k4vrv"
Sep 19 04:57:02.160463 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:57:02.160429    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8bbcca-k4vrv"
Sep 19 04:57:02.454404 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:57:02.454374    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8bbcca-k4vrv"]
Sep 19 04:57:02.454955 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:57:02.454933    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a24cd94_c033_4020_8eba_97bba7ac4b15.slice/crio-888abf0b296f5825f2c07aacfa9634fb78a84a5fdfac43f6ec7eb8c0c1164601 WatchSource:0}: Error finding container 888abf0b296f5825f2c07aacfa9634fb78a84a5fdfac43f6ec7eb8c0c1164601: Status 404 returned error can't find the container with id 888abf0b296f5825f2c07aacfa9634fb78a84a5fdfac43f6ec7eb8c0c1164601
Sep 19 04:57:02.509742 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:57:02.509685    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8bbcca-k4vrv" event={"ID":"8a24cd94-c033-4020-8eba-97bba7ac4b15","Type":"ContainerStarted","Data":"888abf0b296f5825f2c07aacfa9634fb78a84a5fdfac43f6ec7eb8c0c1164601"}
Sep 19 04:57:03.516689 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:57:03.516648    2579 generic.go:356] "Generic (PLEG): container finished" podID="8a24cd94-c033-4020-8eba-97bba7ac4b15" containerID="49c3e5f406e54ab303992f395897dbf581edee0954222e2389ab5e95b10c3dba" exitCode=0
Sep 19 04:57:03.517078 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:57:03.516740    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8bbcca-k4vrv" event={"ID":"8a24cd94-c033-4020-8eba-97bba7ac4b15","Type":"ContainerDied","Data":"49c3e5f406e54ab303992f395897dbf581edee0954222e2389ab5e95b10c3dba"}
Sep 19 04:57:04.655535 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:57:04.655511    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8bbcca-k4vrv"
Sep 19 04:57:04.739489 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:57:04.739457    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a24cd94-c033-4020-8eba-97bba7ac4b15-maas-api-service-ca\") pod \"8a24cd94-c033-4020-8eba-97bba7ac4b15\" (UID: \"8a24cd94-c033-4020-8eba-97bba7ac4b15\") "
Sep 19 04:57:04.739856 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:57:04.739832    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a24cd94-c033-4020-8eba-97bba7ac4b15-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8a24cd94-c033-4020-8eba-97bba7ac4b15" (UID: "8a24cd94-c033-4020-8eba-97bba7ac4b15"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 04:57:04.840220 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:57:04.840140    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a24cd94-c033-4020-8eba-97bba7ac4b15-maas-api-service-ca\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:57:05.529948 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:57:05.529919    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8bbcca-k4vrv"
Sep 19 04:57:05.530113 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:57:05.529949    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8bbcca-k4vrv" event={"ID":"8a24cd94-c033-4020-8eba-97bba7ac4b15","Type":"ContainerDied","Data":"888abf0b296f5825f2c07aacfa9634fb78a84a5fdfac43f6ec7eb8c0c1164601"}
Sep 19 04:57:05.530113 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:57:05.529994    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="888abf0b296f5825f2c07aacfa9634fb78a84a5fdfac43f6ec7eb8c0c1164601"
Sep 19 04:57:12.429616 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:57:12.429588    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8bbcca-29829885-m47zz"]
Sep 19 04:57:12.437852 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:57:12.437824    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8bbcca-29829885-m47zz"]
Sep 19 04:57:12.478724 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:57:12.478675    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8bbcca-5c75fd676b-jc48f"]
Sep 19 04:57:12.478927 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:57:12.478890    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8bbcca-5c75fd676b-jc48f" podUID="aa764d71-eb0c-4967-b58c-a6fe70ccc4a9" containerName="payload-pre-processing" containerID="cri-o://9633d2df8d8fbcc5c6cc6af31db40f887d86e2d9b48ff95103e7e1f9cb088eb4" gracePeriod=30
Sep 19 04:57:12.697771 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:57:12.697662    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8bbcca-k4vrv"]
Sep 19 04:57:12.705866 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:57:12.705827    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8bbcca-k4vrv"]
Sep 19 04:57:12.746014 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:57:12.745989    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8bbcca-5c75fd676b-jc48f"
Sep 19 04:57:12.817851 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:57:12.817771    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pxp94\" (UniqueName: \"kubernetes.io/projected/aa764d71-eb0c-4967-b58c-a6fe70ccc4a9-kube-api-access-pxp94\") pod \"aa764d71-eb0c-4967-b58c-a6fe70ccc4a9\" (UID: \"aa764d71-eb0c-4967-b58c-a6fe70ccc4a9\") "
Sep 19 04:57:12.817851 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:57:12.817819    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa764d71-eb0c-4967-b58c-a6fe70ccc4a9-plugins-config-volume\") pod \"aa764d71-eb0c-4967-b58c-a6fe70ccc4a9\" (UID: \"aa764d71-eb0c-4967-b58c-a6fe70ccc4a9\") "
Sep 19 04:57:12.818368 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:57:12.818338    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa764d71-eb0c-4967-b58c-a6fe70ccc4a9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "aa764d71-eb0c-4967-b58c-a6fe70ccc4a9" (UID: "aa764d71-eb0c-4967-b58c-a6fe70ccc4a9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 04:57:12.820190 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:57:12.820153    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa764d71-eb0c-4967-b58c-a6fe70ccc4a9-kube-api-access-pxp94" (OuterVolumeSpecName: "kube-api-access-pxp94") pod "aa764d71-eb0c-4967-b58c-a6fe70ccc4a9" (UID: "aa764d71-eb0c-4967-b58c-a6fe70ccc4a9"). InnerVolumeSpecName "kube-api-access-pxp94". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 04:57:12.918896 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:57:12.918861    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxp94\" (UniqueName: \"kubernetes.io/projected/aa764d71-eb0c-4967-b58c-a6fe70ccc4a9-kube-api-access-pxp94\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:57:12.918896 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:57:12.918891    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa764d71-eb0c-4967-b58c-a6fe70ccc4a9-plugins-config-volume\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:57:13.350833 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:57:13.350797    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a24cd94-c033-4020-8eba-97bba7ac4b15" path="/var/lib/kubelet/pods/8a24cd94-c033-4020-8eba-97bba7ac4b15/volumes"
Sep 19 04:57:13.351096 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:57:13.351084    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c91ac636-b399-4ae7-a092-f7b3c533de75" path="/var/lib/kubelet/pods/c91ac636-b399-4ae7-a092-f7b3c533de75/volumes"
Sep 19 04:57:13.575938 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:57:13.575898    2579 generic.go:356] "Generic (PLEG): container finished" podID="aa764d71-eb0c-4967-b58c-a6fe70ccc4a9" containerID="9633d2df8d8fbcc5c6cc6af31db40f887d86e2d9b48ff95103e7e1f9cb088eb4" exitCode=0
Sep 19 04:57:13.576368 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:57:13.575935    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8bbcca-5c75fd676b-jc48f" event={"ID":"aa764d71-eb0c-4967-b58c-a6fe70ccc4a9","Type":"ContainerDied","Data":"9633d2df8d8fbcc5c6cc6af31db40f887d86e2d9b48ff95103e7e1f9cb088eb4"}
Sep 19 04:57:13.576368 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:57:13.575963    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8bbcca-5c75fd676b-jc48f"
Sep 19 04:57:13.576368 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:57:13.576024    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8bbcca-5c75fd676b-jc48f" event={"ID":"aa764d71-eb0c-4967-b58c-a6fe70ccc4a9","Type":"ContainerDied","Data":"5ff0ddbb091e0cbcdd0bbf5812b85c54adeb29728127bd79de179344d16703a1"}
Sep 19 04:57:13.576368 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:57:13.576048    2579 scope.go:117] "RemoveContainer" containerID="9633d2df8d8fbcc5c6cc6af31db40f887d86e2d9b48ff95103e7e1f9cb088eb4"
Sep 19 04:57:13.586640 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:57:13.586617    2579 scope.go:117] "RemoveContainer" containerID="9633d2df8d8fbcc5c6cc6af31db40f887d86e2d9b48ff95103e7e1f9cb088eb4"
Sep 19 04:57:13.587030 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:57:13.587010    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9633d2df8d8fbcc5c6cc6af31db40f887d86e2d9b48ff95103e7e1f9cb088eb4\": container with ID starting with 9633d2df8d8fbcc5c6cc6af31db40f887d86e2d9b48ff95103e7e1f9cb088eb4 not found: ID does not exist" containerID="9633d2df8d8fbcc5c6cc6af31db40f887d86e2d9b48ff95103e7e1f9cb088eb4"
Sep 19 04:57:13.587124 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:57:13.587038    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9633d2df8d8fbcc5c6cc6af31db40f887d86e2d9b48ff95103e7e1f9cb088eb4"} err="failed to get container status \"9633d2df8d8fbcc5c6cc6af31db40f887d86e2d9b48ff95103e7e1f9cb088eb4\": rpc error: code = NotFound desc = could not find container \"9633d2df8d8fbcc5c6cc6af31db40f887d86e2d9b48ff95103e7e1f9cb088eb4\": container with ID starting with 9633d2df8d8fbcc5c6cc6af31db40f887d86e2d9b48ff95103e7e1f9cb088eb4 not found: ID does not exist"
Sep 19 04:57:13.598255 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:57:13.598227    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8bbcca-5c75fd676b-jc48f"]
Sep 19 04:57:13.603085 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:57:13.603021    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8bbcca-5c75fd676b-jc48f"]
Sep 19 04:57:15.351330 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:57:15.351296    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa764d71-eb0c-4967-b58c-a6fe70ccc4a9" path="/var/lib/kubelet/pods/aa764d71-eb0c-4967-b58c-a6fe70ccc4a9/volumes"
Sep 19 04:57:22.655061 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:57:22.655022    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7bf95c4bb8-r9rcf"]
Sep 19 04:57:22.655647 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:57:22.655255    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7bf95c4bb8-r9rcf" podUID="6d5aa4b7-5d5f-4ce9-8ac5-79b0005ea2d3" containerName="manager" containerID="cri-o://b94a8accb5e0adac00948d60ace8a1b903a2ed8b08970ef9a2219e8467378699" gracePeriod=30
Sep 19 04:57:22.907454 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:57:22.907391    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7bf95c4bb8-r9rcf"
Sep 19 04:57:23.009850 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:57:23.009815    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bbcpt\" (UniqueName: \"kubernetes.io/projected/6d5aa4b7-5d5f-4ce9-8ac5-79b0005ea2d3-kube-api-access-bbcpt\") pod \"6d5aa4b7-5d5f-4ce9-8ac5-79b0005ea2d3\" (UID: \"6d5aa4b7-5d5f-4ce9-8ac5-79b0005ea2d3\") "
Sep 19 04:57:23.010034 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:57:23.009890    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d5aa4b7-5d5f-4ce9-8ac5-79b0005ea2d3-metrics-certs\") pod \"6d5aa4b7-5d5f-4ce9-8ac5-79b0005ea2d3\" (UID: \"6d5aa4b7-5d5f-4ce9-8ac5-79b0005ea2d3\") "
Sep 19 04:57:23.010034 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:57:23.009955    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6d5aa4b7-5d5f-4ce9-8ac5-79b0005ea2d3-webhook-cert\") pod \"6d5aa4b7-5d5f-4ce9-8ac5-79b0005ea2d3\" (UID: \"6d5aa4b7-5d5f-4ce9-8ac5-79b0005ea2d3\") "
Sep 19 04:57:23.011955 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:57:23.011927    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d5aa4b7-5d5f-4ce9-8ac5-79b0005ea2d3-kube-api-access-bbcpt" (OuterVolumeSpecName: "kube-api-access-bbcpt") pod "6d5aa4b7-5d5f-4ce9-8ac5-79b0005ea2d3" (UID: "6d5aa4b7-5d5f-4ce9-8ac5-79b0005ea2d3"). InnerVolumeSpecName "kube-api-access-bbcpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 04:57:23.012084 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:57:23.012072    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d5aa4b7-5d5f-4ce9-8ac5-79b0005ea2d3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6d5aa4b7-5d5f-4ce9-8ac5-79b0005ea2d3" (UID: "6d5aa4b7-5d5f-4ce9-8ac5-79b0005ea2d3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 04:57:23.012194 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:57:23.012170    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d5aa4b7-5d5f-4ce9-8ac5-79b0005ea2d3-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "6d5aa4b7-5d5f-4ce9-8ac5-79b0005ea2d3" (UID: "6d5aa4b7-5d5f-4ce9-8ac5-79b0005ea2d3"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 04:57:23.111108 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:57:23.111064    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbcpt\" (UniqueName: \"kubernetes.io/projected/6d5aa4b7-5d5f-4ce9-8ac5-79b0005ea2d3-kube-api-access-bbcpt\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:57:23.111108 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:57:23.111102    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d5aa4b7-5d5f-4ce9-8ac5-79b0005ea2d3-metrics-certs\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:57:23.111108 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:57:23.111113    2579 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6d5aa4b7-5d5f-4ce9-8ac5-79b0005ea2d3-webhook-cert\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:57:23.632390 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:57:23.632296    2579 generic.go:356] "Generic (PLEG): container finished" podID="6d5aa4b7-5d5f-4ce9-8ac5-79b0005ea2d3" containerID="b94a8accb5e0adac00948d60ace8a1b903a2ed8b08970ef9a2219e8467378699" exitCode=0
Sep 19 04:57:23.632390 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:57:23.632337    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7bf95c4bb8-r9rcf"
Sep 19 04:57:23.632390 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:57:23.632379    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7bf95c4bb8-r9rcf" event={"ID":"6d5aa4b7-5d5f-4ce9-8ac5-79b0005ea2d3","Type":"ContainerDied","Data":"b94a8accb5e0adac00948d60ace8a1b903a2ed8b08970ef9a2219e8467378699"}
Sep 19 04:57:23.632628 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:57:23.632430    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7bf95c4bb8-r9rcf" event={"ID":"6d5aa4b7-5d5f-4ce9-8ac5-79b0005ea2d3","Type":"ContainerDied","Data":"24f1117692d968d03ecf5636334f1cc1f9577295fc57aa840bd3b994ce9b087e"}
Sep 19 04:57:23.632628 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:57:23.632452    2579 scope.go:117] "RemoveContainer" containerID="b94a8accb5e0adac00948d60ace8a1b903a2ed8b08970ef9a2219e8467378699"
Sep 19 04:57:23.642475 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:57:23.642454    2579 scope.go:117] "RemoveContainer" containerID="b94a8accb5e0adac00948d60ace8a1b903a2ed8b08970ef9a2219e8467378699"
Sep 19 04:57:23.642763 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:57:23.642742    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b94a8accb5e0adac00948d60ace8a1b903a2ed8b08970ef9a2219e8467378699\": container with ID starting with b94a8accb5e0adac00948d60ace8a1b903a2ed8b08970ef9a2219e8467378699 not found: ID does not exist" containerID="b94a8accb5e0adac00948d60ace8a1b903a2ed8b08970ef9a2219e8467378699"
Sep 19 04:57:23.642824 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:57:23.642777    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b94a8accb5e0adac00948d60ace8a1b903a2ed8b08970ef9a2219e8467378699"} err="failed to get container status \"b94a8accb5e0adac00948d60ace8a1b903a2ed8b08970ef9a2219e8467378699\": rpc error: code = NotFound desc = could not find container \"b94a8accb5e0adac00948d60ace8a1b903a2ed8b08970ef9a2219e8467378699\": container with ID starting with b94a8accb5e0adac00948d60ace8a1b903a2ed8b08970ef9a2219e8467378699 not found: ID does not exist"
Sep 19 04:57:23.663609 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:57:23.663578    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7bf95c4bb8-r9rcf"]
Sep 19 04:57:23.679228 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:57:23.679196    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7bf95c4bb8-r9rcf"]
Sep 19 04:57:25.351558 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:57:25.351510    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d5aa4b7-5d5f-4ce9-8ac5-79b0005ea2d3" path="/var/lib/kubelet/pods/6d5aa4b7-5d5f-4ce9-8ac5-79b0005ea2d3/volumes"
Sep 19 04:57:42.030903 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:57:42.030870    2579 scope.go:117] "RemoveContainer" containerID="0da01db5c6bcddf2b3e4c65fe496523d17f729b037c0367d16032a0b01648f72"
Sep 19 04:58:24.488817 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:58:24.488782    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7bf95c4bb8-jgnwk"]
Sep 19 04:58:24.489247 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:58:24.489122    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d5aa4b7-5d5f-4ce9-8ac5-79b0005ea2d3" containerName="manager"
Sep 19 04:58:24.489247 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:58:24.489132    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d5aa4b7-5d5f-4ce9-8ac5-79b0005ea2d3" containerName="manager"
Sep 19 04:58:24.489247 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:58:24.489147    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa764d71-eb0c-4967-b58c-a6fe70ccc4a9" containerName="payload-pre-processing"
Sep 19 04:58:24.489247 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:58:24.489153    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa764d71-eb0c-4967-b58c-a6fe70ccc4a9" containerName="payload-pre-processing"
Sep 19 04:58:24.489247 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:58:24.489175    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a24cd94-c033-4020-8eba-97bba7ac4b15" containerName="revoke-keys"
Sep 19 04:58:24.489247 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:58:24.489182    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a24cd94-c033-4020-8eba-97bba7ac4b15" containerName="revoke-keys"
Sep 19 04:58:24.489247 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:58:24.489233    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d5aa4b7-5d5f-4ce9-8ac5-79b0005ea2d3" containerName="manager"
Sep 19 04:58:24.489247 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:58:24.489240    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a24cd94-c033-4020-8eba-97bba7ac4b15" containerName="revoke-keys"
Sep 19 04:58:24.489247 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:58:24.489245    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa764d71-eb0c-4967-b58c-a6fe70ccc4a9" containerName="payload-pre-processing"
Sep 19 04:58:24.489247 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:58:24.489251    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c19fb42b-e0e4-4122-ae6d-40d5b8bdf7d7" containerName="istio-proxy"
Sep 19 04:58:24.492155 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:58:24.492140    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7bf95c4bb8-jgnwk"
Sep 19 04:58:24.495970 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:58:24.495942    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 19 04:58:24.496114 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:58:24.495946    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-sfrkl\""
Sep 19 04:58:24.496114 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:58:24.495947    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 19 04:58:24.496114 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:58:24.495947    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 19 04:58:24.505376 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:58:24.505357    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7bf95c4bb8-jgnwk"]
Sep 19 04:58:24.674943 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:58:24.674912    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b93962ce-6a54-47a9-9f02-5663970ef50b-webhook-cert\") pod \"maas-controller-7bf95c4bb8-jgnwk\" (UID: \"b93962ce-6a54-47a9-9f02-5663970ef50b\") " pod="opendatahub/maas-controller-7bf95c4bb8-jgnwk"
Sep 19 04:58:24.675113 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:58:24.674975    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b93962ce-6a54-47a9-9f02-5663970ef50b-metrics-certs\") pod \"maas-controller-7bf95c4bb8-jgnwk\" (UID: \"b93962ce-6a54-47a9-9f02-5663970ef50b\") " pod="opendatahub/maas-controller-7bf95c4bb8-jgnwk"
Sep 19 04:58:24.675113 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:58:24.675022    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sp5zw\" (UniqueName: \"kubernetes.io/projected/b93962ce-6a54-47a9-9f02-5663970ef50b-kube-api-access-sp5zw\") pod \"maas-controller-7bf95c4bb8-jgnwk\" (UID: \"b93962ce-6a54-47a9-9f02-5663970ef50b\") " pod="opendatahub/maas-controller-7bf95c4bb8-jgnwk"
Sep 19 04:58:24.776001 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:58:24.775914    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b93962ce-6a54-47a9-9f02-5663970ef50b-metrics-certs\") pod \"maas-controller-7bf95c4bb8-jgnwk\" (UID: \"b93962ce-6a54-47a9-9f02-5663970ef50b\") " pod="opendatahub/maas-controller-7bf95c4bb8-jgnwk"
Sep 19 04:58:24.776001 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:58:24.775963    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sp5zw\" (UniqueName: \"kubernetes.io/projected/b93962ce-6a54-47a9-9f02-5663970ef50b-kube-api-access-sp5zw\") pod \"maas-controller-7bf95c4bb8-jgnwk\" (UID: \"b93962ce-6a54-47a9-9f02-5663970ef50b\") " pod="opendatahub/maas-controller-7bf95c4bb8-jgnwk"
Sep 19 04:58:24.776203 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:58:24.776029    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b93962ce-6a54-47a9-9f02-5663970ef50b-webhook-cert\") pod \"maas-controller-7bf95c4bb8-jgnwk\" (UID: \"b93962ce-6a54-47a9-9f02-5663970ef50b\") " pod="opendatahub/maas-controller-7bf95c4bb8-jgnwk"
Sep 19 04:58:24.778609 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:58:24.778575    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b93962ce-6a54-47a9-9f02-5663970ef50b-webhook-cert\") pod \"maas-controller-7bf95c4bb8-jgnwk\" (UID: \"b93962ce-6a54-47a9-9f02-5663970ef50b\") " pod="opendatahub/maas-controller-7bf95c4bb8-jgnwk"
Sep 19 04:58:24.778609 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:58:24.778598    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b93962ce-6a54-47a9-9f02-5663970ef50b-metrics-certs\") pod \"maas-controller-7bf95c4bb8-jgnwk\" (UID: \"b93962ce-6a54-47a9-9f02-5663970ef50b\") " pod="opendatahub/maas-controller-7bf95c4bb8-jgnwk"
Sep 19 04:58:24.785244 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:58:24.785219    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sp5zw\" (UniqueName: \"kubernetes.io/projected/b93962ce-6a54-47a9-9f02-5663970ef50b-kube-api-access-sp5zw\") pod \"maas-controller-7bf95c4bb8-jgnwk\" (UID: \"b93962ce-6a54-47a9-9f02-5663970ef50b\") " pod="opendatahub/maas-controller-7bf95c4bb8-jgnwk"
Sep 19 04:58:24.802592 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:58:24.802565    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7bf95c4bb8-jgnwk"
Sep 19 04:58:25.078059 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:58:25.077972    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7bf95c4bb8-jgnwk"]
Sep 19 04:58:25.081031 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:58:25.081001    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb93962ce_6a54_47a9_9f02_5663970ef50b.slice/crio-bff2370d907eb107b320b602ddc084356699bffe9650656ad403e97897eb9326 WatchSource:0}: Error finding container bff2370d907eb107b320b602ddc084356699bffe9650656ad403e97897eb9326: Status 404 returned error can't find the container with id bff2370d907eb107b320b602ddc084356699bffe9650656ad403e97897eb9326
Sep 19 04:58:25.082306 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:58:25.082284    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 19 04:58:25.981686 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:58:25.981643    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7bf95c4bb8-jgnwk" event={"ID":"b93962ce-6a54-47a9-9f02-5663970ef50b","Type":"ContainerStarted","Data":"bcd82878b4896a90566b48441103377f5306389ad1fb95219052ccff6a4c9dd8"}
Sep 19 04:58:25.982186 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:58:25.981777    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7bf95c4bb8-jgnwk" event={"ID":"b93962ce-6a54-47a9-9f02-5663970ef50b","Type":"ContainerStarted","Data":"bff2370d907eb107b320b602ddc084356699bffe9650656ad403e97897eb9326"}
Sep 19 04:58:26.004130 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:58:26.004073    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7bf95c4bb8-jgnwk" podStartSLOduration=1.584288115 podStartE2EDuration="2.004055073s" totalImagesPullingTime="419.766958ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 04:58:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:58:26.002750588 +0000 UTC m=+2685.185711737" watchObservedRunningTime="2026-09-19 04:58:26.004055073 +0000 UTC m=+2685.187016212"
Sep 19 04:58:26.987134 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:58:26.987101    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7bf95c4bb8-jgnwk"
Sep 19 04:58:37.994767 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:58:37.994667    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7bf95c4bb8-jgnwk"
Sep 19 04:58:45.275970 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:58:45.275934    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7bf95c4bb8-jgnwk"]
Sep 19 04:58:45.276491 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:58:45.276160    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7bf95c4bb8-jgnwk" podUID="b93962ce-6a54-47a9-9f02-5663970ef50b" containerName="manager" containerID="cri-o://bcd82878b4896a90566b48441103377f5306389ad1fb95219052ccff6a4c9dd8" gracePeriod=30
Sep 19 04:58:45.519444 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:58:45.519422    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7bf95c4bb8-jgnwk"
Sep 19 04:58:45.560477 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:58:45.560389    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b93962ce-6a54-47a9-9f02-5663970ef50b-webhook-cert\") pod \"b93962ce-6a54-47a9-9f02-5663970ef50b\" (UID: \"b93962ce-6a54-47a9-9f02-5663970ef50b\") "
Sep 19 04:58:45.560477 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:58:45.560455    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sp5zw\" (UniqueName: \"kubernetes.io/projected/b93962ce-6a54-47a9-9f02-5663970ef50b-kube-api-access-sp5zw\") pod \"b93962ce-6a54-47a9-9f02-5663970ef50b\" (UID: \"b93962ce-6a54-47a9-9f02-5663970ef50b\") "
Sep 19 04:58:45.560726 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:58:45.560533    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b93962ce-6a54-47a9-9f02-5663970ef50b-metrics-certs\") pod \"b93962ce-6a54-47a9-9f02-5663970ef50b\" (UID: \"b93962ce-6a54-47a9-9f02-5663970ef50b\") "
Sep 19 04:58:45.562729 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:58:45.562680    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b93962ce-6a54-47a9-9f02-5663970ef50b-kube-api-access-sp5zw" (OuterVolumeSpecName: "kube-api-access-sp5zw") pod "b93962ce-6a54-47a9-9f02-5663970ef50b" (UID: "b93962ce-6a54-47a9-9f02-5663970ef50b"). InnerVolumeSpecName "kube-api-access-sp5zw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 04:58:45.562729 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:58:45.562680    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b93962ce-6a54-47a9-9f02-5663970ef50b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b93962ce-6a54-47a9-9f02-5663970ef50b" (UID: "b93962ce-6a54-47a9-9f02-5663970ef50b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 04:58:45.562888 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:58:45.562774    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b93962ce-6a54-47a9-9f02-5663970ef50b-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "b93962ce-6a54-47a9-9f02-5663970ef50b" (UID: "b93962ce-6a54-47a9-9f02-5663970ef50b"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 04:58:45.661993 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:58:45.661962    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b93962ce-6a54-47a9-9f02-5663970ef50b-metrics-certs\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:58:45.661993 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:58:45.661988    2579 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b93962ce-6a54-47a9-9f02-5663970ef50b-webhook-cert\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:58:45.661993 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:58:45.661997    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sp5zw\" (UniqueName: \"kubernetes.io/projected/b93962ce-6a54-47a9-9f02-5663970ef50b-kube-api-access-sp5zw\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 04:58:46.093474 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:58:46.093444    2579 generic.go:356] "Generic (PLEG): container finished" podID="b93962ce-6a54-47a9-9f02-5663970ef50b" containerID="bcd82878b4896a90566b48441103377f5306389ad1fb95219052ccff6a4c9dd8" exitCode=0
Sep 19 04:58:46.093660 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:58:46.093489    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7bf95c4bb8-jgnwk"
Sep 19 04:58:46.093660 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:58:46.093530    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7bf95c4bb8-jgnwk" event={"ID":"b93962ce-6a54-47a9-9f02-5663970ef50b","Type":"ContainerDied","Data":"bcd82878b4896a90566b48441103377f5306389ad1fb95219052ccff6a4c9dd8"}
Sep 19 04:58:46.093660 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:58:46.093586    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7bf95c4bb8-jgnwk" event={"ID":"b93962ce-6a54-47a9-9f02-5663970ef50b","Type":"ContainerDied","Data":"bff2370d907eb107b320b602ddc084356699bffe9650656ad403e97897eb9326"}
Sep 19 04:58:46.093660 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:58:46.093606    2579 scope.go:117] "RemoveContainer" containerID="bcd82878b4896a90566b48441103377f5306389ad1fb95219052ccff6a4c9dd8"
Sep 19 04:58:46.103140 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:58:46.103121    2579 scope.go:117] "RemoveContainer" containerID="bcd82878b4896a90566b48441103377f5306389ad1fb95219052ccff6a4c9dd8"
Sep 19 04:58:46.103390 ip-10-0-129-215 kubenswrapper[2579]: E0919 04:58:46.103372    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcd82878b4896a90566b48441103377f5306389ad1fb95219052ccff6a4c9dd8\": container with ID starting with bcd82878b4896a90566b48441103377f5306389ad1fb95219052ccff6a4c9dd8 not found: ID does not exist" containerID="bcd82878b4896a90566b48441103377f5306389ad1fb95219052ccff6a4c9dd8"
Sep 19 04:58:46.103438 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:58:46.103399    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcd82878b4896a90566b48441103377f5306389ad1fb95219052ccff6a4c9dd8"} err="failed to get container status \"bcd82878b4896a90566b48441103377f5306389ad1fb95219052ccff6a4c9dd8\": rpc error: code = NotFound desc = could not find container \"bcd82878b4896a90566b48441103377f5306389ad1fb95219052ccff6a4c9dd8\": container with ID starting with bcd82878b4896a90566b48441103377f5306389ad1fb95219052ccff6a4c9dd8 not found: ID does not exist"
Sep 19 04:58:46.123238 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:58:46.123205    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7bf95c4bb8-jgnwk"]
Sep 19 04:58:46.125810 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:58:46.125781    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7bf95c4bb8-jgnwk"]
Sep 19 04:58:47.351262 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:58:47.351229    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b93962ce-6a54-47a9-9f02-5663970ef50b" path="/var/lib/kubelet/pods/b93962ce-6a54-47a9-9f02-5663970ef50b/volumes"
Sep 19 04:58:47.388480 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:58:47.388444    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7bf95c4bb8-c4zv7"]
Sep 19 04:58:47.388893 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:58:47.388877    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b93962ce-6a54-47a9-9f02-5663970ef50b" containerName="manager"
Sep 19 04:58:47.388893 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:58:47.388895    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b93962ce-6a54-47a9-9f02-5663970ef50b" containerName="manager"
Sep 19 04:58:47.389026 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:58:47.388976    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b93962ce-6a54-47a9-9f02-5663970ef50b" containerName="manager"
Sep 19 04:58:47.393633 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:58:47.393607    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7bf95c4bb8-c4zv7"
Sep 19 04:58:47.397289 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:58:47.397268    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 19 04:58:47.397416 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:58:47.397301    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 19 04:58:47.397416 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:58:47.397276    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-sfrkl\""
Sep 19 04:58:47.397532 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:58:47.397269    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 19 04:58:47.404083 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:58:47.404060    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7bf95c4bb8-c4zv7"]
Sep 19 04:58:47.477673 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:58:47.477640    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6775f09-fcfa-44c6-b8a8-2087e6eab0f9-metrics-certs\") pod \"maas-controller-7bf95c4bb8-c4zv7\" (UID: \"a6775f09-fcfa-44c6-b8a8-2087e6eab0f9\") " pod="opendatahub/maas-controller-7bf95c4bb8-c4zv7"
Sep 19 04:58:47.477851 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:58:47.477713    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a6775f09-fcfa-44c6-b8a8-2087e6eab0f9-webhook-cert\") pod \"maas-controller-7bf95c4bb8-c4zv7\" (UID: \"a6775f09-fcfa-44c6-b8a8-2087e6eab0f9\") " pod="opendatahub/maas-controller-7bf95c4bb8-c4zv7"
Sep 19 04:58:47.477851 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:58:47.477790    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-249r6\" (UniqueName: \"kubernetes.io/projected/a6775f09-fcfa-44c6-b8a8-2087e6eab0f9-kube-api-access-249r6\") pod \"maas-controller-7bf95c4bb8-c4zv7\" (UID: \"a6775f09-fcfa-44c6-b8a8-2087e6eab0f9\") " pod="opendatahub/maas-controller-7bf95c4bb8-c4zv7"
Sep 19 04:58:47.578550 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:58:47.578522    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6775f09-fcfa-44c6-b8a8-2087e6eab0f9-metrics-certs\") pod \"maas-controller-7bf95c4bb8-c4zv7\" (UID: \"a6775f09-fcfa-44c6-b8a8-2087e6eab0f9\") " pod="opendatahub/maas-controller-7bf95c4bb8-c4zv7"
Sep 19 04:58:47.578749 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:58:47.578564    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a6775f09-fcfa-44c6-b8a8-2087e6eab0f9-webhook-cert\") pod \"maas-controller-7bf95c4bb8-c4zv7\" (UID: \"a6775f09-fcfa-44c6-b8a8-2087e6eab0f9\") " pod="opendatahub/maas-controller-7bf95c4bb8-c4zv7"
Sep 19 04:58:47.578749 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:58:47.578635    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-249r6\" (UniqueName: \"kubernetes.io/projected/a6775f09-fcfa-44c6-b8a8-2087e6eab0f9-kube-api-access-249r6\") pod \"maas-controller-7bf95c4bb8-c4zv7\" (UID: \"a6775f09-fcfa-44c6-b8a8-2087e6eab0f9\") " pod="opendatahub/maas-controller-7bf95c4bb8-c4zv7"
Sep 19 04:58:47.581255 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:58:47.581231    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a6775f09-fcfa-44c6-b8a8-2087e6eab0f9-webhook-cert\") pod \"maas-controller-7bf95c4bb8-c4zv7\" (UID: \"a6775f09-fcfa-44c6-b8a8-2087e6eab0f9\") " pod="opendatahub/maas-controller-7bf95c4bb8-c4zv7"
Sep 19 04:58:47.581356 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:58:47.581231    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6775f09-fcfa-44c6-b8a8-2087e6eab0f9-metrics-certs\") pod \"maas-controller-7bf95c4bb8-c4zv7\" (UID: \"a6775f09-fcfa-44c6-b8a8-2087e6eab0f9\") " pod="opendatahub/maas-controller-7bf95c4bb8-c4zv7"
Sep 19 04:58:47.588128 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:58:47.588100    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-249r6\" (UniqueName: \"kubernetes.io/projected/a6775f09-fcfa-44c6-b8a8-2087e6eab0f9-kube-api-access-249r6\") pod \"maas-controller-7bf95c4bb8-c4zv7\" (UID: \"a6775f09-fcfa-44c6-b8a8-2087e6eab0f9\") " pod="opendatahub/maas-controller-7bf95c4bb8-c4zv7"
Sep 19 04:58:47.704481 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:58:47.704447    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7bf95c4bb8-c4zv7"
Sep 19 04:58:47.980373 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:58:47.980342    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7bf95c4bb8-c4zv7"]
Sep 19 04:58:47.982556 ip-10-0-129-215 kubenswrapper[2579]: W0919 04:58:47.982525    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6775f09_fcfa_44c6_b8a8_2087e6eab0f9.slice/crio-6877555233d49adaa1d676ffd5b4a95453142efe4e30f25c660155731e390553 WatchSource:0}: Error finding container 6877555233d49adaa1d676ffd5b4a95453142efe4e30f25c660155731e390553: Status 404 returned error can't find the container with id 6877555233d49adaa1d676ffd5b4a95453142efe4e30f25c660155731e390553
Sep 19 04:58:48.107113 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:58:48.107078    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7bf95c4bb8-c4zv7" event={"ID":"a6775f09-fcfa-44c6-b8a8-2087e6eab0f9","Type":"ContainerStarted","Data":"6877555233d49adaa1d676ffd5b4a95453142efe4e30f25c660155731e390553"}
Sep 19 04:58:49.114712 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:58:49.114648    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7bf95c4bb8-c4zv7" event={"ID":"a6775f09-fcfa-44c6-b8a8-2087e6eab0f9","Type":"ContainerStarted","Data":"6f77b7bcd5c8b7eccd3a43182e9cbf71d81c8346f5fb136f1f19c40ea83c5807"}
Sep 19 04:58:49.115172 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:58:49.114762    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7bf95c4bb8-c4zv7"
Sep 19 04:58:49.134868 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:58:49.134824    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7bf95c4bb8-c4zv7" podStartSLOduration=1.775817506 podStartE2EDuration="2.13481029s" totalImagesPullingTime="358.992784ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 04:58:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 04:58:49.132060636 +0000 UTC m=+2708.315021783" watchObservedRunningTime="2026-09-19 04:58:49.13481029 +0000 UTC m=+2708.317771437"
Sep 19 04:59:00.123235 ip-10-0-129-215 kubenswrapper[2579]: I0919 04:59:00.123204    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7bf95c4bb8-c4zv7"
Sep 19 05:00:00.171929 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:00:00.171890    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29829900-f29rm"]
Sep 19 05:00:00.175391 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:00:00.175375    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829900-f29rm"
Sep 19 05:00:00.178433 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:00:00.178410    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s6xbq\""
Sep 19 05:00:00.186006 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:00:00.185976    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829900-f29rm"
Sep 19 05:00:00.186197 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:00:00.186124    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29829900-f29rm"]
Sep 19 05:00:00.465011 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:00:00.464986    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29829900-f29rm"]
Sep 19 05:00:00.466899 ip-10-0-129-215 kubenswrapper[2579]: W0919 05:00:00.466872    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7e7effc_d1c2_43e5_8bbc_f7699be8ab12.slice/crio-da1c4df144d746f84a99dd70f2ed8918bb5a33a1af2b88046aa252cb27421c0b WatchSource:0}: Error finding container da1c4df144d746f84a99dd70f2ed8918bb5a33a1af2b88046aa252cb27421c0b: Status 404 returned error can't find the container with id da1c4df144d746f84a99dd70f2ed8918bb5a33a1af2b88046aa252cb27421c0b
Sep 19 05:00:00.509728 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:00:00.509669    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829900-f29rm" event={"ID":"a7e7effc-d1c2-43e5-8bbc-f7699be8ab12","Type":"ContainerStarted","Data":"da1c4df144d746f84a99dd70f2ed8918bb5a33a1af2b88046aa252cb27421c0b"}
Sep 19 05:00:01.517250 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:00:01.517205    2579 generic.go:356] "Generic (PLEG): container finished" podID="a7e7effc-d1c2-43e5-8bbc-f7699be8ab12" containerID="d35ecd90c963db592ddd436411f48e43aaa713323e1f1bef87b8f4faffefe9e6" exitCode=0
Sep 19 05:00:01.517728 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:00:01.517286    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829900-f29rm" event={"ID":"a7e7effc-d1c2-43e5-8bbc-f7699be8ab12","Type":"ContainerDied","Data":"d35ecd90c963db592ddd436411f48e43aaa713323e1f1bef87b8f4faffefe9e6"}
Sep 19 05:00:02.666170 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:00:02.666144    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829900-f29rm"
Sep 19 05:00:03.530546 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:00:03.530515    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829900-f29rm"
Sep 19 05:00:03.530783 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:00:03.530507    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829900-f29rm" event={"ID":"a7e7effc-d1c2-43e5-8bbc-f7699be8ab12","Type":"ContainerDied","Data":"da1c4df144d746f84a99dd70f2ed8918bb5a33a1af2b88046aa252cb27421c0b"}
Sep 19 05:00:03.530783 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:00:03.530729    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da1c4df144d746f84a99dd70f2ed8918bb5a33a1af2b88046aa252cb27421c0b"
Sep 19 05:00:28.184343 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:00:28.184303    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-z2g72"]
Sep 19 05:00:28.184839 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:00:28.184524    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-z2g72" podUID="d2cb2792-51a4-4365-95f5-de0f536efd2c" containerName="manager" containerID="cri-o://f08e74eb0cda9f00f6e9f0f4341bce6864bb32f8d0b020582fdc21234c2ee654" gracePeriod=10
Sep 19 05:00:31.693402 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:00:31.693370    2579 generic.go:356] "Generic (PLEG): container finished" podID="d2cb2792-51a4-4365-95f5-de0f536efd2c" containerID="f08e74eb0cda9f00f6e9f0f4341bce6864bb32f8d0b020582fdc21234c2ee654" exitCode=0
Sep 19 05:00:31.693857 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:00:31.693435    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-z2g72" event={"ID":"d2cb2792-51a4-4365-95f5-de0f536efd2c","Type":"ContainerDied","Data":"f08e74eb0cda9f00f6e9f0f4341bce6864bb32f8d0b020582fdc21234c2ee654"}
Sep 19 05:00:31.693857 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:00:31.693498    2579 scope.go:117] "RemoveContainer" containerID="68de29a2166259ee1c510a5a49c0e986026e078e83b7b550650c3b46be061e68"
Sep 19 05:00:31.727203 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:00:31.727177    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-z2g72"
Sep 19 05:00:31.851493 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:00:31.851405    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d2cb2792-51a4-4365-95f5-de0f536efd2c-extensions-socket-volume\") pod \"d2cb2792-51a4-4365-95f5-de0f536efd2c\" (UID: \"d2cb2792-51a4-4365-95f5-de0f536efd2c\") "
Sep 19 05:00:31.851724 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:00:31.851500    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ttx4j\" (UniqueName: \"kubernetes.io/projected/d2cb2792-51a4-4365-95f5-de0f536efd2c-kube-api-access-ttx4j\") pod \"d2cb2792-51a4-4365-95f5-de0f536efd2c\" (UID: \"d2cb2792-51a4-4365-95f5-de0f536efd2c\") "
Sep 19 05:00:31.851894 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:00:31.851870    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2cb2792-51a4-4365-95f5-de0f536efd2c-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "d2cb2792-51a4-4365-95f5-de0f536efd2c" (UID: "d2cb2792-51a4-4365-95f5-de0f536efd2c"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 05:00:31.853888 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:00:31.853870    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2cb2792-51a4-4365-95f5-de0f536efd2c-kube-api-access-ttx4j" (OuterVolumeSpecName: "kube-api-access-ttx4j") pod "d2cb2792-51a4-4365-95f5-de0f536efd2c" (UID: "d2cb2792-51a4-4365-95f5-de0f536efd2c"). InnerVolumeSpecName "kube-api-access-ttx4j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 05:00:31.952851 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:00:31.952815    2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d2cb2792-51a4-4365-95f5-de0f536efd2c-extensions-socket-volume\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 05:00:31.952851 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:00:31.952847    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttx4j\" (UniqueName: \"kubernetes.io/projected/d2cb2792-51a4-4365-95f5-de0f536efd2c-kube-api-access-ttx4j\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 05:00:32.700712 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:00:32.700672    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-z2g72"
Sep 19 05:00:32.701185 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:00:32.700719    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-z2g72" event={"ID":"d2cb2792-51a4-4365-95f5-de0f536efd2c","Type":"ContainerDied","Data":"dfa61e43c6b05c651acfedbaa7b5a5e03ecf1aa19c33118dbbcb211936572160"}
Sep 19 05:00:32.701185 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:00:32.700785    2579 scope.go:117] "RemoveContainer" containerID="f08e74eb0cda9f00f6e9f0f4341bce6864bb32f8d0b020582fdc21234c2ee654"
Sep 19 05:00:32.748483 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:00:32.748448    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-z2g72"]
Sep 19 05:00:32.759428 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:00:32.759395    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-z2g72"]
Sep 19 05:00:33.352612 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:00:33.352581    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2cb2792-51a4-4365-95f5-de0f536efd2c" path="/var/lib/kubelet/pods/d2cb2792-51a4-4365-95f5-de0f536efd2c/volumes"
Sep 19 05:01:28.872217 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:01:28.872181    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-zs5ck"]
Sep 19 05:01:28.872650 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:01:28.872573    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2cb2792-51a4-4365-95f5-de0f536efd2c" containerName="manager"
Sep 19 05:01:28.872650 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:01:28.872588    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2cb2792-51a4-4365-95f5-de0f536efd2c" containerName="manager"
Sep 19 05:01:28.872650 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:01:28.872601    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2cb2792-51a4-4365-95f5-de0f536efd2c" containerName="manager"
Sep 19 05:01:28.872650 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:01:28.872607    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2cb2792-51a4-4365-95f5-de0f536efd2c" containerName="manager"
Sep 19 05:01:28.872650 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:01:28.872617    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7e7effc-d1c2-43e5-8bbc-f7699be8ab12" containerName="cleanup"
Sep 19 05:01:28.872650 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:01:28.872622    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7e7effc-d1c2-43e5-8bbc-f7699be8ab12" containerName="cleanup"
Sep 19 05:01:28.872880 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:01:28.872680    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2cb2792-51a4-4365-95f5-de0f536efd2c" containerName="manager"
Sep 19 05:01:28.872880 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:01:28.872689    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7e7effc-d1c2-43e5-8bbc-f7699be8ab12" containerName="cleanup"
Sep 19 05:01:28.875649 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:01:28.875634    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-zs5ck"
Sep 19 05:01:28.878721 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:01:28.878673    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-sffsr\""
Sep 19 05:01:28.900152 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:01:28.900130    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-zs5ck"]
Sep 19 05:01:28.945105 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:01:28.945067    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f670455e-7c9e-4e4f-a8bb-7c28a26a5c3f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-54b6d5c989-zs5ck\" (UID: \"f670455e-7c9e-4e4f-a8bb-7c28a26a5c3f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-zs5ck"
Sep 19 05:01:28.945281 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:01:28.945154    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdqd9\" (UniqueName: \"kubernetes.io/projected/f670455e-7c9e-4e4f-a8bb-7c28a26a5c3f-kube-api-access-fdqd9\") pod \"kuadrant-operator-controller-manager-54b6d5c989-zs5ck\" (UID: \"f670455e-7c9e-4e4f-a8bb-7c28a26a5c3f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-zs5ck"
Sep 19 05:01:29.046124 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:01:29.046087    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f670455e-7c9e-4e4f-a8bb-7c28a26a5c3f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-54b6d5c989-zs5ck\" (UID: \"f670455e-7c9e-4e4f-a8bb-7c28a26a5c3f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-zs5ck"
Sep 19 05:01:29.046328 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:01:29.046141    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdqd9\" (UniqueName: \"kubernetes.io/projected/f670455e-7c9e-4e4f-a8bb-7c28a26a5c3f-kube-api-access-fdqd9\") pod \"kuadrant-operator-controller-manager-54b6d5c989-zs5ck\" (UID: \"f670455e-7c9e-4e4f-a8bb-7c28a26a5c3f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-zs5ck"
Sep 19 05:01:29.046535 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:01:29.046510    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f670455e-7c9e-4e4f-a8bb-7c28a26a5c3f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-54b6d5c989-zs5ck\" (UID: \"f670455e-7c9e-4e4f-a8bb-7c28a26a5c3f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-zs5ck"
Sep 19 05:01:29.064469 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:01:29.064437    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdqd9\" (UniqueName: \"kubernetes.io/projected/f670455e-7c9e-4e4f-a8bb-7c28a26a5c3f-kube-api-access-fdqd9\") pod \"kuadrant-operator-controller-manager-54b6d5c989-zs5ck\" (UID: \"f670455e-7c9e-4e4f-a8bb-7c28a26a5c3f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-zs5ck"
Sep 19 05:01:29.185244 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:01:29.185211    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-zs5ck"
Sep 19 05:01:29.475233 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:01:29.475205    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-zs5ck"]
Sep 19 05:01:29.478074 ip-10-0-129-215 kubenswrapper[2579]: W0919 05:01:29.478043    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf670455e_7c9e_4e4f_a8bb_7c28a26a5c3f.slice/crio-54c4e86ded827e40d2348e711d31251a781b06ea20565ac3ef62df3fdd42115c WatchSource:0}: Error finding container 54c4e86ded827e40d2348e711d31251a781b06ea20565ac3ef62df3fdd42115c: Status 404 returned error can't find the container with id 54c4e86ded827e40d2348e711d31251a781b06ea20565ac3ef62df3fdd42115c
Sep 19 05:01:30.011527 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:01:30.011492    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-zs5ck" event={"ID":"f670455e-7c9e-4e4f-a8bb-7c28a26a5c3f","Type":"ContainerStarted","Data":"5baa43d1cb0bdfc3c204fe4ba05aa9749940e14bc182399b3198cdb0ced9ea6f"}
Sep 19 05:01:30.012051 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:01:30.011543    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-zs5ck" event={"ID":"f670455e-7c9e-4e4f-a8bb-7c28a26a5c3f","Type":"ContainerStarted","Data":"54c4e86ded827e40d2348e711d31251a781b06ea20565ac3ef62df3fdd42115c"}
Sep 19 05:01:30.012051 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:01:30.011635    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-zs5ck"
Sep 19 05:01:30.038841 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:01:30.038790    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-zs5ck" podStartSLOduration=2.03877412 podStartE2EDuration="2.03877412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 05:01:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 05:01:30.037235172 +0000 UTC m=+2869.220196320" watchObservedRunningTime="2026-09-19 05:01:30.03877412 +0000 UTC m=+2869.221735326"
Sep 19 05:01:41.019974 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:01:41.019895    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-zs5ck"
Sep 19 05:01:42.099166 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:01:42.099128    2579 scope.go:117] "RemoveContainer" containerID="da14780c5020afc6c734717f8ffada8af51d7e4a9ceb4a9a77025f89fc4a3e5a"
Sep 19 05:01:42.107454 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:01:42.107425    2579 scope.go:117] "RemoveContainer" containerID="d5efbea920b31b8efcf063e8e34c9ca29ccf1016310f9411666f228a37b106da"
Sep 19 05:02:26.992349 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:02:26.992308    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-e8e1b070-kserve-796494657b-66564"]
Sep 19 05:02:26.992829 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:02:26.992816    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2cb2792-51a4-4365-95f5-de0f536efd2c" containerName="manager"
Sep 19 05:02:26.995934 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:02:26.995910    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-e8e1b070-kserve-796494657b-66564"
Sep 19 05:02:27.003885 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:02:27.003855    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-e8e1b070-kserve-self-signed-certs\""
Sep 19 05:02:27.017419 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:02:27.017395    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-e8e1b070-kserve-796494657b-66564"]
Sep 19 05:02:27.078088 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:02:27.078058    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88c35b46-f78c-4739-9434-4d07d4f07eb2-tls-certs\") pod \"e2e-same-id-a-e8e1b070-kserve-796494657b-66564\" (UID: \"88c35b46-f78c-4739-9434-4d07d4f07eb2\") " pod="llm/e2e-same-id-a-e8e1b070-kserve-796494657b-66564"
Sep 19 05:02:27.078255 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:02:27.078099    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88c35b46-f78c-4739-9434-4d07d4f07eb2-model-cache\") pod \"e2e-same-id-a-e8e1b070-kserve-796494657b-66564\" (UID: \"88c35b46-f78c-4739-9434-4d07d4f07eb2\") " pod="llm/e2e-same-id-a-e8e1b070-kserve-796494657b-66564"
Sep 19 05:02:27.078255 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:02:27.078162    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88c35b46-f78c-4739-9434-4d07d4f07eb2-dshm\") pod \"e2e-same-id-a-e8e1b070-kserve-796494657b-66564\" (UID: \"88c35b46-f78c-4739-9434-4d07d4f07eb2\") " pod="llm/e2e-same-id-a-e8e1b070-kserve-796494657b-66564"
Sep 19 05:02:27.078255 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:02:27.078233    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88c35b46-f78c-4739-9434-4d07d4f07eb2-tmp-dir\") pod \"e2e-same-id-a-e8e1b070-kserve-796494657b-66564\" (UID: \"88c35b46-f78c-4739-9434-4d07d4f07eb2\") " pod="llm/e2e-same-id-a-e8e1b070-kserve-796494657b-66564"
Sep 19 05:02:27.078255 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:02:27.078252    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88c35b46-f78c-4739-9434-4d07d4f07eb2-home\") pod \"e2e-same-id-a-e8e1b070-kserve-796494657b-66564\" (UID: \"88c35b46-f78c-4739-9434-4d07d4f07eb2\") " pod="llm/e2e-same-id-a-e8e1b070-kserve-796494657b-66564"
Sep 19 05:02:27.078416 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:02:27.078304    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ws9f6\" (UniqueName: \"kubernetes.io/projected/88c35b46-f78c-4739-9434-4d07d4f07eb2-kube-api-access-ws9f6\") pod \"e2e-same-id-a-e8e1b070-kserve-796494657b-66564\" (UID: \"88c35b46-f78c-4739-9434-4d07d4f07eb2\") " pod="llm/e2e-same-id-a-e8e1b070-kserve-796494657b-66564"
Sep 19 05:02:27.179777 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:02:27.179740    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88c35b46-f78c-4739-9434-4d07d4f07eb2-tmp-dir\") pod \"e2e-same-id-a-e8e1b070-kserve-796494657b-66564\" (UID: \"88c35b46-f78c-4739-9434-4d07d4f07eb2\") " pod="llm/e2e-same-id-a-e8e1b070-kserve-796494657b-66564"
Sep 19 05:02:27.179777 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:02:27.179779    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88c35b46-f78c-4739-9434-4d07d4f07eb2-home\") pod \"e2e-same-id-a-e8e1b070-kserve-796494657b-66564\" (UID: \"88c35b46-f78c-4739-9434-4d07d4f07eb2\") " pod="llm/e2e-same-id-a-e8e1b070-kserve-796494657b-66564"
Sep 19 05:02:27.180059 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:02:27.179808    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ws9f6\" (UniqueName: \"kubernetes.io/projected/88c35b46-f78c-4739-9434-4d07d4f07eb2-kube-api-access-ws9f6\") pod \"e2e-same-id-a-e8e1b070-kserve-796494657b-66564\" (UID: \"88c35b46-f78c-4739-9434-4d07d4f07eb2\") " pod="llm/e2e-same-id-a-e8e1b070-kserve-796494657b-66564"
Sep 19 05:02:27.180059 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:02:27.179834    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88c35b46-f78c-4739-9434-4d07d4f07eb2-tls-certs\") pod \"e2e-same-id-a-e8e1b070-kserve-796494657b-66564\" (UID: \"88c35b46-f78c-4739-9434-4d07d4f07eb2\") " pod="llm/e2e-same-id-a-e8e1b070-kserve-796494657b-66564"
Sep 19 05:02:27.180059 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:02:27.179855    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88c35b46-f78c-4739-9434-4d07d4f07eb2-model-cache\") pod \"e2e-same-id-a-e8e1b070-kserve-796494657b-66564\" (UID: \"88c35b46-f78c-4739-9434-4d07d4f07eb2\") " pod="llm/e2e-same-id-a-e8e1b070-kserve-796494657b-66564"
Sep 19 05:02:27.180059 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:02:27.179895    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88c35b46-f78c-4739-9434-4d07d4f07eb2-dshm\") pod \"e2e-same-id-a-e8e1b070-kserve-796494657b-66564\" (UID: \"88c35b46-f78c-4739-9434-4d07d4f07eb2\") " pod="llm/e2e-same-id-a-e8e1b070-kserve-796494657b-66564"
Sep 19 05:02:27.180272 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:02:27.180235    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88c35b46-f78c-4739-9434-4d07d4f07eb2-tmp-dir\") pod \"e2e-same-id-a-e8e1b070-kserve-796494657b-66564\" (UID: \"88c35b46-f78c-4739-9434-4d07d4f07eb2\") " pod="llm/e2e-same-id-a-e8e1b070-kserve-796494657b-66564"
Sep 19 05:02:27.180318 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:02:27.180265    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88c35b46-f78c-4739-9434-4d07d4f07eb2-home\") pod \"e2e-same-id-a-e8e1b070-kserve-796494657b-66564\" (UID: \"88c35b46-f78c-4739-9434-4d07d4f07eb2\") " pod="llm/e2e-same-id-a-e8e1b070-kserve-796494657b-66564"
Sep 19 05:02:27.180318 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:02:27.180277    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88c35b46-f78c-4739-9434-4d07d4f07eb2-model-cache\") pod \"e2e-same-id-a-e8e1b070-kserve-796494657b-66564\" (UID: \"88c35b46-f78c-4739-9434-4d07d4f07eb2\") " pod="llm/e2e-same-id-a-e8e1b070-kserve-796494657b-66564"
Sep 19 05:02:27.182518 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:02:27.182495    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88c35b46-f78c-4739-9434-4d07d4f07eb2-dshm\") pod \"e2e-same-id-a-e8e1b070-kserve-796494657b-66564\" (UID: \"88c35b46-f78c-4739-9434-4d07d4f07eb2\") " pod="llm/e2e-same-id-a-e8e1b070-kserve-796494657b-66564"
Sep 19 05:02:27.182945 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:02:27.182923    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88c35b46-f78c-4739-9434-4d07d4f07eb2-tls-certs\") pod \"e2e-same-id-a-e8e1b070-kserve-796494657b-66564\" (UID: \"88c35b46-f78c-4739-9434-4d07d4f07eb2\") " pod="llm/e2e-same-id-a-e8e1b070-kserve-796494657b-66564"
Sep 19 05:02:27.189267 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:02:27.189238    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws9f6\" (UniqueName: \"kubernetes.io/projected/88c35b46-f78c-4739-9434-4d07d4f07eb2-kube-api-access-ws9f6\") pod \"e2e-same-id-a-e8e1b070-kserve-796494657b-66564\" (UID: \"88c35b46-f78c-4739-9434-4d07d4f07eb2\") " pod="llm/e2e-same-id-a-e8e1b070-kserve-796494657b-66564"
Sep 19 05:02:27.304716 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:02:27.304600    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-e8e1b070-kserve-796494657b-66564"
Sep 19 05:02:27.582666 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:02:27.582614    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-e8e1b070-kserve-796494657b-66564"]
Sep 19 05:02:27.583720 ip-10-0-129-215 kubenswrapper[2579]: W0919 05:02:27.583678    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod88c35b46_f78c_4739_9434_4d07d4f07eb2.slice/crio-f8b6902e1d01ee80fd31319eb66fceb6efe190dbae209812cea6e6fceadbe65a WatchSource:0}: Error finding container f8b6902e1d01ee80fd31319eb66fceb6efe190dbae209812cea6e6fceadbe65a: Status 404 returned error can't find the container with id f8b6902e1d01ee80fd31319eb66fceb6efe190dbae209812cea6e6fceadbe65a
Sep 19 05:02:28.332577 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:02:28.332538    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-e8e1b070-kserve-796494657b-66564" event={"ID":"88c35b46-f78c-4739-9434-4d07d4f07eb2","Type":"ContainerStarted","Data":"10cc895527e1bc4d6370f0908d8db2a01514243b49bf15983576b148d644e5f7"}
Sep 19 05:02:28.332577 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:02:28.332589    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-e8e1b070-kserve-796494657b-66564" event={"ID":"88c35b46-f78c-4739-9434-4d07d4f07eb2","Type":"ContainerStarted","Data":"f8b6902e1d01ee80fd31319eb66fceb6efe190dbae209812cea6e6fceadbe65a"}
Sep 19 05:02:28.354897 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:02:28.354847    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-e8e1b070-kserve-796494657b-66564" podStartSLOduration=2.354833015 podStartE2EDuration="2.354833015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 05:02:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 05:02:28.35267801 +0000 UTC m=+2927.535639160" watchObservedRunningTime="2026-09-19 05:02:28.354833015 +0000 UTC m=+2927.537794164"
Sep 19 05:02:29.305418 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:02:29.305380    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-e8e1b070-kserve-796494657b-66564"
Sep 19 05:02:36.796766 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:02:36.796730    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-e8e1b070-kserve-6978979bd4-llv4l"]
Sep 19 05:02:36.800376 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:02:36.800353    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-e8e1b070-kserve-6978979bd4-llv4l"
Sep 19 05:02:36.804819 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:02:36.804799    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-e8e1b070-kserve-self-signed-certs\""
Sep 19 05:02:36.812933 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:02:36.812910    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-e8e1b070-kserve-6978979bd4-llv4l"]
Sep 19 05:02:36.975132 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:02:36.975091    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7008e915-113e-46fb-92fa-003c595561ca-model-cache\") pod \"e2e-same-id-b-e8e1b070-kserve-6978979bd4-llv4l\" (UID: \"7008e915-113e-46fb-92fa-003c595561ca\") " pod="llm/e2e-same-id-b-e8e1b070-kserve-6978979bd4-llv4l"
Sep 19 05:02:36.975132 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:02:36.975139    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7008e915-113e-46fb-92fa-003c595561ca-dshm\") pod \"e2e-same-id-b-e8e1b070-kserve-6978979bd4-llv4l\" (UID: \"7008e915-113e-46fb-92fa-003c595561ca\") " pod="llm/e2e-same-id-b-e8e1b070-kserve-6978979bd4-llv4l"
Sep 19 05:02:36.975397 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:02:36.975199    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7nbs\" (UniqueName: \"kubernetes.io/projected/7008e915-113e-46fb-92fa-003c595561ca-kube-api-access-z7nbs\") pod \"e2e-same-id-b-e8e1b070-kserve-6978979bd4-llv4l\" (UID: \"7008e915-113e-46fb-92fa-003c595561ca\") " pod="llm/e2e-same-id-b-e8e1b070-kserve-6978979bd4-llv4l"
Sep 19 05:02:36.975397 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:02:36.975245    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7008e915-113e-46fb-92fa-003c595561ca-tls-certs\") pod \"e2e-same-id-b-e8e1b070-kserve-6978979bd4-llv4l\" (UID: \"7008e915-113e-46fb-92fa-003c595561ca\") " pod="llm/e2e-same-id-b-e8e1b070-kserve-6978979bd4-llv4l"
Sep 19 05:02:36.975397 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:02:36.975338    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7008e915-113e-46fb-92fa-003c595561ca-tmp-dir\") pod \"e2e-same-id-b-e8e1b070-kserve-6978979bd4-llv4l\" (UID: \"7008e915-113e-46fb-92fa-003c595561ca\") " pod="llm/e2e-same-id-b-e8e1b070-kserve-6978979bd4-llv4l"
Sep 19 05:02:36.975397 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:02:36.975388    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7008e915-113e-46fb-92fa-003c595561ca-home\") pod \"e2e-same-id-b-e8e1b070-kserve-6978979bd4-llv4l\" (UID: \"7008e915-113e-46fb-92fa-003c595561ca\") " pod="llm/e2e-same-id-b-e8e1b070-kserve-6978979bd4-llv4l"
Sep 19 05:02:37.076534 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:02:37.076449    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7008e915-113e-46fb-92fa-003c595561ca-dshm\") pod \"e2e-same-id-b-e8e1b070-kserve-6978979bd4-llv4l\" (UID: \"7008e915-113e-46fb-92fa-003c595561ca\") " pod="llm/e2e-same-id-b-e8e1b070-kserve-6978979bd4-llv4l"
Sep 19 05:02:37.076534 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:02:37.076488    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7nbs\" (UniqueName: \"kubernetes.io/projected/7008e915-113e-46fb-92fa-003c595561ca-kube-api-access-z7nbs\") pod \"e2e-same-id-b-e8e1b070-kserve-6978979bd4-llv4l\" (UID: \"7008e915-113e-46fb-92fa-003c595561ca\") " pod="llm/e2e-same-id-b-e8e1b070-kserve-6978979bd4-llv4l"
Sep 19 05:02:37.076534 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:02:37.076513    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7008e915-113e-46fb-92fa-003c595561ca-tls-certs\") pod \"e2e-same-id-b-e8e1b070-kserve-6978979bd4-llv4l\" (UID: \"7008e915-113e-46fb-92fa-003c595561ca\") " pod="llm/e2e-same-id-b-e8e1b070-kserve-6978979bd4-llv4l"
Sep 19 05:02:37.076872 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:02:37.076548    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7008e915-113e-46fb-92fa-003c595561ca-tmp-dir\") pod \"e2e-same-id-b-e8e1b070-kserve-6978979bd4-llv4l\" (UID: \"7008e915-113e-46fb-92fa-003c595561ca\") " pod="llm/e2e-same-id-b-e8e1b070-kserve-6978979bd4-llv4l"
Sep 19 05:02:37.076872 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:02:37.076570    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7008e915-113e-46fb-92fa-003c595561ca-home\") pod \"e2e-same-id-b-e8e1b070-kserve-6978979bd4-llv4l\" (UID: \"7008e915-113e-46fb-92fa-003c595561ca\") " pod="llm/e2e-same-id-b-e8e1b070-kserve-6978979bd4-llv4l"
Sep 19 05:02:37.076872 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:02:37.076598    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7008e915-113e-46fb-92fa-003c595561ca-model-cache\") pod \"e2e-same-id-b-e8e1b070-kserve-6978979bd4-llv4l\" (UID: \"7008e915-113e-46fb-92fa-003c595561ca\") " pod="llm/e2e-same-id-b-e8e1b070-kserve-6978979bd4-llv4l"
Sep 19 05:02:37.077114 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:02:37.077028    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7008e915-113e-46fb-92fa-003c595561ca-tmp-dir\") pod \"e2e-same-id-b-e8e1b070-kserve-6978979bd4-llv4l\" (UID: \"7008e915-113e-46fb-92fa-003c595561ca\") " pod="llm/e2e-same-id-b-e8e1b070-kserve-6978979bd4-llv4l"
Sep 19 05:02:37.077114 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:02:37.077061    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7008e915-113e-46fb-92fa-003c595561ca-model-cache\") pod \"e2e-same-id-b-e8e1b070-kserve-6978979bd4-llv4l\" (UID: \"7008e915-113e-46fb-92fa-003c595561ca\") " pod="llm/e2e-same-id-b-e8e1b070-kserve-6978979bd4-llv4l"
Sep 19 05:02:37.077114 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:02:37.077087    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7008e915-113e-46fb-92fa-003c595561ca-home\") pod \"e2e-same-id-b-e8e1b070-kserve-6978979bd4-llv4l\" (UID: \"7008e915-113e-46fb-92fa-003c595561ca\") " pod="llm/e2e-same-id-b-e8e1b070-kserve-6978979bd4-llv4l"
Sep 19 05:02:37.079120 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:02:37.079093    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7008e915-113e-46fb-92fa-003c595561ca-dshm\") pod \"e2e-same-id-b-e8e1b070-kserve-6978979bd4-llv4l\" (UID: \"7008e915-113e-46fb-92fa-003c595561ca\") " pod="llm/e2e-same-id-b-e8e1b070-kserve-6978979bd4-llv4l"
Sep 19 05:02:37.079685 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:02:37.079658    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7008e915-113e-46fb-92fa-003c595561ca-tls-certs\") pod \"e2e-same-id-b-e8e1b070-kserve-6978979bd4-llv4l\" (UID: \"7008e915-113e-46fb-92fa-003c595561ca\") " pod="llm/e2e-same-id-b-e8e1b070-kserve-6978979bd4-llv4l"
Sep 19 05:02:37.086774 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:02:37.086749    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7nbs\" (UniqueName: \"kubernetes.io/projected/7008e915-113e-46fb-92fa-003c595561ca-kube-api-access-z7nbs\") pod \"e2e-same-id-b-e8e1b070-kserve-6978979bd4-llv4l\" (UID: \"7008e915-113e-46fb-92fa-003c595561ca\") " pod="llm/e2e-same-id-b-e8e1b070-kserve-6978979bd4-llv4l"
Sep 19 05:02:37.110658 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:02:37.110626    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-e8e1b070-kserve-6978979bd4-llv4l"
Sep 19 05:02:37.305052 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:02:37.305006    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-e8e1b070-kserve-796494657b-66564"
Sep 19 05:02:37.318268 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:02:37.318242    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-e8e1b070-kserve-796494657b-66564"
Sep 19 05:02:37.391475 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:02:37.391449    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-e8e1b070-kserve-6978979bd4-llv4l"]
Sep 19 05:02:37.393138 ip-10-0-129-215 kubenswrapper[2579]: W0919 05:02:37.393090    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7008e915_113e_46fb_92fa_003c595561ca.slice/crio-cd3c4b58afc4eff09bf3d4d3c4e63f554fe7cbd7dbdf10a5d5ae57340b2fbac5 WatchSource:0}: Error finding container cd3c4b58afc4eff09bf3d4d3c4e63f554fe7cbd7dbdf10a5d5ae57340b2fbac5: Status 404 returned error can't find the container with id cd3c4b58afc4eff09bf3d4d3c4e63f554fe7cbd7dbdf10a5d5ae57340b2fbac5
Sep 19 05:02:37.395533 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:02:37.395509    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-e8e1b070-kserve-796494657b-66564"
Sep 19 05:02:38.390552 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:02:38.390515    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-e8e1b070-kserve-6978979bd4-llv4l" event={"ID":"7008e915-113e-46fb-92fa-003c595561ca","Type":"ContainerStarted","Data":"aae2a30999d4e21f1f3acff1e41e71f85722e4153a915c192f5295889b6252a8"}
Sep 19 05:02:38.390996 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:02:38.390571    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-e8e1b070-kserve-6978979bd4-llv4l" event={"ID":"7008e915-113e-46fb-92fa-003c595561ca","Type":"ContainerStarted","Data":"cd3c4b58afc4eff09bf3d4d3c4e63f554fe7cbd7dbdf10a5d5ae57340b2fbac5"}
Sep 19 05:02:38.415683 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:02:38.415634    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-e8e1b070-kserve-6978979bd4-llv4l" podStartSLOduration=2.415616004 podStartE2EDuration="2.415616004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 05:02:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 05:02:38.411557769 +0000 UTC m=+2937.594518936" watchObservedRunningTime="2026-09-19 05:02:38.415616004 +0000 UTC m=+2937.598577154"
Sep 19 05:02:39.111331 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:02:39.111277    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-e8e1b070-kserve-6978979bd4-llv4l"
Sep 19 05:02:42.131406 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:02:42.131374    2579 scope.go:117] "RemoveContainer" containerID="2fb0860b54cbfaabac7d83ae13e52ed875e038c8063184fc8cf7b018175ffcfb"
Sep 19 05:02:47.111380 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:02:47.111338    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-e8e1b070-kserve-6978979bd4-llv4l"
Sep 19 05:02:47.124360 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:02:47.124331    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-e8e1b070-kserve-6978979bd4-llv4l"
Sep 19 05:02:47.459266 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:02:47.459237    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-e8e1b070-kserve-6978979bd4-llv4l"
Sep 19 05:03:02.089726 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:03:02.089674    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-e8e1b070-kserve-796494657b-66564"]
Sep 19 05:03:07.320767 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:03:07.320731    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-e8e1b070-kserve-6978979bd4-llv4l"]
Sep 19 05:03:17.109264 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:03:17.109198    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-e8e1b070-kserve-796494657b-66564" podUID="88c35b46-f78c-4739-9434-4d07d4f07eb2" containerName="main" containerID="cri-o://10cc895527e1bc4d6370f0908d8db2a01514243b49bf15983576b148d644e5f7" gracePeriod=45
Sep 19 05:03:17.363926 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:03:17.363858    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-e8e1b070-kserve-796494657b-66564"
Sep 19 05:03:17.468354 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:03:17.468322    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88c35b46-f78c-4739-9434-4d07d4f07eb2-dshm\") pod \"88c35b46-f78c-4739-9434-4d07d4f07eb2\" (UID: \"88c35b46-f78c-4739-9434-4d07d4f07eb2\") "
Sep 19 05:03:17.468543 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:03:17.468367    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88c35b46-f78c-4739-9434-4d07d4f07eb2-home\") pod \"88c35b46-f78c-4739-9434-4d07d4f07eb2\" (UID: \"88c35b46-f78c-4739-9434-4d07d4f07eb2\") "
Sep 19 05:03:17.468543 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:03:17.468393    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ws9f6\" (UniqueName: \"kubernetes.io/projected/88c35b46-f78c-4739-9434-4d07d4f07eb2-kube-api-access-ws9f6\") pod \"88c35b46-f78c-4739-9434-4d07d4f07eb2\" (UID: \"88c35b46-f78c-4739-9434-4d07d4f07eb2\") "
Sep 19 05:03:17.468543 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:03:17.468429    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88c35b46-f78c-4739-9434-4d07d4f07eb2-tmp-dir\") pod \"88c35b46-f78c-4739-9434-4d07d4f07eb2\" (UID: \"88c35b46-f78c-4739-9434-4d07d4f07eb2\") "
Sep 19 05:03:17.468543 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:03:17.468474    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88c35b46-f78c-4739-9434-4d07d4f07eb2-tls-certs\") pod \"88c35b46-f78c-4739-9434-4d07d4f07eb2\" (UID: \"88c35b46-f78c-4739-9434-4d07d4f07eb2\") "
Sep 19 05:03:17.468543 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:03:17.468541    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88c35b46-f78c-4739-9434-4d07d4f07eb2-model-cache\") pod \"88c35b46-f78c-4739-9434-4d07d4f07eb2\" (UID: \"88c35b46-f78c-4739-9434-4d07d4f07eb2\") "
Sep 19 05:03:17.468861 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:03:17.468657    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88c35b46-f78c-4739-9434-4d07d4f07eb2-home" (OuterVolumeSpecName: "home") pod "88c35b46-f78c-4739-9434-4d07d4f07eb2" (UID: "88c35b46-f78c-4739-9434-4d07d4f07eb2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 05:03:17.468861 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:03:17.468757    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88c35b46-f78c-4739-9434-4d07d4f07eb2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "88c35b46-f78c-4739-9434-4d07d4f07eb2" (UID: "88c35b46-f78c-4739-9434-4d07d4f07eb2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 05:03:17.468957 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:03:17.468903    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88c35b46-f78c-4739-9434-4d07d4f07eb2-model-cache" (OuterVolumeSpecName: "model-cache") pod "88c35b46-f78c-4739-9434-4d07d4f07eb2" (UID: "88c35b46-f78c-4739-9434-4d07d4f07eb2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 05:03:17.468957 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:03:17.468904    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88c35b46-f78c-4739-9434-4d07d4f07eb2-home\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 05:03:17.468957 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:03:17.468948    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88c35b46-f78c-4739-9434-4d07d4f07eb2-tmp-dir\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 05:03:17.470848 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:03:17.470824    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88c35b46-f78c-4739-9434-4d07d4f07eb2-dshm" (OuterVolumeSpecName: "dshm") pod "88c35b46-f78c-4739-9434-4d07d4f07eb2" (UID: "88c35b46-f78c-4739-9434-4d07d4f07eb2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 05:03:17.470848 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:03:17.470827    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88c35b46-f78c-4739-9434-4d07d4f07eb2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "88c35b46-f78c-4739-9434-4d07d4f07eb2" (UID: "88c35b46-f78c-4739-9434-4d07d4f07eb2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 05:03:17.470848 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:03:17.470851    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88c35b46-f78c-4739-9434-4d07d4f07eb2-kube-api-access-ws9f6" (OuterVolumeSpecName: "kube-api-access-ws9f6") pod "88c35b46-f78c-4739-9434-4d07d4f07eb2" (UID: "88c35b46-f78c-4739-9434-4d07d4f07eb2"). InnerVolumeSpecName "kube-api-access-ws9f6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 05:03:17.570236 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:03:17.570209    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88c35b46-f78c-4739-9434-4d07d4f07eb2-dshm\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 05:03:17.570236 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:03:17.570234    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ws9f6\" (UniqueName: \"kubernetes.io/projected/88c35b46-f78c-4739-9434-4d07d4f07eb2-kube-api-access-ws9f6\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 05:03:17.570236 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:03:17.570244    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88c35b46-f78c-4739-9434-4d07d4f07eb2-tls-certs\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 05:03:17.570468 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:03:17.570255    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88c35b46-f78c-4739-9434-4d07d4f07eb2-model-cache\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 05:03:17.618707 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:03:17.618605    2579 generic.go:356] "Generic (PLEG): container finished" podID="88c35b46-f78c-4739-9434-4d07d4f07eb2" containerID="10cc895527e1bc4d6370f0908d8db2a01514243b49bf15983576b148d644e5f7" exitCode=0
Sep 19 05:03:17.618707 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:03:17.618654    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-e8e1b070-kserve-796494657b-66564"
Sep 19 05:03:17.618707 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:03:17.618678    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-e8e1b070-kserve-796494657b-66564" event={"ID":"88c35b46-f78c-4739-9434-4d07d4f07eb2","Type":"ContainerDied","Data":"10cc895527e1bc4d6370f0908d8db2a01514243b49bf15983576b148d644e5f7"}
Sep 19 05:03:17.618939 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:03:17.618741    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-e8e1b070-kserve-796494657b-66564" event={"ID":"88c35b46-f78c-4739-9434-4d07d4f07eb2","Type":"ContainerDied","Data":"f8b6902e1d01ee80fd31319eb66fceb6efe190dbae209812cea6e6fceadbe65a"}
Sep 19 05:03:17.618939 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:03:17.618762    2579 scope.go:117] "RemoveContainer" containerID="10cc895527e1bc4d6370f0908d8db2a01514243b49bf15983576b148d644e5f7"
Sep 19 05:03:17.629456 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:03:17.629429    2579 scope.go:117] "RemoveContainer" containerID="10cc895527e1bc4d6370f0908d8db2a01514243b49bf15983576b148d644e5f7"
Sep 19 05:03:17.629787 ip-10-0-129-215 kubenswrapper[2579]: E0919 05:03:17.629764    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10cc895527e1bc4d6370f0908d8db2a01514243b49bf15983576b148d644e5f7\": container with ID starting with 10cc895527e1bc4d6370f0908d8db2a01514243b49bf15983576b148d644e5f7 not found: ID does not exist" containerID="10cc895527e1bc4d6370f0908d8db2a01514243b49bf15983576b148d644e5f7"
Sep 19 05:03:17.629880 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:03:17.629795    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10cc895527e1bc4d6370f0908d8db2a01514243b49bf15983576b148d644e5f7"} err="failed to get container status \"10cc895527e1bc4d6370f0908d8db2a01514243b49bf15983576b148d644e5f7\": rpc error: code = NotFound desc = could not find container \"10cc895527e1bc4d6370f0908d8db2a01514243b49bf15983576b148d644e5f7\": container with ID starting with 10cc895527e1bc4d6370f0908d8db2a01514243b49bf15983576b148d644e5f7 not found: ID does not exist"
Sep 19 05:03:17.668975 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:03:17.668941    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-e8e1b070-kserve-796494657b-66564"]
Sep 19 05:03:17.675373 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:03:17.675346    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-e8e1b070-kserve-796494657b-66564"]
Sep 19 05:03:19.351730 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:03:19.351676    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88c35b46-f78c-4739-9434-4d07d4f07eb2" path="/var/lib/kubelet/pods/88c35b46-f78c-4739-9434-4d07d4f07eb2/volumes"
Sep 19 05:03:22.341174 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:03:22.341105    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-e8e1b070-kserve-6978979bd4-llv4l" podUID="7008e915-113e-46fb-92fa-003c595561ca" containerName="main" containerID="cri-o://aae2a30999d4e21f1f3acff1e41e71f85722e4153a915c192f5295889b6252a8" gracePeriod=45
Sep 19 05:03:22.447202 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:03:22.447154    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-b-e8e1b070-kserve-6978979bd4-llv4l" podUID="7008e915-113e-46fb-92fa-003c595561ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.116:8000/ready\": dial tcp 10.134.0.116:8000: connect: connection refused"
Sep 19 05:03:22.609712 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:03:22.609625    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-e8e1b070-kserve-6978979bd4-llv4l"
Sep 19 05:03:22.649256 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:03:22.649218    2579 generic.go:356] "Generic (PLEG): container finished" podID="7008e915-113e-46fb-92fa-003c595561ca" containerID="aae2a30999d4e21f1f3acff1e41e71f85722e4153a915c192f5295889b6252a8" exitCode=0
Sep 19 05:03:22.649423 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:03:22.649312    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-e8e1b070-kserve-6978979bd4-llv4l"
Sep 19 05:03:22.649423 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:03:22.649352    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-e8e1b070-kserve-6978979bd4-llv4l" event={"ID":"7008e915-113e-46fb-92fa-003c595561ca","Type":"ContainerDied","Data":"aae2a30999d4e21f1f3acff1e41e71f85722e4153a915c192f5295889b6252a8"}
Sep 19 05:03:22.649423 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:03:22.649394    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-e8e1b070-kserve-6978979bd4-llv4l" event={"ID":"7008e915-113e-46fb-92fa-003c595561ca","Type":"ContainerDied","Data":"cd3c4b58afc4eff09bf3d4d3c4e63f554fe7cbd7dbdf10a5d5ae57340b2fbac5"}
Sep 19 05:03:22.649423 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:03:22.649411    2579 scope.go:117] "RemoveContainer" containerID="aae2a30999d4e21f1f3acff1e41e71f85722e4153a915c192f5295889b6252a8"
Sep 19 05:03:22.660181 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:03:22.660165    2579 scope.go:117] "RemoveContainer" containerID="aae2a30999d4e21f1f3acff1e41e71f85722e4153a915c192f5295889b6252a8"
Sep 19 05:03:22.660467 ip-10-0-129-215 kubenswrapper[2579]: E0919 05:03:22.660447    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aae2a30999d4e21f1f3acff1e41e71f85722e4153a915c192f5295889b6252a8\": container with ID starting with aae2a30999d4e21f1f3acff1e41e71f85722e4153a915c192f5295889b6252a8 not found: ID does not exist" containerID="aae2a30999d4e21f1f3acff1e41e71f85722e4153a915c192f5295889b6252a8"
Sep 19 05:03:22.660537 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:03:22.660477    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aae2a30999d4e21f1f3acff1e41e71f85722e4153a915c192f5295889b6252a8"} err="failed to get container status \"aae2a30999d4e21f1f3acff1e41e71f85722e4153a915c192f5295889b6252a8\": rpc error: code = NotFound desc = could not find container \"aae2a30999d4e21f1f3acff1e41e71f85722e4153a915c192f5295889b6252a8\": container with ID starting with aae2a30999d4e21f1f3acff1e41e71f85722e4153a915c192f5295889b6252a8 not found: ID does not exist"
Sep 19 05:03:22.715822 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:03:22.715791    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7008e915-113e-46fb-92fa-003c595561ca-model-cache\") pod \"7008e915-113e-46fb-92fa-003c595561ca\" (UID: \"7008e915-113e-46fb-92fa-003c595561ca\") "
Sep 19 05:03:22.716000 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:03:22.715851    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z7nbs\" (UniqueName: \"kubernetes.io/projected/7008e915-113e-46fb-92fa-003c595561ca-kube-api-access-z7nbs\") pod \"7008e915-113e-46fb-92fa-003c595561ca\" (UID: \"7008e915-113e-46fb-92fa-003c595561ca\") "
Sep 19 05:03:22.716000 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:03:22.715888    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7008e915-113e-46fb-92fa-003c595561ca-tls-certs\") pod \"7008e915-113e-46fb-92fa-003c595561ca\" (UID: \"7008e915-113e-46fb-92fa-003c595561ca\") "
Sep 19 05:03:22.716000 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:03:22.715955    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7008e915-113e-46fb-92fa-003c595561ca-dshm\") pod \"7008e915-113e-46fb-92fa-003c595561ca\" (UID: \"7008e915-113e-46fb-92fa-003c595561ca\") "
Sep 19 05:03:22.716000 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:03:22.715978    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7008e915-113e-46fb-92fa-003c595561ca-tmp-dir\") pod \"7008e915-113e-46fb-92fa-003c595561ca\" (UID: \"7008e915-113e-46fb-92fa-003c595561ca\") "
Sep 19 05:03:22.716215 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:03:22.716046    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7008e915-113e-46fb-92fa-003c595561ca-home\") pod \"7008e915-113e-46fb-92fa-003c595561ca\" (UID: \"7008e915-113e-46fb-92fa-003c595561ca\") "
Sep 19 05:03:22.716215 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:03:22.716067    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7008e915-113e-46fb-92fa-003c595561ca-model-cache" (OuterVolumeSpecName: "model-cache") pod "7008e915-113e-46fb-92fa-003c595561ca" (UID: "7008e915-113e-46fb-92fa-003c595561ca"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 05:03:22.716324 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:03:22.716252    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7008e915-113e-46fb-92fa-003c595561ca-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7008e915-113e-46fb-92fa-003c595561ca" (UID: "7008e915-113e-46fb-92fa-003c595561ca"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 05:03:22.716324 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:03:22.716298    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7008e915-113e-46fb-92fa-003c595561ca-home" (OuterVolumeSpecName: "home") pod "7008e915-113e-46fb-92fa-003c595561ca" (UID: "7008e915-113e-46fb-92fa-003c595561ca"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 05:03:22.716431 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:03:22.716364    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7008e915-113e-46fb-92fa-003c595561ca-tmp-dir\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 05:03:22.716431 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:03:22.716386    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7008e915-113e-46fb-92fa-003c595561ca-model-cache\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 05:03:22.718282 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:03:22.718253    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7008e915-113e-46fb-92fa-003c595561ca-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7008e915-113e-46fb-92fa-003c595561ca" (UID: "7008e915-113e-46fb-92fa-003c595561ca"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 05:03:22.718382 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:03:22.718286    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7008e915-113e-46fb-92fa-003c595561ca-kube-api-access-z7nbs" (OuterVolumeSpecName: "kube-api-access-z7nbs") pod "7008e915-113e-46fb-92fa-003c595561ca" (UID: "7008e915-113e-46fb-92fa-003c595561ca"). InnerVolumeSpecName "kube-api-access-z7nbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 05:03:22.718382 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:03:22.718296    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7008e915-113e-46fb-92fa-003c595561ca-dshm" (OuterVolumeSpecName: "dshm") pod "7008e915-113e-46fb-92fa-003c595561ca" (UID: "7008e915-113e-46fb-92fa-003c595561ca"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 05:03:22.817177 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:03:22.817150    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7008e915-113e-46fb-92fa-003c595561ca-tls-certs\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 05:03:22.817177 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:03:22.817174    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7008e915-113e-46fb-92fa-003c595561ca-dshm\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 05:03:22.817177 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:03:22.817182    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7008e915-113e-46fb-92fa-003c595561ca-home\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 05:03:22.817423 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:03:22.817192    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7nbs\" (UniqueName: \"kubernetes.io/projected/7008e915-113e-46fb-92fa-003c595561ca-kube-api-access-z7nbs\") on node \"ip-10-0-129-215.ec2.internal\" DevicePath \"\""
Sep 19 05:03:22.978449 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:03:22.978420    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-e8e1b070-kserve-6978979bd4-llv4l"]
Sep 19 05:03:22.986362 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:03:22.986334    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-e8e1b070-kserve-6978979bd4-llv4l"]
Sep 19 05:03:23.352269 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:03:23.352193    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7008e915-113e-46fb-92fa-003c595561ca" path="/var/lib/kubelet/pods/7008e915-113e-46fb-92fa-003c595561ca/volumes"
Sep 19 05:03:42.157475 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:03:42.157436    2579 scope.go:117] "RemoveContainer" containerID="b491fa3b9bbdfe53d240d6b13e18edb0584eb09bb252bf5a573283d8732af4bb"
Sep 19 05:03:42.168355 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:03:42.168333    2579 scope.go:117] "RemoveContainer" containerID="49c3e5f406e54ab303992f395897dbf581edee0954222e2389ab5e95b10c3dba"
Sep 19 05:04:17.708063 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:17.708012    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-bd667f646-lx5br_0b3506fd-4ba6-4525-a63f-c0aff0f06b90/authorino/0.log"
Sep 19 05:04:23.261084 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:23.261051    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-54765d59fd-4ddzl_06647565-1498-4fe1-a34d-f31450f84ee4/manager/0.log"
Sep 19 05:04:23.382828 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:23.382793    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-78f7d99c47-p42kb_7a1ac630-2440-475f-aa6f-c6a416a80c72/manager/0.log"
Sep 19 05:04:23.517985 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:23.517897    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7bf95c4bb8-c4zv7_a6775f09-fcfa-44c6-b8a8-2087e6eab0f9/manager/0.log"
Sep 19 05:04:23.650777 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:23.650741    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-77df98bf98-rzvj8_6e9a8fd5-94c9-4432-8542-07b855ea91d6/server/0.log"
Sep 19 05:04:24.336471 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:24.336439    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29829885-xlqqv_0bd23183-a191-4026-9927-76cfe375ca16/cleanup/0.log"
Sep 19 05:04:24.447646 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:24.447613    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29829900-f29rm_a7e7effc-d1c2-43e5-8bbc-f7699be8ab12/cleanup/0.log"
Sep 19 05:04:25.366563 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:25.366533    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-bd667f646-lx5br_0b3506fd-4ba6-4525-a63f-c0aff0f06b90/authorino/0.log"
Sep 19 05:04:25.493675 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:25.493643    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6c7cb5c9cb-g98cq_79e96918-0b4e-4940-bb71-ccdfb8b118e2/manager/0.log"
Sep 19 05:04:25.625918 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:25.625832    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b449b7679-jxlq7_557f44c8-ad27-4fda-aa1f-2c04cda00c33/manager/0.log"
Sep 19 05:04:25.874346 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:25.874307    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-54b6d5c989-zs5ck_f670455e-7c9e-4e4f-a8bb-7c28a26a5c3f/manager/0.log"
Sep 19 05:04:26.118837 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:26.118795    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6d9b9f85cb-627xv_a64e50da-98d3-4659-ab01-486ba441519a/manager/0.log"
Sep 19 05:04:26.474710 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:26.474659    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cgg7jw_382f9c83-8f9d-4e1d-a6d8-a2de02a9fc3c/istio-proxy/0.log"
Sep 19 05:04:26.958032 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:26.957999    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-jkrrv_2130d608-c9dd-4f97-8348-bb1a10b7fec5/istio-proxy/1.log"
Sep 19 05:04:27.079193 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:27.079155    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-6987f5b5b9-jb9kg_0651ba9f-b7df-4e1e-88ba-d8cae0519cd7/istio-proxy/0.log"
Sep 19 05:04:27.201877 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:27.201845    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-8wv77_98f8b146-35a4-45fa-9fa7-29ce6d46902e/payload-pre-processing/0.log"
Sep 19 05:04:27.327788 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:27.327681    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-sjhch_fa5e4819-0ed6-44af-a94a-c3529a77ef4b/payload-processing/0.log"
Sep 19 05:04:27.438370 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:27.438334    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6cddf8756d-5pc66_c1ebe82f-7422-4a2c-9f72-7eef8335ffb2/router/0.log"
Sep 19 05:04:27.898161 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:27.898117    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-5d57fc946d-9dls8_fd418b3a-d757-43bc-a02e-7d82c7356079/main/0.log"
Sep 19 05:04:28.355045 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:28.355018    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-89945f574-l5xsz_dd909eb5-27da-46c3-a4b8-1ed1265a2a0b/main/0.log"
Sep 19 05:04:34.054089 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:34.054015    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zp5t8/must-gather-nkbgz"]
Sep 19 05:04:34.054449 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:34.054374    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7008e915-113e-46fb-92fa-003c595561ca" containerName="main"
Sep 19 05:04:34.054449 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:34.054384    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7008e915-113e-46fb-92fa-003c595561ca" containerName="main"
Sep 19 05:04:34.054449 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:34.054403    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88c35b46-f78c-4739-9434-4d07d4f07eb2" containerName="main"
Sep 19 05:04:34.054449 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:34.054409    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="88c35b46-f78c-4739-9434-4d07d4f07eb2" containerName="main"
Sep 19 05:04:34.054590 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:34.054477    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="88c35b46-f78c-4739-9434-4d07d4f07eb2" containerName="main"
Sep 19 05:04:34.054590 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:34.054486    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7008e915-113e-46fb-92fa-003c595561ca" containerName="main"
Sep 19 05:04:34.057513 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:34.057493    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zp5t8/must-gather-nkbgz"
Sep 19 05:04:34.066849 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:34.066828    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zp5t8\"/\"openshift-service-ca.crt\""
Sep 19 05:04:34.066987 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:34.066827    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-zp5t8\"/\"default-dockercfg-b5t6l\""
Sep 19 05:04:34.067408 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:34.067389    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zp5t8\"/\"kube-root-ca.crt\""
Sep 19 05:04:34.093759 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:34.093714    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zp5t8/must-gather-nkbgz"]
Sep 19 05:04:34.195637 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:34.195597    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/738c1e26-214c-461a-9ca6-c71cec3a956a-must-gather-output\") pod \"must-gather-nkbgz\" (UID: \"738c1e26-214c-461a-9ca6-c71cec3a956a\") " pod="openshift-must-gather-zp5t8/must-gather-nkbgz"
Sep 19 05:04:34.195637 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:34.195637    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcrwd\" (UniqueName: \"kubernetes.io/projected/738c1e26-214c-461a-9ca6-c71cec3a956a-kube-api-access-dcrwd\") pod \"must-gather-nkbgz\" (UID: \"738c1e26-214c-461a-9ca6-c71cec3a956a\") " pod="openshift-must-gather-zp5t8/must-gather-nkbgz"
Sep 19 05:04:34.297095 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:34.297054    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/738c1e26-214c-461a-9ca6-c71cec3a956a-must-gather-output\") pod \"must-gather-nkbgz\" (UID: \"738c1e26-214c-461a-9ca6-c71cec3a956a\") " pod="openshift-must-gather-zp5t8/must-gather-nkbgz"
Sep 19 05:04:34.297095 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:34.297096    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcrwd\" (UniqueName: \"kubernetes.io/projected/738c1e26-214c-461a-9ca6-c71cec3a956a-kube-api-access-dcrwd\") pod \"must-gather-nkbgz\" (UID: \"738c1e26-214c-461a-9ca6-c71cec3a956a\") " pod="openshift-must-gather-zp5t8/must-gather-nkbgz"
Sep 19 05:04:34.297435 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:34.297412    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/738c1e26-214c-461a-9ca6-c71cec3a956a-must-gather-output\") pod \"must-gather-nkbgz\" (UID: \"738c1e26-214c-461a-9ca6-c71cec3a956a\") " pod="openshift-must-gather-zp5t8/must-gather-nkbgz"
Sep 19 05:04:34.306051 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:34.305986    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcrwd\" (UniqueName: \"kubernetes.io/projected/738c1e26-214c-461a-9ca6-c71cec3a956a-kube-api-access-dcrwd\") pod \"must-gather-nkbgz\" (UID: \"738c1e26-214c-461a-9ca6-c71cec3a956a\") " pod="openshift-must-gather-zp5t8/must-gather-nkbgz"
Sep 19 05:04:34.367214 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:34.367176    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zp5t8/must-gather-nkbgz"
Sep 19 05:04:34.639871 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:34.639832    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zp5t8/must-gather-nkbgz"]
Sep 19 05:04:34.642051 ip-10-0-129-215 kubenswrapper[2579]: W0919 05:04:34.642026    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod738c1e26_214c_461a_9ca6_c71cec3a956a.slice/crio-a28565d163031956c0600a4bd1326ccdddbf99062998e2990a08ff812a6ec3f8 WatchSource:0}: Error finding container a28565d163031956c0600a4bd1326ccdddbf99062998e2990a08ff812a6ec3f8: Status 404 returned error can't find the container with id a28565d163031956c0600a4bd1326ccdddbf99062998e2990a08ff812a6ec3f8
Sep 19 05:04:34.643867 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:34.643848    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 19 05:04:35.047742 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:35.047682    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zp5t8/must-gather-nkbgz" event={"ID":"738c1e26-214c-461a-9ca6-c71cec3a956a","Type":"ContainerStarted","Data":"a28565d163031956c0600a4bd1326ccdddbf99062998e2990a08ff812a6ec3f8"}
Sep 19 05:04:36.058263 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:36.058211    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zp5t8/must-gather-nkbgz" event={"ID":"738c1e26-214c-461a-9ca6-c71cec3a956a","Type":"ContainerStarted","Data":"4408b09739b82faab42f3e2242de1c5b605196110bf734114ce95323116ff0fd"}
Sep 19 05:04:36.058774 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:36.058281    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zp5t8/must-gather-nkbgz" event={"ID":"738c1e26-214c-461a-9ca6-c71cec3a956a","Type":"ContainerStarted","Data":"89cc49badcfc2537ec4a56f2f79ebf600dd561fd95144e6334d6b3399e570548"}
Sep 19 05:04:36.077006 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:36.076957    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-zp5t8/must-gather-nkbgz" podStartSLOduration=1.327001624 podStartE2EDuration="2.076939851s" totalImagesPullingTime="749.938227ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 05:04:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 05:04:36.075829787 +0000 UTC m=+3055.258790934" watchObservedRunningTime="2026-09-19 05:04:36.076939851 +0000 UTC m=+3055.259901000"
Sep 19 05:04:37.202534 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:37.202493    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-5t728_64cb49fc-ede9-4369-aa8b-76a95d1edab6/global-pull-secret-syncer/0.log"
Sep 19 05:04:37.500654 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:37.500563    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-szpwc_f4b6c76b-7816-4e8b-9904-2f0cb5faca57/konnectivity-agent/0.log"
Sep 19 05:04:37.525668 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:37.525626    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-215.ec2.internal_5cccb5a09b0064784f816095b5ca12fb/haproxy/0.log"
Sep 19 05:04:42.573553 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:42.573320    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-bd667f646-lx5br_0b3506fd-4ba6-4525-a63f-c0aff0f06b90/authorino/0.log"
Sep 19 05:04:42.610237 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:42.610207    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6c7cb5c9cb-g98cq_79e96918-0b4e-4940-bb71-ccdfb8b118e2/manager/0.log"
Sep 19 05:04:42.645717 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:42.645229    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b449b7679-jxlq7_557f44c8-ad27-4fda-aa1f-2c04cda00c33/manager/0.log"
Sep 19 05:04:42.751208 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:42.751165    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-54b6d5c989-zs5ck_f670455e-7c9e-4e4f-a8bb-7c28a26a5c3f/manager/0.log"
Sep 19 05:04:42.889551 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:42.889464    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6d9b9f85cb-627xv_a64e50da-98d3-4659-ab01-486ba441519a/manager/0.log"
Sep 19 05:04:44.948730 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:44.948678    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-h52kg_71273c3e-c393-4009-945f-f7f42c668d75/node-exporter/0.log"
Sep 19 05:04:44.975260 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:44.975230    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-h52kg_71273c3e-c393-4009-945f-f7f42c668d75/kube-rbac-proxy/0.log"
Sep 19 05:04:45.000455 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:45.000429    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-h52kg_71273c3e-c393-4009-945f-f7f42c668d75/init-textfile/0.log"
Sep 19 05:04:47.286879 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:47.286842    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zp5t8/perf-node-gather-daemonset-7w9kp"]
Sep 19 05:04:47.292916 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:47.292888    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zp5t8/perf-node-gather-daemonset-7w9kp"
Sep 19 05:04:47.322910 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:47.322770    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zp5t8/perf-node-gather-daemonset-7w9kp"]
Sep 19 05:04:47.335068 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:47.335034    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d2489d0c-65b5-4285-a43b-cf4abf90f5cc-lib-modules\") pod \"perf-node-gather-daemonset-7w9kp\" (UID: \"d2489d0c-65b5-4285-a43b-cf4abf90f5cc\") " pod="openshift-must-gather-zp5t8/perf-node-gather-daemonset-7w9kp"
Sep 19 05:04:47.335436 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:47.335411    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d2489d0c-65b5-4285-a43b-cf4abf90f5cc-podres\") pod \"perf-node-gather-daemonset-7w9kp\" (UID: \"d2489d0c-65b5-4285-a43b-cf4abf90f5cc\") " pod="openshift-must-gather-zp5t8/perf-node-gather-daemonset-7w9kp"
Sep 19 05:04:47.335673 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:47.335652    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kc6gw\" (UniqueName: \"kubernetes.io/projected/d2489d0c-65b5-4285-a43b-cf4abf90f5cc-kube-api-access-kc6gw\") pod \"perf-node-gather-daemonset-7w9kp\" (UID: \"d2489d0c-65b5-4285-a43b-cf4abf90f5cc\") " pod="openshift-must-gather-zp5t8/perf-node-gather-daemonset-7w9kp"
Sep 19 05:04:47.335843 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:47.335826    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d2489d0c-65b5-4285-a43b-cf4abf90f5cc-proc\") pod \"perf-node-gather-daemonset-7w9kp\" (UID: \"d2489d0c-65b5-4285-a43b-cf4abf90f5cc\") " pod="openshift-must-gather-zp5t8/perf-node-gather-daemonset-7w9kp"
Sep 19 05:04:47.336048 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:47.336030    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d2489d0c-65b5-4285-a43b-cf4abf90f5cc-sys\") pod \"perf-node-gather-daemonset-7w9kp\" (UID: \"d2489d0c-65b5-4285-a43b-cf4abf90f5cc\") " pod="openshift-must-gather-zp5t8/perf-node-gather-daemonset-7w9kp"
Sep 19 05:04:47.436776 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:47.436739    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kc6gw\" (UniqueName: \"kubernetes.io/projected/d2489d0c-65b5-4285-a43b-cf4abf90f5cc-kube-api-access-kc6gw\") pod \"perf-node-gather-daemonset-7w9kp\" (UID: \"d2489d0c-65b5-4285-a43b-cf4abf90f5cc\") " pod="openshift-must-gather-zp5t8/perf-node-gather-daemonset-7w9kp"
Sep 19 05:04:47.437065 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:47.437042    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d2489d0c-65b5-4285-a43b-cf4abf90f5cc-proc\") pod \"perf-node-gather-daemonset-7w9kp\" (UID: \"d2489d0c-65b5-4285-a43b-cf4abf90f5cc\") " pod="openshift-must-gather-zp5t8/perf-node-gather-daemonset-7w9kp"
Sep 19 05:04:47.437259 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:47.437241    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d2489d0c-65b5-4285-a43b-cf4abf90f5cc-sys\") pod \"perf-node-gather-daemonset-7w9kp\" (UID: \"d2489d0c-65b5-4285-a43b-cf4abf90f5cc\") " pod="openshift-must-gather-zp5t8/perf-node-gather-daemonset-7w9kp"
Sep 19 05:04:47.437449 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:47.437245    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d2489d0c-65b5-4285-a43b-cf4abf90f5cc-proc\") pod \"perf-node-gather-daemonset-7w9kp\" (UID: \"d2489d0c-65b5-4285-a43b-cf4abf90f5cc\") " pod="openshift-must-gather-zp5t8/perf-node-gather-daemonset-7w9kp"
Sep 19 05:04:47.437579 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:47.437292    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d2489d0c-65b5-4285-a43b-cf4abf90f5cc-sys\") pod \"perf-node-gather-daemonset-7w9kp\" (UID: \"d2489d0c-65b5-4285-a43b-cf4abf90f5cc\") " pod="openshift-must-gather-zp5t8/perf-node-gather-daemonset-7w9kp"
Sep 19 05:04:47.437755 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:47.437736    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d2489d0c-65b5-4285-a43b-cf4abf90f5cc-lib-modules\") pod \"perf-node-gather-daemonset-7w9kp\" (UID: \"d2489d0c-65b5-4285-a43b-cf4abf90f5cc\") " pod="openshift-must-gather-zp5t8/perf-node-gather-daemonset-7w9kp"
Sep 19 05:04:47.437893 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:47.437828    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d2489d0c-65b5-4285-a43b-cf4abf90f5cc-lib-modules\") pod \"perf-node-gather-daemonset-7w9kp\" (UID: \"d2489d0c-65b5-4285-a43b-cf4abf90f5cc\") " pod="openshift-must-gather-zp5t8/perf-node-gather-daemonset-7w9kp"
Sep 19 05:04:47.438050 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:47.438031    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d2489d0c-65b5-4285-a43b-cf4abf90f5cc-podres\") pod \"perf-node-gather-daemonset-7w9kp\" (UID: \"d2489d0c-65b5-4285-a43b-cf4abf90f5cc\") " pod="openshift-must-gather-zp5t8/perf-node-gather-daemonset-7w9kp"
Sep 19 05:04:47.438203 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:47.438185    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d2489d0c-65b5-4285-a43b-cf4abf90f5cc-podres\") pod \"perf-node-gather-daemonset-7w9kp\" (UID: \"d2489d0c-65b5-4285-a43b-cf4abf90f5cc\") " pod="openshift-must-gather-zp5t8/perf-node-gather-daemonset-7w9kp"
Sep 19 05:04:47.447902 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:47.447875    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kc6gw\" (UniqueName: \"kubernetes.io/projected/d2489d0c-65b5-4285-a43b-cf4abf90f5cc-kube-api-access-kc6gw\") pod \"perf-node-gather-daemonset-7w9kp\" (UID: \"d2489d0c-65b5-4285-a43b-cf4abf90f5cc\") " pod="openshift-must-gather-zp5t8/perf-node-gather-daemonset-7w9kp"
Sep 19 05:04:47.618898 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:47.618815    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zp5t8/perf-node-gather-daemonset-7w9kp"
Sep 19 05:04:47.925030 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:47.924994    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zp5t8/perf-node-gather-daemonset-7w9kp"]
Sep 19 05:04:47.927866 ip-10-0-129-215 kubenswrapper[2579]: W0919 05:04:47.927830    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd2489d0c_65b5_4285_a43b_cf4abf90f5cc.slice/crio-86b21b544b94817094f8dfe42edc649b2e6aac139ec90b121ca6c6ee81f3d5c8 WatchSource:0}: Error finding container 86b21b544b94817094f8dfe42edc649b2e6aac139ec90b121ca6c6ee81f3d5c8: Status 404 returned error can't find the container with id 86b21b544b94817094f8dfe42edc649b2e6aac139ec90b121ca6c6ee81f3d5c8
Sep 19 05:04:48.150311 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:48.150271    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zp5t8/perf-node-gather-daemonset-7w9kp" event={"ID":"d2489d0c-65b5-4285-a43b-cf4abf90f5cc","Type":"ContainerStarted","Data":"41a3e4c6d3614e58bfd8837f6b9d50d0cd16ba2b80ca90a61d5b069086d750f4"}
Sep 19 05:04:48.150507 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:48.150338    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zp5t8/perf-node-gather-daemonset-7w9kp" event={"ID":"d2489d0c-65b5-4285-a43b-cf4abf90f5cc","Type":"ContainerStarted","Data":"86b21b544b94817094f8dfe42edc649b2e6aac139ec90b121ca6c6ee81f3d5c8"}
Sep 19 05:04:48.150588 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:48.150556    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-zp5t8/perf-node-gather-daemonset-7w9kp"
Sep 19 05:04:48.186292 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:48.186157    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-zp5t8/perf-node-gather-daemonset-7w9kp" podStartSLOduration=1.186134659 podStartE2EDuration="1.186134659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 05:04:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 05:04:48.183284404 +0000 UTC m=+3067.366245555" watchObservedRunningTime="2026-09-19 05:04:48.186134659 +0000 UTC m=+3067.369095809"
Sep 19 05:04:48.380564 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:48.380525    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5cff7757cb-j49nf_b26f88b6-5bc5-46dd-992b-3c214d4a4d04/console/0.log"
Sep 19 05:04:48.440773 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:48.440664    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6594f4c75-rhcwm_c0cd206c-8a93-45d6-80d6-edfc51d23f45/download-server/0.log"
Sep 19 05:04:50.222137 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:50.222107    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-n8jb9_7a6cf6e3-7960-4902-9c40-375a4de502ac/dns/0.log"
Sep 19 05:04:50.244569 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:50.244528    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-n8jb9_7a6cf6e3-7960-4902-9c40-375a4de502ac/kube-rbac-proxy/0.log"
Sep 19 05:04:50.276310 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:50.276282    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2ptq2_a5194aa7-dba0-46dc-b3e9-91a4ea69e79e/dns-node-resolver/0.log"
Sep 19 05:04:50.956148 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:50.956081    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5bzkh_a8d1a2a7-a76e-4741-be3a-d87124254f5d/node-ca/0.log"
Sep 19 05:04:52.042271 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:52.042230    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cgg7jw_382f9c83-8f9d-4e1d-a6d8-a2de02a9fc3c/istio-proxy/0.log"
Sep 19 05:04:52.577832 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:52.577794    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-jkrrv_2130d608-c9dd-4f97-8348-bb1a10b7fec5/istio-proxy/0.log"
Sep 19 05:04:52.582658 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:52.582625    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-jkrrv_2130d608-c9dd-4f97-8348-bb1a10b7fec5/istio-proxy/1.log"
Sep 19 05:04:52.607138 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:52.607101    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-6987f5b5b9-jb9kg_0651ba9f-b7df-4e1e-88ba-d8cae0519cd7/istio-proxy/0.log"
Sep 19 05:04:52.655657 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:52.655622    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-8wv77_98f8b146-35a4-45fa-9fa7-29ce6d46902e/payload-pre-processing/0.log"
Sep 19 05:04:52.706239 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:52.706193    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-sjhch_fa5e4819-0ed6-44af-a94a-c3529a77ef4b/payload-processing/0.log"
Sep 19 05:04:52.736560 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:52.736531    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6cddf8756d-5pc66_c1ebe82f-7422-4a2c-9f72-7eef8335ffb2/router/0.log"
Sep 19 05:04:53.717087 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:53.717060    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8759c_e9d28ff6-a630-4517-acd6-2c596abcef2a/serve-healthcheck-canary/0.log"
Sep 19 05:04:54.172785 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:54.171744    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-zp5t8/perf-node-gather-daemonset-7w9kp"
Sep 19 05:04:54.250619 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:54.250588    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-prdpb_1633a682-5c86-4a59-b25a-2d049b6f3577/insights-operator/0.log"
Sep 19 05:04:54.254970 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:54.254934    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-prdpb_1633a682-5c86-4a59-b25a-2d049b6f3577/insights-operator/1.log"
Sep 19 05:04:54.425517 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:54.425435    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-t7mz5_7ef03827-5da3-44f3-84fd-7a9581528469/kube-rbac-proxy/0.log"
Sep 19 05:04:54.448905 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:54.448875    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-t7mz5_7ef03827-5da3-44f3-84fd-7a9581528469/exporter/0.log"
Sep 19 05:04:54.473642 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:54.473584    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-t7mz5_7ef03827-5da3-44f3-84fd-7a9581528469/extractor/0.log"
Sep 19 05:04:56.936336 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:56.936303    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cgg7jw_382f9c83-8f9d-4e1d-a6d8-a2de02a9fc3c/istio-proxy/0.log"
Sep 19 05:04:57.102928 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:57.102896    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-fc5f7b9db-6qm7t_7f60da12-b971-4cd4-a8e3-de671d1f1d37/manager/0.log"
Sep 19 05:04:57.130541 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:57.130506    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-fc5f7b9db-6qm7t_7f60da12-b971-4cd4-a8e3-de671d1f1d37/copy-manifests/0.log"
Sep 19 05:04:57.441065 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:57.441037    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-jkrrv_2130d608-c9dd-4f97-8348-bb1a10b7fec5/istio-proxy/0.log"
Sep 19 05:04:57.444411 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:57.444384    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-jkrrv_2130d608-c9dd-4f97-8348-bb1a10b7fec5/istio-proxy/1.log"
Sep 19 05:04:57.464284 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:57.464259    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-6987f5b5b9-jb9kg_0651ba9f-b7df-4e1e-88ba-d8cae0519cd7/istio-proxy/0.log"
Sep 19 05:04:57.500311 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:57.500271    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-8wv77_98f8b146-35a4-45fa-9fa7-29ce6d46902e/payload-pre-processing/0.log"
Sep 19 05:04:57.544386 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:57.544342    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-sjhch_fa5e4819-0ed6-44af-a94a-c3529a77ef4b/payload-processing/0.log"
Sep 19 05:04:57.562443 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:57.562417    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6cddf8756d-5pc66_c1ebe82f-7422-4a2c-9f72-7eef8335ffb2/router/0.log"
Sep 19 05:04:59.148099 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:59.148005    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cgg7jw_382f9c83-8f9d-4e1d-a6d8-a2de02a9fc3c/istio-proxy/0.log"
Sep 19 05:04:59.402760 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:59.402707    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-54765d59fd-4ddzl_06647565-1498-4fe1-a34d-f31450f84ee4/manager/0.log"
Sep 19 05:04:59.671942 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:59.671853    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-jkrrv_2130d608-c9dd-4f97-8348-bb1a10b7fec5/istio-proxy/0.log"
Sep 19 05:04:59.680195 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:59.680128    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-jkrrv_2130d608-c9dd-4f97-8348-bb1a10b7fec5/istio-proxy/1.log"
Sep 19 05:04:59.709562 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:59.709511    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-6987f5b5b9-jb9kg_0651ba9f-b7df-4e1e-88ba-d8cae0519cd7/istio-proxy/0.log"
Sep 19 05:04:59.773660 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:59.773612    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-8wv77_98f8b146-35a4-45fa-9fa7-29ce6d46902e/payload-pre-processing/0.log"
Sep 19 05:04:59.792803 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:59.792767    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-78f7d99c47-p42kb_7a1ac630-2440-475f-aa6f-c6a416a80c72/manager/0.log"
Sep 19 05:04:59.817340 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:59.817311    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-sjhch_fa5e4819-0ed6-44af-a94a-c3529a77ef4b/payload-processing/0.log"
Sep 19 05:04:59.837590 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:59.837559    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6cddf8756d-5pc66_c1ebe82f-7422-4a2c-9f72-7eef8335ffb2/router/0.log"
Sep 19 05:04:59.875647 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:59.875608    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7bf95c4bb8-c4zv7_a6775f09-fcfa-44c6-b8a8-2087e6eab0f9/manager/0.log"
Sep 19 05:04:59.901483 ip-10-0-129-215 kubenswrapper[2579]: I0919 05:04:59.901455    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-77df98bf98-rzvj8_6e9a8fd5-94c9-4432-8542-07b855ea91d6/server/0.log"