Sep 11 23:01:14.568890 ip-10-0-128-161 systemd[1]: Starting Kubernetes Kubelet...
Sep 11 23:01:14.996764 ip-10-0-128-161 kubenswrapper[2546]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 23:01:14.996764 ip-10-0-128-161 kubenswrapper[2546]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 11 23:01:14.996764 ip-10-0-128-161 kubenswrapper[2546]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 23:01:14.996764 ip-10-0-128-161 kubenswrapper[2546]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 11 23:01:14.997691 ip-10-0-128-161 kubenswrapper[2546]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 23:01:14.997817 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:14.997736    2546 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 11 23:01:15.003226 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003211    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 23:01:15.003226 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003225    2546 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 23:01:15.003291 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003230    2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 23:01:15.003291 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003233    2546 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 23:01:15.003291 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003236    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 23:01:15.003291 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003239    2546 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 23:01:15.003291 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003242    2546 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 23:01:15.003291 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003245    2546 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 23:01:15.003291 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003248    2546 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 23:01:15.003291 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003250    2546 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 23:01:15.003291 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003253    2546 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 23:01:15.003291 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003255    2546 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 23:01:15.003291 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003258    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 23:01:15.003291 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003261    2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 23:01:15.003291 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003264    2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 23:01:15.003291 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003266    2546 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 23:01:15.003291 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003269    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 23:01:15.003291 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003272    2546 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 23:01:15.003291 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003275    2546 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 23:01:15.003291 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003277    2546 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 23:01:15.003291 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003280    2546 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 23:01:15.003291 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003283    2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 23:01:15.003774 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003285    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 23:01:15.003774 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003287    2546 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 23:01:15.003774 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003290    2546 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 23:01:15.003774 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003294    2546 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 23:01:15.003774 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003297    2546 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 23:01:15.003774 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003300    2546 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 23:01:15.003774 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003303    2546 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 23:01:15.003774 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003305    2546 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 23:01:15.003774 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003308    2546 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 23:01:15.003774 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003311    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 23:01:15.003774 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003313    2546 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 23:01:15.003774 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003316    2546 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 23:01:15.003774 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003319    2546 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 23:01:15.003774 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003322    2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 23:01:15.003774 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003324    2546 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 23:01:15.003774 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003327    2546 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 23:01:15.003774 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003329    2546 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 23:01:15.003774 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003332    2546 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 23:01:15.003774 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003334    2546 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 23:01:15.003774 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003336    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 23:01:15.004254 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003340    2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 23:01:15.004254 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003342    2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 23:01:15.004254 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003345    2546 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 23:01:15.004254 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003347    2546 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 23:01:15.004254 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003350    2546 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 23:01:15.004254 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003352    2546 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 23:01:15.004254 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003355    2546 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 23:01:15.004254 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003357    2546 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 23:01:15.004254 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003359    2546 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 23:01:15.004254 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003362    2546 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 23:01:15.004254 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003366    2546 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 23:01:15.004254 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003370    2546 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 23:01:15.004254 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003372    2546 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 23:01:15.004254 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003376    2546 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 23:01:15.004254 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003378    2546 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 23:01:15.004254 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003381    2546 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 23:01:15.004254 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003384    2546 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 23:01:15.004254 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003386    2546 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 23:01:15.004254 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003389    2546 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 23:01:15.004747 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003393    2546 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 23:01:15.004747 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003395    2546 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 23:01:15.004747 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003397    2546 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 23:01:15.004747 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003400    2546 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 23:01:15.004747 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003403    2546 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 23:01:15.004747 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003405    2546 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 23:01:15.004747 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003408    2546 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 23:01:15.004747 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003410    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 23:01:15.004747 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003412    2546 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 23:01:15.004747 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003415    2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 23:01:15.004747 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003419    2546 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 23:01:15.004747 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003422    2546 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 23:01:15.004747 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003425    2546 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 23:01:15.004747 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003428    2546 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 23:01:15.004747 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003431    2546 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 23:01:15.004747 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003433    2546 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 23:01:15.004747 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003436    2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 23:01:15.004747 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003438    2546 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 23:01:15.004747 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003441    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 23:01:15.005202 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003443    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 23:01:15.005202 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003446    2546 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 23:01:15.005202 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003448    2546 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 23:01:15.005202 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003450    2546 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 23:01:15.005202 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003453    2546 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 23:01:15.005202 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003455    2546 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 23:01:15.005202 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003457    2546 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 23:01:15.005202 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003462    2546 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 23:01:15.005202 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003466    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 23:01:15.005202 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003469    2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 23:01:15.005202 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003471    2546 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 23:01:15.005202 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003474    2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 23:01:15.005202 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003476    2546 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 23:01:15.005202 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003479    2546 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 23:01:15.005202 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003481    2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 23:01:15.005202 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003484    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 23:01:15.005202 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003486    2546 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 23:01:15.005202 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003489    2546 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 23:01:15.005202 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.003491    2546 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 23:01:15.005710 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004190    2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 23:01:15.005710 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004196    2546 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 23:01:15.005710 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004200    2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 23:01:15.005710 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004203    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 23:01:15.005710 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004205    2546 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 23:01:15.005710 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004207    2546 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 23:01:15.005710 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004210    2546 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 23:01:15.005710 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004213    2546 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 23:01:15.005710 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004215    2546 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 23:01:15.005710 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004217    2546 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 23:01:15.005710 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004220    2546 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 23:01:15.005710 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004222    2546 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 23:01:15.005710 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004225    2546 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 23:01:15.005710 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004229    2546 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 23:01:15.005710 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004232    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 23:01:15.005710 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004235    2546 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 23:01:15.005710 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004238    2546 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 23:01:15.005710 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004240    2546 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 23:01:15.005710 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004243    2546 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 23:01:15.005710 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004245    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 23:01:15.006203 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004249    2546 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 23:01:15.006203 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004251    2546 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 23:01:15.006203 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004254    2546 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 23:01:15.006203 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004257    2546 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 23:01:15.006203 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004259    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 23:01:15.006203 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004262    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 23:01:15.006203 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004264    2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 23:01:15.006203 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004267    2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 23:01:15.006203 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004269    2546 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 23:01:15.006203 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004272    2546 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 23:01:15.006203 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004274    2546 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 23:01:15.006203 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004290    2546 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 23:01:15.006203 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004295    2546 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 23:01:15.006203 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004297    2546 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 23:01:15.006203 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004300    2546 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 23:01:15.006203 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004302    2546 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 23:01:15.006203 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004305    2546 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 23:01:15.006203 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004308    2546 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 23:01:15.006203 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004311    2546 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 23:01:15.006680 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004313    2546 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 23:01:15.006680 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004315    2546 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 23:01:15.006680 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004318    2546 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 23:01:15.006680 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004321    2546 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 23:01:15.006680 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004323    2546 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 23:01:15.006680 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004325    2546 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 23:01:15.006680 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004328    2546 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 23:01:15.006680 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004332    2546 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 23:01:15.006680 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004335    2546 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 23:01:15.006680 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004338    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 23:01:15.006680 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004340    2546 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 23:01:15.006680 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004343    2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 23:01:15.006680 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004345    2546 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 23:01:15.006680 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004349    2546 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 23:01:15.006680 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004352    2546 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 23:01:15.006680 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004355    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 23:01:15.006680 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004357    2546 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 23:01:15.006680 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004361    2546 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 23:01:15.006680 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004364    2546 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 23:01:15.007136 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004367    2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 23:01:15.007136 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004371    2546 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 23:01:15.007136 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004373    2546 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 23:01:15.007136 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004376    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 23:01:15.007136 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004378    2546 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 23:01:15.007136 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004381    2546 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 23:01:15.007136 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004384    2546 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 23:01:15.007136 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004387    2546 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 23:01:15.007136 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004389    2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 23:01:15.007136 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004392    2546 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 23:01:15.007136 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004395    2546 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 23:01:15.007136 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004398    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 23:01:15.007136 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004400    2546 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 23:01:15.007136 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004403    2546 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 23:01:15.007136 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004405    2546 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 23:01:15.007136 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004408    2546 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 23:01:15.007136 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004410    2546 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 23:01:15.007136 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004413    2546 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 23:01:15.007136 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004415    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 23:01:15.007136 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004418    2546 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 23:01:15.007721 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004421    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 23:01:15.007721 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004424    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 23:01:15.007721 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004426    2546 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 23:01:15.007721 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004429    2546 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 23:01:15.007721 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004431    2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 23:01:15.007721 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004433    2546 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 23:01:15.007721 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004437    2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 23:01:15.007721 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004440    2546 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 23:01:15.007721 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004443    2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 23:01:15.007721 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004445    2546 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 23:01:15.007721 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004447    2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 23:01:15.007721 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004450    2546 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 23:01:15.007721 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004453    2546 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 23:01:15.007721 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004455    2546 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 23:01:15.007721 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004458    2546 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 23:01:15.007721 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004460    2546 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 23:01:15.007721 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004462    2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 23:01:15.007721 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004465    2546 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 23:01:15.007721 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004467    2546 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 23:01:15.007721 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004471    2546 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 23:01:15.008221 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004473    2546 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 23:01:15.008221 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004854    2546 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 23:01:15.008221 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004860    2546 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 23:01:15.008221 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004862    2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 23:01:15.008221 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004865    2546 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 23:01:15.008221 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004867    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 23:01:15.008221 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004870    2546 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 23:01:15.008221 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004873    2546 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 23:01:15.008221 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004875    2546 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 23:01:15.008221 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004878    2546 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 23:01:15.008221 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004881    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 23:01:15.008221 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004883    2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 23:01:15.008221 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004886    2546 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 23:01:15.008221 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004889    2546 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 23:01:15.008221 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004891    2546 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 23:01:15.008221 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004894    2546 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 23:01:15.008221 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004896    2546 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 23:01:15.008221 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004899    2546 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 23:01:15.008221 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004902    2546 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 23:01:15.008221 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004905    2546 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 23:01:15.008742 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004908    2546 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 23:01:15.008742 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004910    2546 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 23:01:15.008742 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004912    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 23:01:15.008742 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004915    2546 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 23:01:15.008742 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004918    2546 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 23:01:15.008742 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004920    2546 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 23:01:15.008742 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004923    2546 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 23:01:15.008742 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004925    2546 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 23:01:15.008742 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004928    2546 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 23:01:15.008742 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004930    2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 23:01:15.008742 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004932    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 23:01:15.008742 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004936    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 23:01:15.008742 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004940    2546 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 23:01:15.008742 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004943    2546 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 23:01:15.008742 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004946    2546 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 23:01:15.008742 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004948    2546 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 23:01:15.008742 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004951    2546 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 23:01:15.008742 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004955    2546 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 23:01:15.008742 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004958    2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 23:01:15.009206 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004961    2546 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 23:01:15.009206 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004965    2546 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 23:01:15.009206 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004968    2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 23:01:15.009206 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004970    2546 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 23:01:15.009206 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004973    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 23:01:15.009206 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004976    2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 23:01:15.009206 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004978    2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 23:01:15.009206 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004980    2546 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 23:01:15.009206 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004983    2546 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 23:01:15.009206 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004985    2546 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 23:01:15.009206 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004987    2546 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 23:01:15.009206 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004990    2546 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 23:01:15.009206 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004992    2546 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 23:01:15.009206 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004995    2546 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 23:01:15.009206 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.004997    2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 23:01:15.009206 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.005000    2546 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 23:01:15.009206 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.005002    2546 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 23:01:15.009206 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.005005    2546 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 23:01:15.009206 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.005007    2546 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 23:01:15.009681 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.005010    2546 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 23:01:15.009681 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.005012    2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 23:01:15.009681 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.005014    2546 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 23:01:15.009681 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.005017    2546 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 23:01:15.009681 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.005019    2546 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 23:01:15.009681 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.005022    2546 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 23:01:15.009681 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.005025    2546 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 23:01:15.009681 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.005027    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 23:01:15.009681 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.005030    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 23:01:15.009681 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.005032    2546 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 23:01:15.009681 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.005035    2546 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 23:01:15.009681 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.005037    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 23:01:15.009681 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.005040    2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 23:01:15.009681 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.005042    2546 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 23:01:15.009681 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.005044    2546 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 23:01:15.009681 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.005047    2546 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 23:01:15.009681 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.005049    2546 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 23:01:15.009681 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.005052    2546 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 23:01:15.009681 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.005054    2546 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 23:01:15.009681 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.005057    2546 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 23:01:15.010185 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.005059    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 23:01:15.010185 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.005061    2546 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 23:01:15.010185 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.005064    2546 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 23:01:15.010185 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.005066    2546 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 23:01:15.010185 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.005073    2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 23:01:15.010185 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.005076    2546 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 23:01:15.010185 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.005078    2546 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 23:01:15.010185 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.005081    2546 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 23:01:15.010185 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.005084    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 23:01:15.010185 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.005087    2546 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 23:01:15.010185 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.005089    2546 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 23:01:15.010185 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.005092    2546 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 23:01:15.010185 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.005094    2546 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 23:01:15.010185 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.005097    2546 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 23:01:15.010185 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.005100    2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 23:01:15.010185 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.005102    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 23:01:15.010185 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.005105    2546 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 23:01:15.010185 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.005108    2546 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 23:01:15.010185 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.005111    2546 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 23:01:15.010679 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.005114    2546 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 23:01:15.010679 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.005116    2546 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 23:01:15.010679 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.005119    2546 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 23:01:15.010679 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005200    2546 flags.go:64] FLAG: --address="0.0.0.0"
Sep 11 23:01:15.010679 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005251    2546 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 11 23:01:15.010679 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005257    2546 flags.go:64] FLAG: --anonymous-auth="true"
Sep 11 23:01:15.010679 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005262    2546 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 11 23:01:15.010679 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005266    2546 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 11 23:01:15.010679 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005270    2546 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 11 23:01:15.010679 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005275    2546 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 11 23:01:15.010679 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005282    2546 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 11 23:01:15.010679 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005285    2546 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 11 23:01:15.010679 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005288    2546 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 11 23:01:15.010679 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005292    2546 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 11 23:01:15.010679 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005295    2546 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 11 23:01:15.010679 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005298    2546 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 11 23:01:15.010679 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005301    2546 flags.go:64] FLAG: --cgroup-root=""
Sep 11 23:01:15.010679 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005305    2546 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 11 23:01:15.010679 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005308    2546 flags.go:64] FLAG: --client-ca-file=""
Sep 11 23:01:15.010679 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005311    2546 flags.go:64] FLAG: --cloud-provider="external"
Sep 11 23:01:15.010679 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005313    2546 flags.go:64] FLAG: --cluster-dns="[]"
Sep 11 23:01:15.010679 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005318    2546 flags.go:64] FLAG: --cluster-domain=""
Sep 11 23:01:15.010679 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005321    2546 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 11 23:01:15.011246 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005324    2546 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 11 23:01:15.011246 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005327    2546 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 11 23:01:15.011246 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005332    2546 flags.go:64] FLAG: --container-log-max-files="5"
Sep 11 23:01:15.011246 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005335    2546 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 11 23:01:15.011246 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005338    2546 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 11 23:01:15.011246 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005341    2546 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 11 23:01:15.011246 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005344    2546 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 11 23:01:15.011246 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005347    2546 flags.go:64] FLAG: --contention-profiling="false"
Sep 11 23:01:15.011246 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005350    2546 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 11 23:01:15.011246 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005353    2546 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 11 23:01:15.011246 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005358    2546 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 11 23:01:15.011246 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005361    2546 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 11 23:01:15.011246 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005365    2546 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 11 23:01:15.011246 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005368    2546 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 11 23:01:15.011246 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005371    2546 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 11 23:01:15.011246 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005373    2546 flags.go:64] FLAG: --enable-load-reader="false"
Sep 11 23:01:15.011246 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005376    2546 flags.go:64] FLAG: --enable-server="true"
Sep 11 23:01:15.011246 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005379    2546 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 11 23:01:15.011246 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005384    2546 flags.go:64] FLAG: --event-burst="100"
Sep 11 23:01:15.011246 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005387    2546 flags.go:64] FLAG: --event-qps="50"
Sep 11 23:01:15.011246 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005390    2546 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 11 23:01:15.011246 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005393    2546 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 11 23:01:15.011246 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005396    2546 flags.go:64] FLAG: --eviction-hard=""
Sep 11 23:01:15.011246 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005400    2546 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 11 23:01:15.011837 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005403    2546 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 11 23:01:15.011837 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005405    2546 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 11 23:01:15.011837 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005408    2546 flags.go:64] FLAG: --eviction-soft=""
Sep 11 23:01:15.011837 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005411    2546 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 11 23:01:15.011837 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005414    2546 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 11 23:01:15.011837 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005417    2546 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 11 23:01:15.011837 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005420    2546 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 11 23:01:15.011837 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005423    2546 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 11 23:01:15.011837 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005426    2546 flags.go:64] FLAG: --fail-swap-on="true"
Sep 11 23:01:15.011837 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005428    2546 flags.go:64] FLAG: --feature-gates=""
Sep 11 23:01:15.011837 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005432    2546 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 11 23:01:15.011837 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005435    2546 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 11 23:01:15.011837 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005438    2546 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 11 23:01:15.011837 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005441    2546 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 11 23:01:15.011837 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005444    2546 flags.go:64] FLAG: --healthz-port="10248"
Sep 11 23:01:15.011837 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005447    2546 flags.go:64] FLAG: --help="false"
Sep 11 23:01:15.011837 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005450    2546 flags.go:64] FLAG: --hostname-override="ip-10-0-128-161.ec2.internal"
Sep 11 23:01:15.011837 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005454    2546 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 11 23:01:15.011837 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005457    2546 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 11 23:01:15.011837 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005460    2546 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 11 23:01:15.011837 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005463    2546 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 11 23:01:15.011837 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005467    2546 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 11 23:01:15.011837 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005470    2546 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 11 23:01:15.011837 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005473    2546 flags.go:64] FLAG: --image-service-endpoint=""
Sep 11 23:01:15.012432 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005475    2546 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 11 23:01:15.012432 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005478    2546 flags.go:64] FLAG: --kube-api-burst="100"
Sep 11 23:01:15.012432 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005481    2546 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 11 23:01:15.012432 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005484    2546 flags.go:64] FLAG: --kube-api-qps="50"
Sep 11 23:01:15.012432 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005487    2546 flags.go:64] FLAG: --kube-reserved=""
Sep 11 23:01:15.012432 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005490    2546 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 11 23:01:15.012432 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005493    2546 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 11 23:01:15.012432 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005496    2546 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 11 23:01:15.012432 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005499    2546 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 11 23:01:15.012432 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005501    2546 flags.go:64] FLAG: --lock-file=""
Sep 11 23:01:15.012432 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005504    2546 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 11 23:01:15.012432 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005507    2546 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 11 23:01:15.012432 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005511    2546 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 11 23:01:15.012432 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005516    2546 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 11 23:01:15.012432 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005519    2546 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 11 23:01:15.012432 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005522    2546 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 11 23:01:15.012432 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005525    2546 flags.go:64] FLAG: --logging-format="text"
Sep 11 23:01:15.012432 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005530    2546 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 11 23:01:15.012432 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005533    2546 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 11 23:01:15.012432 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005536    2546 flags.go:64] FLAG: --manifest-url=""
Sep 11 23:01:15.012432 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005539    2546 flags.go:64] FLAG: --manifest-url-header=""
Sep 11 23:01:15.012432 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005543    2546 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 11 23:01:15.012432 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005546    2546 flags.go:64] FLAG: --max-open-files="1000000"
Sep 11 23:01:15.012432 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005550    2546 flags.go:64] FLAG: --max-pods="110"
Sep 11 23:01:15.012432 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005553    2546 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 11 23:01:15.013059 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005556    2546 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 11 23:01:15.013059 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005560    2546 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 11 23:01:15.013059 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005563    2546 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 11 23:01:15.013059 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005566    2546 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 11 23:01:15.013059 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005569    2546 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 11 23:01:15.013059 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005572    2546 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 11 23:01:15.013059 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005593    2546 flags.go:64] FLAG: --node-status-max-images="50"
Sep 11 23:01:15.013059 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005596    2546 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 11 23:01:15.013059 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005599    2546 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 11 23:01:15.013059 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005603    2546 flags.go:64] FLAG: --pod-cidr=""
Sep 11 23:01:15.013059 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005606    2546 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 11 23:01:15.013059 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005611    2546 flags.go:64] FLAG: --pod-manifest-path=""
Sep 11 23:01:15.013059 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005614    2546 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 11 23:01:15.013059 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005618    2546 flags.go:64] FLAG: --pods-per-core="0"
Sep 11 23:01:15.013059 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005620    2546 flags.go:64] FLAG: --port="10250"
Sep 11 23:01:15.013059 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005624    2546 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 11 23:01:15.013059 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005626    2546 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-06dc07a086d311c29"
Sep 11 23:01:15.013059 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005630    2546 flags.go:64] FLAG: --qos-reserved=""
Sep 11 23:01:15.013059 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005633    2546 flags.go:64] FLAG: --read-only-port="10255"
Sep 11 23:01:15.013059 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005636    2546 flags.go:64] FLAG: --register-node="true"
Sep 11 23:01:15.013059 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005638    2546 flags.go:64] FLAG: --register-with-taints=""
Sep 11 23:01:15.013059 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005642    2546 flags.go:64] FLAG: --registry-burst="10"
Sep 11 23:01:15.013059 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005645    2546 flags.go:64] FLAG: --registry-qps="5"
Sep 11 23:01:15.013059 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005648    2546 flags.go:64] FLAG: --reserved-cpus=""
Sep 11 23:01:15.013659 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005655    2546 flags.go:64] FLAG: --reserved-memory=""
Sep 11 23:01:15.013659 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005659    2546 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 11 23:01:15.013659 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005662    2546 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 11 23:01:15.013659 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005665    2546 flags.go:64] FLAG: --rotate-certificates="false"
Sep 11 23:01:15.013659 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005668    2546 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 11 23:01:15.013659 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005671    2546 flags.go:64] FLAG: --runonce="false"
Sep 11 23:01:15.013659 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005673    2546 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 11 23:01:15.013659 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005676    2546 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 11 23:01:15.013659 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005679    2546 flags.go:64] FLAG: --seccomp-default="false"
Sep 11 23:01:15.013659 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005682    2546 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 11 23:01:15.013659 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005685    2546 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 11 23:01:15.013659 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005688    2546 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 11 23:01:15.013659 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005692    2546 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 11 23:01:15.013659 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005695    2546 flags.go:64] FLAG: --storage-driver-password="root"
Sep 11 23:01:15.013659 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005698    2546 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 11 23:01:15.013659 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005701    2546 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 11 23:01:15.013659 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005704    2546 flags.go:64] FLAG: --storage-driver-user="root"
Sep 11 23:01:15.013659 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005707    2546 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 11 23:01:15.013659 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005710    2546 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 11 23:01:15.013659 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005713    2546 flags.go:64] FLAG: --system-cgroups=""
Sep 11 23:01:15.013659 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005715    2546 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 11 23:01:15.013659 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005721    2546 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 11 23:01:15.013659 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005724    2546 flags.go:64] FLAG: --tls-cert-file=""
Sep 11 23:01:15.013659 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005727    2546 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 11 23:01:15.013659 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005731    2546 flags.go:64] FLAG: --tls-min-version=""
Sep 11 23:01:15.014312 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005734    2546 flags.go:64] FLAG: --tls-private-key-file=""
Sep 11 23:01:15.014312 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005738    2546 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 11 23:01:15.014312 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005741    2546 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 11 23:01:15.014312 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005744    2546 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 11 23:01:15.014312 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005747    2546 flags.go:64] FLAG: --v="2"
Sep 11 23:01:15.014312 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005751    2546 flags.go:64] FLAG: --version="false"
Sep 11 23:01:15.014312 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005755    2546 flags.go:64] FLAG: --vmodule=""
Sep 11 23:01:15.014312 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005760    2546 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 11 23:01:15.014312 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.005763    2546 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 11 23:01:15.014312 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.005876    2546 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 23:01:15.014312 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.005880    2546 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 23:01:15.014312 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.005883    2546 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 23:01:15.014312 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.005886    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 23:01:15.014312 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.005888    2546 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 23:01:15.014312 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.005891    2546 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 23:01:15.014312 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.005894    2546 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 23:01:15.014312 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.005896    2546 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 23:01:15.014312 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.005898    2546 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 23:01:15.014312 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.005901    2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 23:01:15.014312 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.005903    2546 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 23:01:15.014312 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.005906    2546 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 23:01:15.014312 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.005909    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 23:01:15.014978 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.005912    2546 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 23:01:15.014978 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.005915    2546 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 23:01:15.014978 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.005917    2546 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 23:01:15.014978 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.005920    2546 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 23:01:15.014978 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.005922    2546 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 23:01:15.014978 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.005924    2546 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 23:01:15.014978 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.005927    2546 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 23:01:15.014978 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.005929    2546 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 23:01:15.014978 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.005932    2546 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 23:01:15.014978 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.005934    2546 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 23:01:15.014978 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.005937    2546 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 23:01:15.014978 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.005940    2546 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 23:01:15.014978 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.005945    2546 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 23:01:15.014978 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.005949    2546 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 23:01:15.014978 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.005952    2546 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 23:01:15.014978 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.005954    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 23:01:15.014978 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.005957    2546 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 23:01:15.014978 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.005961    2546 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 23:01:15.014978 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.005964    2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 23:01:15.014978 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.005967    2546 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 23:01:15.015779 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.005969    2546 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 23:01:15.015779 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.005972    2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 23:01:15.015779 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.005974    2546 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 23:01:15.015779 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.005977    2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 23:01:15.015779 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.005979    2546 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 23:01:15.015779 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.005982    2546 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 23:01:15.015779 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.005984    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 23:01:15.015779 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.005987    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 23:01:15.015779 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.005989    2546 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 23:01:15.015779 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.005991    2546 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 23:01:15.015779 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.005995    2546 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 23:01:15.015779 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.006002    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 23:01:15.015779 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.006005    2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 23:01:15.015779 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.006008    2546 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 23:01:15.015779 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.006011    2546 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 23:01:15.015779 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.006014    2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 23:01:15.015779 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.006016    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 23:01:15.015779 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.006019    2546 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 23:01:15.015779 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.006021    2546 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 23:01:15.016291 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.006023    2546 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 23:01:15.016291 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.006026    2546 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 23:01:15.016291 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.006028    2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 23:01:15.016291 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.006031    2546 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 23:01:15.016291 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.006034    2546 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 23:01:15.016291 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.006037    2546 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 23:01:15.016291 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.006040    2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 23:01:15.016291 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.006042    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 23:01:15.016291 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.006045    2546 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 23:01:15.016291 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.006047    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 23:01:15.016291 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.006051    2546 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 23:01:15.016291 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.006053    2546 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 23:01:15.016291 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.006056    2546 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 23:01:15.016291 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.006058    2546 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 23:01:15.016291 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.006061    2546 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 23:01:15.016291 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.006064    2546 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 23:01:15.016291 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.006066    2546 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 23:01:15.016291 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.006069    2546 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 23:01:15.016291 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.006071    2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 23:01:15.016291 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.006073    2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 23:01:15.016811 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.006076    2546 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 23:01:15.016811 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.006078    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 23:01:15.016811 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.006081    2546 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 23:01:15.016811 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.006084    2546 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 23:01:15.016811 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.006086    2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 23:01:15.016811 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.006089    2546 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 23:01:15.016811 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.006092    2546 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 23:01:15.016811 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.006094    2546 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 23:01:15.016811 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.006097    2546 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 23:01:15.016811 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.006099    2546 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 23:01:15.016811 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.006102    2546 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 23:01:15.016811 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.006104    2546 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 23:01:15.016811 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.006106    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 23:01:15.016811 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.006109    2546 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 23:01:15.016811 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.006111    2546 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 23:01:15.016811 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.006114    2546 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 23:01:15.016811 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.006126    2546 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 23:01:15.016811 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.006129    2546 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 23:01:15.016811 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.006131    2546 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 23:01:15.016811 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.006134    2546 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 23:01:15.017289 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.006137    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 23:01:15.017289 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.006140    2546 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 23:01:15.017289 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.006144    2546 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 23:01:15.017289 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.006147    2546 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 23:01:15.017289 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.006150    2546 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 23:01:15.017289 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.006152    2546 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 23:01:15.017289 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.006155    2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 23:01:15.017289 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.006849    2546 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 23:01:15.017289 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.016364    2546 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 11 23:01:15.017289 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.016379    2546 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 11 23:01:15.017289 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.016393    2546 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 11 23:01:15.017289 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.016399    2546 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 11 23:01:15.017289 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016451    2546 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 23:01:15.017289 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016456    2546 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 23:01:15.017675 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016459    2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 23:01:15.017675 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016464    2546 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 23:01:15.017675 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016468    2546 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 23:01:15.017675 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016471    2546 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 23:01:15.017675 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016474    2546 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 23:01:15.017675 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016477    2546 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 23:01:15.017675 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016480    2546 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 23:01:15.017675 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016483    2546 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 23:01:15.017675 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016486    2546 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 23:01:15.017675 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016489    2546 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 23:01:15.017675 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016491    2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 23:01:15.017675 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016494    2546 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 23:01:15.017675 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016496    2546 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 23:01:15.017675 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016499    2546 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 23:01:15.017675 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016501    2546 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 23:01:15.017675 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016504    2546 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 23:01:15.017675 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016507    2546 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 23:01:15.017675 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016510    2546 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 23:01:15.017675 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016512    2546 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 23:01:15.018168 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016514    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 23:01:15.018168 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016517    2546 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 23:01:15.018168 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016519    2546 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 23:01:15.018168 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016522    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 23:01:15.018168 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016525    2546 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 23:01:15.018168 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016527    2546 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 23:01:15.018168 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016529    2546 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 23:01:15.018168 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016532    2546 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 23:01:15.018168 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016536    2546 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 23:01:15.018168 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016538    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 23:01:15.018168 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016541    2546 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 23:01:15.018168 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016544    2546 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 23:01:15.018168 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016546    2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 23:01:15.018168 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016549    2546 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 23:01:15.018168 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016551    2546 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 23:01:15.018168 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016554    2546 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 23:01:15.018168 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016556    2546 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 23:01:15.018168 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016559    2546 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 23:01:15.018168 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016561    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 23:01:15.018168 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016564    2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 23:01:15.018673 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016566    2546 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 23:01:15.018673 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016569    2546 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 23:01:15.018673 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016571    2546 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 23:01:15.018673 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016574    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 23:01:15.018673 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016593    2546 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 23:01:15.018673 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016598    2546 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 23:01:15.018673 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016601    2546 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 23:01:15.018673 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016604    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 23:01:15.018673 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016606    2546 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 23:01:15.018673 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016609    2546 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 23:01:15.018673 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016612    2546 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 23:01:15.018673 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016614    2546 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 23:01:15.018673 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016617    2546 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 23:01:15.018673 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016619    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 23:01:15.018673 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016622    2546 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 23:01:15.018673 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016624    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 23:01:15.018673 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016627    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 23:01:15.018673 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016629    2546 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 23:01:15.018673 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016632    2546 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 23:01:15.018673 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016634    2546 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 23:01:15.019149 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016637    2546 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 23:01:15.019149 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016640    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 23:01:15.019149 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016644    2546 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 23:01:15.019149 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016648    2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 23:01:15.019149 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016651    2546 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 23:01:15.019149 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016654    2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 23:01:15.019149 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016657    2546 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 23:01:15.019149 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016660    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 23:01:15.019149 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016663    2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 23:01:15.019149 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016666    2546 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 23:01:15.019149 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016668    2546 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 23:01:15.019149 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016671    2546 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 23:01:15.019149 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016673    2546 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 23:01:15.019149 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016676    2546 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 23:01:15.019149 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016678    2546 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 23:01:15.019149 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016681    2546 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 23:01:15.019149 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016685    2546 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 23:01:15.019149 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016688    2546 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 23:01:15.019149 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016692    2546 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 23:01:15.019750 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016695    2546 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 23:01:15.019750 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016698    2546 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 23:01:15.019750 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016700    2546 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 23:01:15.019750 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016704    2546 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 23:01:15.019750 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016706    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 23:01:15.019750 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016709    2546 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 23:01:15.019750 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016712    2546 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 23:01:15.019750 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016714    2546 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 23:01:15.019750 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016716    2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 23:01:15.019750 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016719    2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 23:01:15.019750 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016722    2546 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 23:01:15.019750 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016724    2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 23:01:15.019750 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016727    2546 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 23:01:15.019750 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016730    2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 23:01:15.019750 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016732    2546 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 23:01:15.019750 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016735    2546 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 23:01:15.019750 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016737    2546 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 23:01:15.019750 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016740    2546 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 23:01:15.019750 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016742    2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 23:01:15.020194 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.016747    2546 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 23:01:15.020194 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016853    2546 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 23:01:15.020194 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016858    2546 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 23:01:15.020194 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016861    2546 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 23:01:15.020194 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016865    2546 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 23:01:15.020194 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016868    2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 23:01:15.020194 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016871    2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 23:01:15.020194 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016875    2546 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 23:01:15.020194 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016878    2546 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 23:01:15.020194 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016881    2546 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 23:01:15.020194 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016884    2546 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 23:01:15.020194 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016887    2546 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 23:01:15.020194 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016890    2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 23:01:15.020194 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016893    2546 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 23:01:15.020548 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016895    2546 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 23:01:15.020548 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016898    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 23:01:15.020548 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016900    2546 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 23:01:15.020548 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016903    2546 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 23:01:15.020548 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016905    2546 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 23:01:15.020548 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016907    2546 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 23:01:15.020548 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016910    2546 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 23:01:15.020548 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016912    2546 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 23:01:15.020548 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016915    2546 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 23:01:15.020548 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016917    2546 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 23:01:15.020548 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016920    2546 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 23:01:15.020548 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016923    2546 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 23:01:15.020548 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016926    2546 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 23:01:15.020548 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016929    2546 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 23:01:15.020548 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016931    2546 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 23:01:15.020548 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016934    2546 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 23:01:15.020548 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016936    2546 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 23:01:15.020548 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016939    2546 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 23:01:15.020548 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016941    2546 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 23:01:15.020548 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016944    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 23:01:16.305640 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016946    2546 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 23:01:16.305640 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016949    2546 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 23:01:16.305640 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016951    2546 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 23:01:16.305640 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016954    2546 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 23:01:16.305640 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016956    2546 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 23:01:16.305640 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016958    2546 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 23:01:16.305640 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016961    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 23:01:16.305640 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016963    2546 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 23:01:16.305640 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016966    2546 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 23:01:16.305640 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016968    2546 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 23:01:16.305640 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016970    2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 23:01:16.305640 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016973    2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 23:01:16.305640 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016976    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 23:01:16.305640 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016978    2546 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 23:01:16.305640 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016981    2546 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 23:01:16.305640 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016983    2546 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 23:01:16.305640 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016985    2546 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 23:01:16.305640 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016988    2546 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 23:01:16.305640 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016991    2546 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 23:01:16.306403 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016994    2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 23:01:16.306403 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016997    2546 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 23:01:16.306403 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.016999    2546 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 23:01:16.306403 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.017002    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 23:01:16.306403 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.017004    2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 23:01:16.306403 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.017007    2546 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 23:01:16.306403 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.017009    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 23:01:16.306403 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.017012    2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 23:01:16.306403 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.017014    2546 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 23:01:16.306403 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.017017    2546 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 23:01:16.306403 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.017020    2546 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 23:01:16.306403 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.017023    2546 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 23:01:16.306403 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.017025    2546 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 23:01:16.306403 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.017028    2546 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 23:01:16.306403 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.017031    2546 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 23:01:16.306403 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.017033    2546 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 23:01:16.306403 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.017035    2546 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 23:01:16.306403 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.017038    2546 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 23:01:16.306403 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.017040    2546 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 23:01:16.306403 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.017043    2546 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 23:01:16.306941 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.017045    2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 23:01:16.306941 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.017047    2546 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 23:01:16.306941 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.017050    2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 23:01:16.306941 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.017052    2546 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 23:01:16.306941 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.017054    2546 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 23:01:16.306941 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.017057    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 23:01:16.306941 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.017060    2546 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 23:01:16.306941 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.017062    2546 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 23:01:16.306941 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.017065    2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 23:01:16.306941 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.017067    2546 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 23:01:16.306941 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.017070    2546 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 23:01:16.306941 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.017072    2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 23:01:16.306941 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.017074    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 23:01:16.306941 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.017077    2546 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 23:01:16.306941 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.017079    2546 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 23:01:16.306941 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.017082    2546 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 23:01:16.306941 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.017084    2546 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 23:01:16.306941 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.017087    2546 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 23:01:16.306941 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.017089    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 23:01:16.306941 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.017092    2546 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 23:01:16.307474 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.017095    2546 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 23:01:16.307474 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.017097    2546 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 23:01:16.307474 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.017100    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 23:01:16.307474 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.017102    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 23:01:16.307474 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.017105    2546 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 23:01:16.307474 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.017107    2546 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 23:01:16.307474 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:15.017110    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 23:01:16.307474 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.017114    2546 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 23:01:16.307474 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:15.017207    2546 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 11 23:01:16.307698 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.307673    2546 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 11 23:01:16.308842 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.308829    2546 server.go:1019] "Starting client certificate rotation"
Sep 11 23:01:16.308956 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.308941    2546 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 23:01:16.308989 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.308971    2546 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 23:01:16.311910 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.311897    2546 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 23:01:16.313111 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.313094    2546 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 23:01:16.327319 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.327300    2546 log.go:25] "Validated CRI v1 runtime API"
Sep 11 23:01:16.330566 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.330549    2546 log.go:25] "Validated CRI v1 image API"
Sep 11 23:01:16.333789 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.333768    2546 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 11 23:01:16.339915 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.339896    2546 fs.go:135] Filesystem UUIDs: map[7925e31f-16a4-4f19-9c16-96f4ce533b5c:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 8d01ce10-5b47-44d6-a8e2-0b4a2c19cec8:/dev/nvme0n1p4]
Sep 11 23:01:16.340001 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.339919    2546 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 11 23:01:16.343533 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.343516    2546 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 23:01:16.345988 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.345880    2546 manager.go:216] Machine: {Timestamp:2026-09-11 23:01:16.344478038 +0000 UTC m=+0.377607245 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3090185 MemoryCapacity:32812158976 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec25193bc77f4817be1765bc8e57aea0 SystemUUID:ec25193b-c77f-4817-be17-65bc8e57aea0 BootID:99213e8e-f71d-4e55-8a14-48f9e0d00908 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:5d:28:68:54:01 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:5d:28:68:54:01 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ce:73:7d:a4:fb:ae Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812158976 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 11 23:01:16.346065 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.346004    2546 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 11 23:01:16.346106 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.346095    2546 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 11 23:01:16.349572 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.349551    2546 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 11 23:01:16.349730 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.349576    2546 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-161.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 11 23:01:16.350463 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.350453    2546 topology_manager.go:138] "Creating topology manager with none policy"
Sep 11 23:01:16.350499 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.350466    2546 container_manager_linux.go:309] "Creating device plugin manager"
Sep 11 23:01:16.350499 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.350474    2546 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 23:01:16.351327 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.351317    2546 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 23:01:16.352559 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.352549    2546 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 11 23:01:16.357316 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.357304    2546 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 23:01:16.357438 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.357429    2546 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 11 23:01:16.357478 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.357470    2546 kubelet.go:486] "Attempting to sync node with API server"
Sep 11 23:01:16.357507 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.357487    2546 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 11 23:01:16.357538 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.357511    2546 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 11 23:01:16.357538 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.357531    2546 kubelet.go:398] "Adding apiserver pod source"
Sep 11 23:01:16.357629 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.357554    2546 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 11 23:01:16.358255 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.358241    2546 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 23:01:16.358322 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.358266    2546 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 23:01:16.362182 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.362143    2546 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 11 23:01:16.364508 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.364493    2546 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 11 23:01:16.364555 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.364525    2546 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 11 23:01:16.366592 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.366579    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 11 23:01:16.366630 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.366599    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 11 23:01:16.366630 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.366610    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 11 23:01:16.366630 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.366618    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 11 23:01:16.366630 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.366627    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 11 23:01:16.366630 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.366632    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 11 23:01:16.366766 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.366645    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 11 23:01:16.366766 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.366650    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 11 23:01:16.366766 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.366657    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 11 23:01:16.366766 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.366662    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 11 23:01:16.366766 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.366670    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 11 23:01:16.366766 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.366676    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 11 23:01:16.367587 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.367576    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 11 23:01:16.367626 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.367589    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 11 23:01:16.370632 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.370614    2546 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-161.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 11 23:01:16.370719 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:16.370688    2546 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-161.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 11 23:01:16.370761 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:16.370716    2546 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 11 23:01:16.371684 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.371671    2546 server.go:1268] "Started kubelet"
Sep 11 23:01:16.371766 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.371742    2546 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 11 23:01:16.371869 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.371761    2546 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 11 23:01:16.371923 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.371893    2546 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 11 23:01:16.372692 ip-10-0-128-161 systemd[1]: Started Kubernetes Kubelet.
Sep 11 23:01:16.373036 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.373021    2546 server.go:310] "Adding debug handlers to kubelet server"
Sep 11 23:01:16.374106 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.374091    2546 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 11 23:01:16.376333 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:16.376313    2546 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 11 23:01:16.377715 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:16.376636    2546 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-161.ec2.internal.18d46685b1be1dd2  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-161.ec2.internal,UID:ip-10-0-128-161.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-161.ec2.internal,},FirstTimestamp:2026-09-11 23:01:16.371639762 +0000 UTC m=+0.404768969,LastTimestamp:2026-09-11 23:01:16.371639762 +0000 UTC m=+0.404768969,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-161.ec2.internal,}"
Sep 11 23:01:16.381658 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.381643    2546 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 11 23:01:16.381780 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:16.381758    2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-161.ec2.internal\" not found"
Sep 11 23:01:16.381908 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.381883    2546 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 11 23:01:16.382524 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.382508    2546 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 11 23:01:16.382599 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.382523    2546 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 11 23:01:16.382599 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.382568    2546 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 11 23:01:16.382731 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.382711    2546 reconstruct.go:97] "Volume reconstruction finished"
Sep 11 23:01:16.382731 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.382726    2546 reconciler.go:29] "Reconciler: start to sync state"
Sep 11 23:01:16.383683 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.383663    2546 factory.go:55] Registering systemd factory
Sep 11 23:01:16.383683 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.383684    2546 factory.go:223] Registration of the systemd container factory successfully
Sep 11 23:01:16.383914 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.383901    2546 factory.go:153] Registering CRI-O factory
Sep 11 23:01:16.383975 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.383921    2546 factory.go:223] Registration of the crio container factory successfully
Sep 11 23:01:16.384034 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.383982    2546 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 11 23:01:16.384034 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.384002    2546 factory.go:103] Registering Raw factory
Sep 11 23:01:16.384034 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.384018    2546 manager.go:1196] Started watching for new ooms in manager
Sep 11 23:01:16.384476 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.384440    2546 manager.go:349] Starting recovery of all containers
Sep 11 23:01:16.386007 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:16.385977    2546 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 11 23:01:16.386007 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:16.385951    2546 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-161.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 11 23:01:16.394529 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.392653    2546 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9llmj"
Sep 11 23:01:16.396592 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.396577    2546 manager.go:354] Recovery completed
Sep 11 23:01:16.400036 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.400013    2546 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9llmj"
Sep 11 23:01:16.400808 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.400790    2546 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 23:01:16.402618 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.402600    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-161.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 23:01:16.402682 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.402632    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-161.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 23:01:16.402682 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.402643    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-161.ec2.internal" event="NodeHasSufficientPID"
Sep 11 23:01:16.403257 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.403239    2546 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 11 23:01:16.403257 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.403257    2546 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 11 23:01:16.403357 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.403276    2546 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 23:01:16.403357 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.403256    2546 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 23:01:16.404001 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.403985    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-161.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 23:01:16.404074 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.404011    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-161.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 23:01:16.404074 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.404021    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-161.ec2.internal" event="NodeHasSufficientPID"
Sep 11 23:01:16.405061 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:16.404984    2546 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-161.ec2.internal.18d46685b396d234  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-161.ec2.internal,UID:ip-10-0-128-161.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-161.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-161.ec2.internal,},FirstTimestamp:2026-09-11 23:01:16.402618932 +0000 UTC m=+0.435748140,LastTimestamp:2026-09-11 23:01:16.402618932 +0000 UTC m=+0.435748140,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-161.ec2.internal,}"
Sep 11 23:01:16.405341 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.405331    2546 policy_none.go:49] "None policy: Start"
Sep 11 23:01:16.405387 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.405347    2546 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 11 23:01:16.405387 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.405357    2546 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 11 23:01:16.407343 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.407332    2546 policy_none.go:47] "Start"
Sep 11 23:01:16.443740 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.443718    2546 manager.go:337] "Starting Device Plugin manager"
Sep 11 23:01:16.443854 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:16.443754    2546 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 11 23:01:16.443854 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.443764    2546 server.go:87] "Starting device plugin registration server"
Sep 11 23:01:16.444209 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.444195    2546 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 11 23:01:16.444244 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.444219    2546 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 11 23:01:16.444523 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.444499    2546 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 11 23:01:16.444619 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.444577    2546 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 11 23:01:16.444619 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.444587    2546 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 11 23:01:16.445367 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:16.445224    2546 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 11 23:01:16.445367 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:16.445278    2546 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-161.ec2.internal\" not found"
Sep 11 23:01:16.485867 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.485756    2546 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 11 23:01:16.487372 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.487348    2546 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 11 23:01:16.487475 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.487412    2546 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 11 23:01:16.487475 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.487441    2546 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 11 23:01:16.487572 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:16.487491    2546 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 11 23:01:16.489534 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.489515    2546 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 23:01:16.544755 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.544723    2546 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 23:01:16.545753 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.545736    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-161.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 23:01:16.545858 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.545772    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-161.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 23:01:16.545858 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.545789    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-161.ec2.internal" event="NodeHasSufficientPID"
Sep 11 23:01:16.545858 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.545819    2546 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-161.ec2.internal"
Sep 11 23:01:16.553023 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.553007    2546 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-161.ec2.internal"
Sep 11 23:01:16.553023 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:16.553029    2546 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-161.ec2.internal\": node \"ip-10-0-128-161.ec2.internal\" not found"
Sep 11 23:01:16.578853 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:16.578831    2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-161.ec2.internal\" not found"
Sep 11 23:01:16.588206 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.588186    2546 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-161.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-161.ec2.internal"]
Sep 11 23:01:16.588271 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.588253    2546 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 23:01:16.589186 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.589170    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-161.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 23:01:16.589246 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.589205    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-161.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 23:01:16.589246 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.589220    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-161.ec2.internal" event="NodeHasSufficientPID"
Sep 11 23:01:16.591470 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.591457    2546 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 23:01:16.591601 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.591584    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-161.ec2.internal"
Sep 11 23:01:16.591701 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.591643    2546 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 23:01:16.592556 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.592539    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-161.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 23:01:16.592658 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.592570    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-161.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 23:01:16.592658 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.592586    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-161.ec2.internal" event="NodeHasSufficientPID"
Sep 11 23:01:16.594616 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.594596    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-161.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 23:01:16.594706 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.594629    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-161.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 23:01:16.594706 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.594642    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-161.ec2.internal" event="NodeHasSufficientPID"
Sep 11 23:01:16.594917 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.594901    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-161.ec2.internal"
Sep 11 23:01:16.594967 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.594928    2546 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 23:01:16.595707 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.595690    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-161.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 23:01:16.595778 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.595722    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-161.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 23:01:16.595778 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.595733    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-161.ec2.internal" event="NodeHasSufficientPID"
Sep 11 23:01:16.624970 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:16.624950    2546 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-161.ec2.internal\" not found" node="ip-10-0-128-161.ec2.internal"
Sep 11 23:01:16.629439 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:16.629423    2546 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-161.ec2.internal\" not found" node="ip-10-0-128-161.ec2.internal"
Sep 11 23:01:16.679724 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:16.679689    2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-161.ec2.internal\" not found"
Sep 11 23:01:16.685034 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.685013    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e80b9e5ab597f3b17c8733c311f97f1f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-161.ec2.internal\" (UID: \"e80b9e5ab597f3b17c8733c311f97f1f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-161.ec2.internal"
Sep 11 23:01:16.685089 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.685047    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e80b9e5ab597f3b17c8733c311f97f1f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-161.ec2.internal\" (UID: \"e80b9e5ab597f3b17c8733c311f97f1f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-161.ec2.internal"
Sep 11 23:01:16.685089 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.685065    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fbf860b5b0097ab5f709024d70f9c3d5-config\") pod \"kube-apiserver-proxy-ip-10-0-128-161.ec2.internal\" (UID: \"fbf860b5b0097ab5f709024d70f9c3d5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-161.ec2.internal"
Sep 11 23:01:16.780271 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:16.780208    2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-161.ec2.internal\" not found"
Sep 11 23:01:16.785569 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.785550    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e80b9e5ab597f3b17c8733c311f97f1f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-161.ec2.internal\" (UID: \"e80b9e5ab597f3b17c8733c311f97f1f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-161.ec2.internal"
Sep 11 23:01:16.785619 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.785581    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e80b9e5ab597f3b17c8733c311f97f1f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-161.ec2.internal\" (UID: \"e80b9e5ab597f3b17c8733c311f97f1f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-161.ec2.internal"
Sep 11 23:01:16.785619 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.785604    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fbf860b5b0097ab5f709024d70f9c3d5-config\") pod \"kube-apiserver-proxy-ip-10-0-128-161.ec2.internal\" (UID: \"fbf860b5b0097ab5f709024d70f9c3d5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-161.ec2.internal"
Sep 11 23:01:16.785679 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.785639    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e80b9e5ab597f3b17c8733c311f97f1f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-161.ec2.internal\" (UID: \"e80b9e5ab597f3b17c8733c311f97f1f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-161.ec2.internal"
Sep 11 23:01:16.785679 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.785673    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fbf860b5b0097ab5f709024d70f9c3d5-config\") pod \"kube-apiserver-proxy-ip-10-0-128-161.ec2.internal\" (UID: \"fbf860b5b0097ab5f709024d70f9c3d5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-161.ec2.internal"
Sep 11 23:01:16.785737 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.785676    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e80b9e5ab597f3b17c8733c311f97f1f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-161.ec2.internal\" (UID: \"e80b9e5ab597f3b17c8733c311f97f1f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-161.ec2.internal"
Sep 11 23:01:16.881005 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:16.880960    2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-161.ec2.internal\" not found"
Sep 11 23:01:16.927428 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.927388    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-161.ec2.internal"
Sep 11 23:01:16.932397 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:16.932258    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-161.ec2.internal"
Sep 11 23:01:16.981920 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:16.981886    2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-161.ec2.internal\" not found"
Sep 11 23:01:17.082391 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:17.082314    2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-161.ec2.internal\" not found"
Sep 11 23:01:17.182794 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:17.182756    2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-161.ec2.internal\" not found"
Sep 11 23:01:17.225922 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.225891    2546 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 23:01:17.282799 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.282764    2546 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-161.ec2.internal"
Sep 11 23:01:17.295023 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.295003    2546 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 23:01:17.296807 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.296791    2546 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-161.ec2.internal"
Sep 11 23:01:17.305193 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.305175    2546 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 23:01:17.308291 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.308268    2546 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 11 23:01:17.308413 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.308394    2546 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 11 23:01:17.308452 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.308414    2546 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 11 23:01:17.358652 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.358580    2546 apiserver.go:52] "Watching apiserver"
Sep 11 23:01:17.379209 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.379180    2546 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 11 23:01:17.381467 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.381443    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-jdr7s","openshift-ovn-kubernetes/ovnkube-node-p7244","kube-system/global-pull-secret-syncer-5n55v","kube-system/konnectivity-agent-mnljs","openshift-image-registry/node-ca-zgtd5","openshift-multus/multus-lhmpr","openshift-multus/network-metrics-daemon-vbgsz","openshift-network-operator/iptables-alerter-zdnn9","kube-system/kube-apiserver-proxy-ip-10-0-128-161.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kv48f","openshift-cluster-node-tuning-operator/tuned-mrmnp","openshift-dns/node-resolver-vqs5t","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-161.ec2.internal","openshift-multus/multus-additional-cni-plugins-bv5gf"]
Sep 11 23:01:17.382711 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.382692    2546 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 11 23:01:17.383905 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.383890    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vbgsz"
Sep 11 23:01:17.383968 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:17.383952    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vbgsz" podUID="50607ccd-47f9-463a-bdc7-181484a00caa"
Sep 11 23:01:17.386130 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.386114    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mrmnp"
Sep 11 23:01:17.388177 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.388144    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5n55v"
Sep 11 23:01:17.388780 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:17.388631    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5n55v" podUID="3ad2cd03-8eae-49c4-8c94-ea45de3d3c7a"
Sep 11 23:01:17.388780 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.388690    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50607ccd-47f9-463a-bdc7-181484a00caa-metrics-certs\") pod \"network-metrics-daemon-vbgsz\" (UID: \"50607ccd-47f9-463a-bdc7-181484a00caa\") " pod="openshift-multus/network-metrics-daemon-vbgsz"
Sep 11 23:01:17.388780 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.388732    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5znv4\" (UniqueName: \"kubernetes.io/projected/50607ccd-47f9-463a-bdc7-181484a00caa-kube-api-access-5znv4\") pod \"network-metrics-daemon-vbgsz\" (UID: \"50607ccd-47f9-463a-bdc7-181484a00caa\") " pod="openshift-multus/network-metrics-daemon-vbgsz"
Sep 11 23:01:17.391048 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.389136    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 11 23:01:17.391048 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.389244    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 11 23:01:17.391048 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.389864    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9f5hm\""
Sep 11 23:01:17.391267 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.391107    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mnljs"
Sep 11 23:01:17.393649 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.393630    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zgtd5"
Sep 11 23:01:17.395657 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.395640    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-nf7mg\""
Sep 11 23:01:17.395796 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.395781    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-lhmpr"
Sep 11 23:01:17.395871 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.395810    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 11 23:01:17.396178 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.396145    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 11 23:01:17.396678 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.396665    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 11 23:01:17.396770 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.396753    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-85jbp\""
Sep 11 23:01:17.397126 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.397113    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 11 23:01:17.397317 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.397304    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 11 23:01:17.397365 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.397352    2546 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 23:01:17.397988 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.397967    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bv5gf"
Sep 11 23:01:17.398654 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.398636    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-dlgt8\""
Sep 11 23:01:17.398793 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.398779    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 11 23:01:17.399640 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.399616    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 11 23:01:17.399733 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.399718    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 11 23:01:17.400080 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.400067    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zdnn9"
Sep 11 23:01:17.400127 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.400084    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 11 23:01:17.400529 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.400515    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-nlbgt\""
Sep 11 23:01:17.400651 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.400636    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 11 23:01:17.400705 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.400698    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 11 23:01:17.402533 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.402426    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-p7244"
Sep 11 23:01:17.402762 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.402738    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 11 23:01:17.403114 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.403093    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 11 23:01:17.403303 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.403131    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 11 23:01:17.403303 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.403217    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-s2r6w\""
Sep 11 23:01:17.404824 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.404807    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kv48f"
Sep 11 23:01:17.405373 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.405356    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 11 23:01:17.405468 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.405455    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 11 23:01:17.406540 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.406496    2546 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-10 22:56:16 +0000 UTC" deadline="2028-06-12 01:16:51.71447038 +0000 UTC"
Sep 11 23:01:17.406540 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.406538    2546 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15338h15m34.307934376s"
Sep 11 23:01:17.406628 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.406557    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 11 23:01:17.406792 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.406776    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 11 23:01:17.406894 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.406777    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 11 23:01:17.406894 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.406779    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-jw4fx\""
Sep 11 23:01:17.406894 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.406839    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 11 23:01:17.407057 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.407044    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jdr7s"
Sep 11 23:01:17.407108 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:17.407093    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jdr7s" podUID="223f5b7c-785a-4cfb-9196-6f4adb77d14a"
Sep 11 23:01:17.407592 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.407578    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 11 23:01:17.407659 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.407584    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-94smz\""
Sep 11 23:01:17.407725 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.407708    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 11 23:01:17.407798 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.407786    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 11 23:01:17.409314 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.409296    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vqs5t"
Sep 11 23:01:17.411760 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.411745    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-8g7gw\""
Sep 11 23:01:17.411975 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.411959    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 11 23:01:17.412052 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.412023    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 11 23:01:17.417416 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.417400    2546 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-4sfsh"
Sep 11 23:01:17.428305 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.428289    2546 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-4sfsh"
Sep 11 23:01:17.435493 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:17.435465    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbf860b5b0097ab5f709024d70f9c3d5.slice/crio-1efa56b3e5654cffd8dc7fea3b053ff9f72ba8cc665704676e43db32096d056c WatchSource:0}: Error finding container 1efa56b3e5654cffd8dc7fea3b053ff9f72ba8cc665704676e43db32096d056c: Status 404 returned error can't find the container with id 1efa56b3e5654cffd8dc7fea3b053ff9f72ba8cc665704676e43db32096d056c
Sep 11 23:01:17.435719 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:17.435701    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode80b9e5ab597f3b17c8733c311f97f1f.slice/crio-3d12d9c32b711f69942c2dcc21e1f1532887aed067e07419fb26f9e9a599f9b8 WatchSource:0}: Error finding container 3d12d9c32b711f69942c2dcc21e1f1532887aed067e07419fb26f9e9a599f9b8: Status 404 returned error can't find the container with id 3d12d9c32b711f69942c2dcc21e1f1532887aed067e07419fb26f9e9a599f9b8
Sep 11 23:01:17.440232 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.440217    2546 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 23:01:17.484051 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.484031    2546 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 11 23:01:17.488849 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.488831    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a489f10b-bf16-40dc-aa5f-dc1d8228aa32-hostroot\") pod \"multus-lhmpr\" (UID: \"a489f10b-bf16-40dc-aa5f-dc1d8228aa32\") " pod="openshift-multus/multus-lhmpr"
Sep 11 23:01:17.488952 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.488863    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a489f10b-bf16-40dc-aa5f-dc1d8228aa32-multus-conf-dir\") pod \"multus-lhmpr\" (UID: \"a489f10b-bf16-40dc-aa5f-dc1d8228aa32\") " pod="openshift-multus/multus-lhmpr"
Sep 11 23:01:17.488952 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.488884    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlp9l\" (UniqueName: \"kubernetes.io/projected/8916993f-54e0-4346-bf2b-032ac759a2a6-kube-api-access-hlp9l\") pod \"aws-ebs-csi-driver-node-kv48f\" (UID: \"8916993f-54e0-4346-bf2b-032ac759a2a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kv48f"
Sep 11 23:01:17.488952 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.488905    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b0a8af3-932c-4091-b7a0-180fe1751a3a-host-kubelet\") pod \"ovnkube-node-p7244\" (UID: \"6b0a8af3-932c-4091-b7a0-180fe1751a3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7244"
Sep 11 23:01:17.488952 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.488926    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a489f10b-bf16-40dc-aa5f-dc1d8228aa32-host-var-lib-kubelet\") pod \"multus-lhmpr\" (UID: \"a489f10b-bf16-40dc-aa5f-dc1d8228aa32\") " pod="openshift-multus/multus-lhmpr"
Sep 11 23:01:17.489103 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.488962    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b0a8af3-932c-4091-b7a0-180fe1751a3a-etc-openvswitch\") pod \"ovnkube-node-p7244\" (UID: \"6b0a8af3-932c-4091-b7a0-180fe1751a3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7244"
Sep 11 23:01:17.489103 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.488981    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6b0a8af3-932c-4091-b7a0-180fe1751a3a-log-socket\") pod \"ovnkube-node-p7244\" (UID: \"6b0a8af3-932c-4091-b7a0-180fe1751a3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7244"
Sep 11 23:01:17.489103 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.488996    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6b0a8af3-932c-4091-b7a0-180fe1751a3a-host-cni-netd\") pod \"ovnkube-node-p7244\" (UID: \"6b0a8af3-932c-4091-b7a0-180fe1751a3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7244"
Sep 11 23:01:17.489103 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.489039    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a489f10b-bf16-40dc-aa5f-dc1d8228aa32-host-run-k8s-cni-cncf-io\") pod \"multus-lhmpr\" (UID: \"a489f10b-bf16-40dc-aa5f-dc1d8228aa32\") " pod="openshift-multus/multus-lhmpr"
Sep 11 23:01:17.489103 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.489079    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c27dbcc1-5027-44f3-9a8a-d1253a431256-etc-sysctl-conf\") pod \"tuned-mrmnp\" (UID: \"c27dbcc1-5027-44f3-9a8a-d1253a431256\") " pod="openshift-cluster-node-tuning-operator/tuned-mrmnp"
Sep 11 23:01:17.489103 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.489102    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0155dec9-f8e2-4007-9861-b8c870223adb-cnibin\") pod \"multus-additional-cni-plugins-bv5gf\" (UID: \"0155dec9-f8e2-4007-9861-b8c870223adb\") " pod="openshift-multus/multus-additional-cni-plugins-bv5gf"
Sep 11 23:01:17.489340 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.489131    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6b0a8af3-932c-4091-b7a0-180fe1751a3a-systemd-units\") pod \"ovnkube-node-p7244\" (UID: \"6b0a8af3-932c-4091-b7a0-180fe1751a3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7244"
Sep 11 23:01:17.489340 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.489181    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6b0a8af3-932c-4091-b7a0-180fe1751a3a-host-slash\") pod \"ovnkube-node-p7244\" (UID: \"6b0a8af3-932c-4091-b7a0-180fe1751a3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7244"
Sep 11 23:01:17.489340 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.489200    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6b0a8af3-932c-4091-b7a0-180fe1751a3a-run-systemd\") pod \"ovnkube-node-p7244\" (UID: \"6b0a8af3-932c-4091-b7a0-180fe1751a3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7244"
Sep 11 23:01:17.489340 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.489214    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b0a8af3-932c-4091-b7a0-180fe1751a3a-var-lib-openvswitch\") pod \"ovnkube-node-p7244\" (UID: \"6b0a8af3-932c-4091-b7a0-180fe1751a3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7244"
Sep 11 23:01:17.489340 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.489229    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c27dbcc1-5027-44f3-9a8a-d1253a431256-etc-kubernetes\") pod \"tuned-mrmnp\" (UID: \"c27dbcc1-5027-44f3-9a8a-d1253a431256\") " pod="openshift-cluster-node-tuning-operator/tuned-mrmnp"
Sep 11 23:01:17.489340 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.489264    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a489f10b-bf16-40dc-aa5f-dc1d8228aa32-host-var-lib-cni-bin\") pod \"multus-lhmpr\" (UID: \"a489f10b-bf16-40dc-aa5f-dc1d8228aa32\") " pod="openshift-multus/multus-lhmpr"
Sep 11 23:01:17.489340 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.489302    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8916993f-54e0-4346-bf2b-032ac759a2a6-device-dir\") pod \"aws-ebs-csi-driver-node-kv48f\" (UID: \"8916993f-54e0-4346-bf2b-032ac759a2a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kv48f"
Sep 11 23:01:17.489340 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.489320    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6b0a8af3-932c-4091-b7a0-180fe1751a3a-host-run-netns\") pod \"ovnkube-node-p7244\" (UID: \"6b0a8af3-932c-4091-b7a0-180fe1751a3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7244"
Sep 11 23:01:17.489340 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.489342    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6b0a8af3-932c-4091-b7a0-180fe1751a3a-host-cni-bin\") pod \"ovnkube-node-p7244\" (UID: \"6b0a8af3-932c-4091-b7a0-180fe1751a3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7244"
Sep 11 23:01:17.489710 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.489362    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b0a8af3-932c-4091-b7a0-180fe1751a3a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-p7244\" (UID: \"6b0a8af3-932c-4091-b7a0-180fe1751a3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7244"
Sep 11 23:01:17.489710 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.489382    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c27dbcc1-5027-44f3-9a8a-d1253a431256-sys\") pod \"tuned-mrmnp\" (UID: \"c27dbcc1-5027-44f3-9a8a-d1253a431256\") " pod="openshift-cluster-node-tuning-operator/tuned-mrmnp"
Sep 11 23:01:17.489710 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.489401    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50607ccd-47f9-463a-bdc7-181484a00caa-metrics-certs\") pod \"network-metrics-daemon-vbgsz\" (UID: \"50607ccd-47f9-463a-bdc7-181484a00caa\") " pod="openshift-multus/network-metrics-daemon-vbgsz"
Sep 11 23:01:17.489710 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.489446    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c27dbcc1-5027-44f3-9a8a-d1253a431256-etc-modprobe-d\") pod \"tuned-mrmnp\" (UID: \"c27dbcc1-5027-44f3-9a8a-d1253a431256\") " pod="openshift-cluster-node-tuning-operator/tuned-mrmnp"
Sep 11 23:01:17.489710 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.489471    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a489f10b-bf16-40dc-aa5f-dc1d8228aa32-multus-daemon-config\") pod \"multus-lhmpr\" (UID: \"a489f10b-bf16-40dc-aa5f-dc1d8228aa32\") " pod="openshift-multus/multus-lhmpr"
Sep 11 23:01:17.489710 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.489494    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx85g\" (UniqueName: \"kubernetes.io/projected/a489f10b-bf16-40dc-aa5f-dc1d8228aa32-kube-api-access-bx85g\") pod \"multus-lhmpr\" (UID: \"a489f10b-bf16-40dc-aa5f-dc1d8228aa32\") " pod="openshift-multus/multus-lhmpr"
Sep 11 23:01:17.489710 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.489513    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0155dec9-f8e2-4007-9861-b8c870223adb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bv5gf\" (UID: \"0155dec9-f8e2-4007-9861-b8c870223adb\") " pod="openshift-multus/multus-additional-cni-plugins-bv5gf"
Sep 11 23:01:17.489710 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.489543    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvp92\" (UniqueName: \"kubernetes.io/projected/157e8f79-c688-49dd-a216-86c09ffb4992-kube-api-access-cvp92\") pod \"iptables-alerter-zdnn9\" (UID: \"157e8f79-c688-49dd-a216-86c09ffb4992\") " pod="openshift-network-operator/iptables-alerter-zdnn9"
Sep 11 23:01:17.489710 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:17.489563    2546 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 23:01:17.489710 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.489578    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8916993f-54e0-4346-bf2b-032ac759a2a6-sys-fs\") pod \"aws-ebs-csi-driver-node-kv48f\" (UID: \"8916993f-54e0-4346-bf2b-032ac759a2a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kv48f"
Sep 11 23:01:17.489710 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:17.489625    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50607ccd-47f9-463a-bdc7-181484a00caa-metrics-certs podName:50607ccd-47f9-463a-bdc7-181484a00caa nodeName:}" failed. No retries permitted until 2026-09-11 23:01:17.989604507 +0000 UTC m=+2.022733702 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/50607ccd-47f9-463a-bdc7-181484a00caa-metrics-certs") pod "network-metrics-daemon-vbgsz" (UID: "50607ccd-47f9-463a-bdc7-181484a00caa") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 23:01:17.489710 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.489672    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3ad2cd03-8eae-49c4-8c94-ea45de3d3c7a-original-pull-secret\") pod \"global-pull-secret-syncer-5n55v\" (UID: \"3ad2cd03-8eae-49c4-8c94-ea45de3d3c7a\") " pod="kube-system/global-pull-secret-syncer-5n55v"
Sep 11 23:01:17.489710 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.489701    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a489f10b-bf16-40dc-aa5f-dc1d8228aa32-multus-socket-dir-parent\") pod \"multus-lhmpr\" (UID: \"a489f10b-bf16-40dc-aa5f-dc1d8228aa32\") " pod="openshift-multus/multus-lhmpr"
Sep 11 23:01:17.490308 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.489743    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a489f10b-bf16-40dc-aa5f-dc1d8228aa32-host-var-lib-cni-multus\") pod \"multus-lhmpr\" (UID: \"a489f10b-bf16-40dc-aa5f-dc1d8228aa32\") " pod="openshift-multus/multus-lhmpr"
Sep 11 23:01:17.490308 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.489764    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/aa21ad55-513d-46a1-89ce-78e88c3e0efa-serviceca\") pod \"node-ca-zgtd5\" (UID: \"aa21ad55-513d-46a1-89ce-78e88c3e0efa\") " pod="openshift-image-registry/node-ca-zgtd5"
Sep 11 23:01:17.490308 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.489788    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbptk\" (UniqueName: \"kubernetes.io/projected/aa21ad55-513d-46a1-89ce-78e88c3e0efa-kube-api-access-xbptk\") pod \"node-ca-zgtd5\" (UID: \"aa21ad55-513d-46a1-89ce-78e88c3e0efa\") " pod="openshift-image-registry/node-ca-zgtd5"
Sep 11 23:01:17.490308 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.489810    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6b0a8af3-932c-4091-b7a0-180fe1751a3a-node-log\") pod \"ovnkube-node-p7244\" (UID: \"6b0a8af3-932c-4091-b7a0-180fe1751a3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7244"
Sep 11 23:01:17.490308 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.489825    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b0a8af3-932c-4091-b7a0-180fe1751a3a-host-run-ovn-kubernetes\") pod \"ovnkube-node-p7244\" (UID: \"6b0a8af3-932c-4091-b7a0-180fe1751a3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7244"
Sep 11 23:01:17.490308 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.489839    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6b0a8af3-932c-4091-b7a0-180fe1751a3a-env-overrides\") pod \"ovnkube-node-p7244\" (UID: \"6b0a8af3-932c-4091-b7a0-180fe1751a3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7244"
Sep 11 23:01:17.490308 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.489866    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c27dbcc1-5027-44f3-9a8a-d1253a431256-run\") pod \"tuned-mrmnp\" (UID: \"c27dbcc1-5027-44f3-9a8a-d1253a431256\") " pod="openshift-cluster-node-tuning-operator/tuned-mrmnp"
Sep 11 23:01:17.490308 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.489910    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5znv4\" (UniqueName: \"kubernetes.io/projected/50607ccd-47f9-463a-bdc7-181484a00caa-kube-api-access-5znv4\") pod \"network-metrics-daemon-vbgsz\" (UID: \"50607ccd-47f9-463a-bdc7-181484a00caa\") " pod="openshift-multus/network-metrics-daemon-vbgsz"
Sep 11 23:01:17.490308 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.489927    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c27dbcc1-5027-44f3-9a8a-d1253a431256-tmp\") pod \"tuned-mrmnp\" (UID: \"c27dbcc1-5027-44f3-9a8a-d1253a431256\") " pod="openshift-cluster-node-tuning-operator/tuned-mrmnp"
Sep 11 23:01:17.490308 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.489943    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/157e8f79-c688-49dd-a216-86c09ffb4992-host-slash\") pod \"iptables-alerter-zdnn9\" (UID: \"157e8f79-c688-49dd-a216-86c09ffb4992\") " pod="openshift-network-operator/iptables-alerter-zdnn9"
Sep 11 23:01:17.490308 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.489979    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8916993f-54e0-4346-bf2b-032ac759a2a6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kv48f\" (UID: \"8916993f-54e0-4346-bf2b-032ac759a2a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kv48f"
Sep 11 23:01:17.490308 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.490002    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9dcaac8a-b529-4ba2-a3f3-a617817741c2-tmp-dir\") pod \"node-resolver-vqs5t\" (UID: \"9dcaac8a-b529-4ba2-a3f3-a617817741c2\") " pod="openshift-dns/node-resolver-vqs5t"
Sep 11 23:01:17.490308 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.490025    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c27dbcc1-5027-44f3-9a8a-d1253a431256-var-lib-kubelet\") pod \"tuned-mrmnp\" (UID: \"c27dbcc1-5027-44f3-9a8a-d1253a431256\") " pod="openshift-cluster-node-tuning-operator/tuned-mrmnp"
Sep 11 23:01:17.490308 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.490046    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnhjx\" (UniqueName: \"kubernetes.io/projected/c27dbcc1-5027-44f3-9a8a-d1253a431256-kube-api-access-xnhjx\") pod \"tuned-mrmnp\" (UID: \"c27dbcc1-5027-44f3-9a8a-d1253a431256\") " pod="openshift-cluster-node-tuning-operator/tuned-mrmnp"
Sep 11 23:01:17.490308 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.490061    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a489f10b-bf16-40dc-aa5f-dc1d8228aa32-host-run-netns\") pod \"multus-lhmpr\" (UID: \"a489f10b-bf16-40dc-aa5f-dc1d8228aa32\") " pod="openshift-multus/multus-lhmpr"
Sep 11 23:01:17.490308 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.490076    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0155dec9-f8e2-4007-9861-b8c870223adb-system-cni-dir\") pod \"multus-additional-cni-plugins-bv5gf\" (UID: \"0155dec9-f8e2-4007-9861-b8c870223adb\") " pod="openshift-multus/multus-additional-cni-plugins-bv5gf"
Sep 11 23:01:17.491020 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.490090    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qqt9\" (UniqueName: \"kubernetes.io/projected/0155dec9-f8e2-4007-9861-b8c870223adb-kube-api-access-9qqt9\") pod \"multus-additional-cni-plugins-bv5gf\" (UID: \"0155dec9-f8e2-4007-9861-b8c870223adb\") " pod="openshift-multus/multus-additional-cni-plugins-bv5gf"
Sep 11 23:01:17.491020 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.490106    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/157e8f79-c688-49dd-a216-86c09ffb4992-iptables-alerter-script\") pod \"iptables-alerter-zdnn9\" (UID: \"157e8f79-c688-49dd-a216-86c09ffb4992\") " pod="openshift-network-operator/iptables-alerter-zdnn9"
Sep 11 23:01:17.491020 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.490123    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d2174c19-0971-419a-bc04-0c4662d5ff3c-agent-certs\") pod \"konnectivity-agent-mnljs\" (UID: \"d2174c19-0971-419a-bc04-0c4662d5ff3c\") " pod="kube-system/konnectivity-agent-mnljs"
Sep 11 23:01:17.491020 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.490168    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c27dbcc1-5027-44f3-9a8a-d1253a431256-etc-systemd\") pod \"tuned-mrmnp\" (UID: \"c27dbcc1-5027-44f3-9a8a-d1253a431256\") " pod="openshift-cluster-node-tuning-operator/tuned-mrmnp"
Sep 11 23:01:17.491020 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.490190    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c27dbcc1-5027-44f3-9a8a-d1253a431256-etc-sysconfig\") pod \"tuned-mrmnp\" (UID: \"c27dbcc1-5027-44f3-9a8a-d1253a431256\") " pod="openshift-cluster-node-tuning-operator/tuned-mrmnp"
Sep 11 23:01:17.491020 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.490211    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d2174c19-0971-419a-bc04-0c4662d5ff3c-konnectivity-ca\") pod \"konnectivity-agent-mnljs\" (UID: \"d2174c19-0971-419a-bc04-0c4662d5ff3c\") " pod="kube-system/konnectivity-agent-mnljs"
Sep 11 23:01:17.491020 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.490231    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdk7z\" (UniqueName: \"kubernetes.io/projected/6b0a8af3-932c-4091-b7a0-180fe1751a3a-kube-api-access-tdk7z\") pod \"ovnkube-node-p7244\" (UID: \"6b0a8af3-932c-4091-b7a0-180fe1751a3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7244"
Sep 11 23:01:17.491020 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.490251    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c27dbcc1-5027-44f3-9a8a-d1253a431256-etc-tuned\") pod \"tuned-mrmnp\" (UID: \"c27dbcc1-5027-44f3-9a8a-d1253a431256\") " pod="openshift-cluster-node-tuning-operator/tuned-mrmnp"
Sep 11 23:01:17.491020 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.490272    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a489f10b-bf16-40dc-aa5f-dc1d8228aa32-system-cni-dir\") pod \"multus-lhmpr\" (UID: \"a489f10b-bf16-40dc-aa5f-dc1d8228aa32\") " pod="openshift-multus/multus-lhmpr"
Sep 11 23:01:17.491020 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.490295    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0155dec9-f8e2-4007-9861-b8c870223adb-cni-binary-copy\") pod \"multus-additional-cni-plugins-bv5gf\" (UID: \"0155dec9-f8e2-4007-9861-b8c870223adb\") " pod="openshift-multus/multus-additional-cni-plugins-bv5gf"
Sep 11 23:01:17.491020 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.490326    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3ad2cd03-8eae-49c4-8c94-ea45de3d3c7a-kubelet-config\") pod \"global-pull-secret-syncer-5n55v\" (UID: \"3ad2cd03-8eae-49c4-8c94-ea45de3d3c7a\") " pod="kube-system/global-pull-secret-syncer-5n55v"
Sep 11 23:01:17.491020 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.490351    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3ad2cd03-8eae-49c4-8c94-ea45de3d3c7a-dbus\") pod \"global-pull-secret-syncer-5n55v\" (UID: \"3ad2cd03-8eae-49c4-8c94-ea45de3d3c7a\") " pod="kube-system/global-pull-secret-syncer-5n55v"
Sep 11 23:01:17.491020 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.490373    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a489f10b-bf16-40dc-aa5f-dc1d8228aa32-multus-cni-dir\") pod \"multus-lhmpr\" (UID: \"a489f10b-bf16-40dc-aa5f-dc1d8228aa32\") " pod="openshift-multus/multus-lhmpr"
Sep 11 23:01:17.491020 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.490389    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8916993f-54e0-4346-bf2b-032ac759a2a6-socket-dir\") pod \"aws-ebs-csi-driver-node-kv48f\" (UID: \"8916993f-54e0-4346-bf2b-032ac759a2a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kv48f"
Sep 11 23:01:17.491020 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.490422    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8916993f-54e0-4346-bf2b-032ac759a2a6-etc-selinux\") pod \"aws-ebs-csi-driver-node-kv48f\" (UID: \"8916993f-54e0-4346-bf2b-032ac759a2a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kv48f"
Sep 11 23:01:17.491020 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.490463    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/aa21ad55-513d-46a1-89ce-78e88c3e0efa-host\") pod \"node-ca-zgtd5\" (UID: \"aa21ad55-513d-46a1-89ce-78e88c3e0efa\") " pod="openshift-image-registry/node-ca-zgtd5"
Sep 11 23:01:17.491812 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.490504    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b0a8af3-932c-4091-b7a0-180fe1751a3a-run-openvswitch\") pod \"ovnkube-node-p7244\" (UID: \"6b0a8af3-932c-4091-b7a0-180fe1751a3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7244"
Sep 11 23:01:17.491812 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.490531    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6b0a8af3-932c-4091-b7a0-180fe1751a3a-run-ovn\") pod \"ovnkube-node-p7244\" (UID: \"6b0a8af3-932c-4091-b7a0-180fe1751a3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7244"
Sep 11 23:01:17.491812 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.490558    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6b0a8af3-932c-4091-b7a0-180fe1751a3a-ovnkube-config\") pod \"ovnkube-node-p7244\" (UID: \"6b0a8af3-932c-4091-b7a0-180fe1751a3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7244"
Sep 11 23:01:17.491812 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.490580    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6b0a8af3-932c-4091-b7a0-180fe1751a3a-ovnkube-script-lib\") pod \"ovnkube-node-p7244\" (UID: \"6b0a8af3-932c-4091-b7a0-180fe1751a3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7244"
Sep 11 23:01:17.491812 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.490610    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c27dbcc1-5027-44f3-9a8a-d1253a431256-lib-modules\") pod \"tuned-mrmnp\" (UID: \"c27dbcc1-5027-44f3-9a8a-d1253a431256\") " pod="openshift-cluster-node-tuning-operator/tuned-mrmnp"
Sep 11 23:01:17.491812 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.490648    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8916993f-54e0-4346-bf2b-032ac759a2a6-registration-dir\") pod \"aws-ebs-csi-driver-node-kv48f\" (UID: \"8916993f-54e0-4346-bf2b-032ac759a2a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kv48f"
Sep 11 23:01:17.491812 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.490672    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9dcaac8a-b529-4ba2-a3f3-a617817741c2-hosts-file\") pod \"node-resolver-vqs5t\" (UID: \"9dcaac8a-b529-4ba2-a3f3-a617817741c2\") " pod="openshift-dns/node-resolver-vqs5t"
Sep 11 23:01:17.491812 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.490696    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cn25v\" (UniqueName: \"kubernetes.io/projected/223f5b7c-785a-4cfb-9196-6f4adb77d14a-kube-api-access-cn25v\") pod \"network-check-target-jdr7s\" (UID: \"223f5b7c-785a-4cfb-9196-6f4adb77d14a\") " pod="openshift-network-diagnostics/network-check-target-jdr7s"
Sep 11 23:01:17.491812 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.490728    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c27dbcc1-5027-44f3-9a8a-d1253a431256-etc-sysctl-d\") pod \"tuned-mrmnp\" (UID: \"c27dbcc1-5027-44f3-9a8a-d1253a431256\") " pod="openshift-cluster-node-tuning-operator/tuned-mrmnp"
Sep 11 23:01:17.491812 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.490754    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a489f10b-bf16-40dc-aa5f-dc1d8228aa32-cnibin\") pod \"multus-lhmpr\" (UID: \"a489f10b-bf16-40dc-aa5f-dc1d8228aa32\") " pod="openshift-multus/multus-lhmpr"
Sep 11 23:01:17.491812 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.490786    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a489f10b-bf16-40dc-aa5f-dc1d8228aa32-host-run-multus-certs\") pod \"multus-lhmpr\" (UID: \"a489f10b-bf16-40dc-aa5f-dc1d8228aa32\") " pod="openshift-multus/multus-lhmpr"
Sep 11 23:01:17.491812 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.490814    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0155dec9-f8e2-4007-9861-b8c870223adb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bv5gf\" (UID: \"0155dec9-f8e2-4007-9861-b8c870223adb\") " pod="openshift-multus/multus-additional-cni-plugins-bv5gf"
Sep 11 23:01:17.491812 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.490834    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6b0a8af3-932c-4091-b7a0-180fe1751a3a-ovn-node-metrics-cert\") pod \"ovnkube-node-p7244\" (UID: \"6b0a8af3-932c-4091-b7a0-180fe1751a3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7244"
Sep 11 23:01:17.491812 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.490853    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a489f10b-bf16-40dc-aa5f-dc1d8228aa32-cni-binary-copy\") pod \"multus-lhmpr\" (UID: \"a489f10b-bf16-40dc-aa5f-dc1d8228aa32\") " pod="openshift-multus/multus-lhmpr"
Sep 11 23:01:17.491812 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.490878    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a489f10b-bf16-40dc-aa5f-dc1d8228aa32-etc-kubernetes\") pod \"multus-lhmpr\" (UID: \"a489f10b-bf16-40dc-aa5f-dc1d8228aa32\") " pod="openshift-multus/multus-lhmpr"
Sep 11 23:01:17.491812 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.490899    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c27dbcc1-5027-44f3-9a8a-d1253a431256-host\") pod \"tuned-mrmnp\" (UID: \"c27dbcc1-5027-44f3-9a8a-d1253a431256\") " pod="openshift-cluster-node-tuning-operator/tuned-mrmnp"
Sep 11 23:01:17.492355 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.490918    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0155dec9-f8e2-4007-9861-b8c870223adb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bv5gf\" (UID: \"0155dec9-f8e2-4007-9861-b8c870223adb\") " pod="openshift-multus/multus-additional-cni-plugins-bv5gf"
Sep 11 23:01:17.492355 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.490948    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsvr6\" (UniqueName: \"kubernetes.io/projected/9dcaac8a-b529-4ba2-a3f3-a617817741c2-kube-api-access-bsvr6\") pod \"node-resolver-vqs5t\" (UID: \"9dcaac8a-b529-4ba2-a3f3-a617817741c2\") " pod="openshift-dns/node-resolver-vqs5t"
Sep 11 23:01:17.492355 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.490952    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-161.ec2.internal" event={"ID":"e80b9e5ab597f3b17c8733c311f97f1f","Type":"ContainerStarted","Data":"3d12d9c32b711f69942c2dcc21e1f1532887aed067e07419fb26f9e9a599f9b8"}
Sep 11 23:01:17.492355 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.492145    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-161.ec2.internal" event={"ID":"fbf860b5b0097ab5f709024d70f9c3d5","Type":"ContainerStarted","Data":"1efa56b3e5654cffd8dc7fea3b053ff9f72ba8cc665704676e43db32096d056c"}
Sep 11 23:01:17.498863 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.498846    2546 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 11 23:01:17.502334 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.502312    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5znv4\" (UniqueName: \"kubernetes.io/projected/50607ccd-47f9-463a-bdc7-181484a00caa-kube-api-access-5znv4\") pod \"network-metrics-daemon-vbgsz\" (UID: \"50607ccd-47f9-463a-bdc7-181484a00caa\") " pod="openshift-multus/network-metrics-daemon-vbgsz"
Sep 11 23:01:17.591297 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.591265    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b0a8af3-932c-4091-b7a0-180fe1751a3a-host-run-ovn-kubernetes\") pod \"ovnkube-node-p7244\" (UID: \"6b0a8af3-932c-4091-b7a0-180fe1751a3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7244"
Sep 11 23:01:17.591297 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.591295    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6b0a8af3-932c-4091-b7a0-180fe1751a3a-env-overrides\") pod \"ovnkube-node-p7244\" (UID: \"6b0a8af3-932c-4091-b7a0-180fe1751a3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7244"
Sep 11 23:01:17.591525 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.591314    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c27dbcc1-5027-44f3-9a8a-d1253a431256-run\") pod \"tuned-mrmnp\" (UID: \"c27dbcc1-5027-44f3-9a8a-d1253a431256\") " pod="openshift-cluster-node-tuning-operator/tuned-mrmnp"
Sep 11 23:01:17.591525 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.591340    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c27dbcc1-5027-44f3-9a8a-d1253a431256-tmp\") pod \"tuned-mrmnp\" (UID: \"c27dbcc1-5027-44f3-9a8a-d1253a431256\") " pod="openshift-cluster-node-tuning-operator/tuned-mrmnp"
Sep 11 23:01:17.591525 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.591366    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/157e8f79-c688-49dd-a216-86c09ffb4992-host-slash\") pod \"iptables-alerter-zdnn9\" (UID: \"157e8f79-c688-49dd-a216-86c09ffb4992\") " pod="openshift-network-operator/iptables-alerter-zdnn9"
Sep 11 23:01:17.591525 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.591374    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b0a8af3-932c-4091-b7a0-180fe1751a3a-host-run-ovn-kubernetes\") pod \"ovnkube-node-p7244\" (UID: \"6b0a8af3-932c-4091-b7a0-180fe1751a3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7244"
Sep 11 23:01:17.591525 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.591388    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8916993f-54e0-4346-bf2b-032ac759a2a6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kv48f\" (UID: \"8916993f-54e0-4346-bf2b-032ac759a2a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kv48f"
Sep 11 23:01:17.591525 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.591422    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9dcaac8a-b529-4ba2-a3f3-a617817741c2-tmp-dir\") pod \"node-resolver-vqs5t\" (UID: \"9dcaac8a-b529-4ba2-a3f3-a617817741c2\") " pod="openshift-dns/node-resolver-vqs5t"
Sep 11 23:01:17.591525 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.591438    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/157e8f79-c688-49dd-a216-86c09ffb4992-host-slash\") pod \"iptables-alerter-zdnn9\" (UID: \"157e8f79-c688-49dd-a216-86c09ffb4992\") " pod="openshift-network-operator/iptables-alerter-zdnn9"
Sep 11 23:01:17.591525 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.591445    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8916993f-54e0-4346-bf2b-032ac759a2a6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kv48f\" (UID: \"8916993f-54e0-4346-bf2b-032ac759a2a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kv48f"
Sep 11 23:01:17.591525 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.591474    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c27dbcc1-5027-44f3-9a8a-d1253a431256-var-lib-kubelet\") pod \"tuned-mrmnp\" (UID: \"c27dbcc1-5027-44f3-9a8a-d1253a431256\") " pod="openshift-cluster-node-tuning-operator/tuned-mrmnp"
Sep 11 23:01:17.591525 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.591447    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c27dbcc1-5027-44f3-9a8a-d1253a431256-var-lib-kubelet\") pod \"tuned-mrmnp\" (UID: \"c27dbcc1-5027-44f3-9a8a-d1253a431256\") " pod="openshift-cluster-node-tuning-operator/tuned-mrmnp"
Sep 11 23:01:17.591525 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.591498    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnhjx\" (UniqueName: \"kubernetes.io/projected/c27dbcc1-5027-44f3-9a8a-d1253a431256-kube-api-access-xnhjx\") pod \"tuned-mrmnp\" (UID: \"c27dbcc1-5027-44f3-9a8a-d1253a431256\") " pod="openshift-cluster-node-tuning-operator/tuned-mrmnp"
Sep 11 23:01:17.591525 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.591513    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a489f10b-bf16-40dc-aa5f-dc1d8228aa32-host-run-netns\") pod \"multus-lhmpr\" (UID: \"a489f10b-bf16-40dc-aa5f-dc1d8228aa32\") " pod="openshift-multus/multus-lhmpr"
Sep 11 23:01:17.592079 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.591534    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0155dec9-f8e2-4007-9861-b8c870223adb-system-cni-dir\") pod \"multus-additional-cni-plugins-bv5gf\" (UID: \"0155dec9-f8e2-4007-9861-b8c870223adb\") " pod="openshift-multus/multus-additional-cni-plugins-bv5gf"
Sep 11 23:01:17.592079 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.591559    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qqt9\" (UniqueName: \"kubernetes.io/projected/0155dec9-f8e2-4007-9861-b8c870223adb-kube-api-access-9qqt9\") pod \"multus-additional-cni-plugins-bv5gf\" (UID: \"0155dec9-f8e2-4007-9861-b8c870223adb\") " pod="openshift-multus/multus-additional-cni-plugins-bv5gf"
Sep 11 23:01:17.592079 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.591582    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0155dec9-f8e2-4007-9861-b8c870223adb-system-cni-dir\") pod \"multus-additional-cni-plugins-bv5gf\" (UID: \"0155dec9-f8e2-4007-9861-b8c870223adb\") " pod="openshift-multus/multus-additional-cni-plugins-bv5gf"
Sep 11 23:01:17.592079 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.591612    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/157e8f79-c688-49dd-a216-86c09ffb4992-iptables-alerter-script\") pod \"iptables-alerter-zdnn9\" (UID: \"157e8f79-c688-49dd-a216-86c09ffb4992\") " pod="openshift-network-operator/iptables-alerter-zdnn9"
Sep 11 23:01:17.592079 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.591420    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c27dbcc1-5027-44f3-9a8a-d1253a431256-run\") pod \"tuned-mrmnp\" (UID: \"c27dbcc1-5027-44f3-9a8a-d1253a431256\") " pod="openshift-cluster-node-tuning-operator/tuned-mrmnp"
Sep 11 23:01:17.592079 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.591638    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d2174c19-0971-419a-bc04-0c4662d5ff3c-agent-certs\") pod \"konnectivity-agent-mnljs\" (UID: \"d2174c19-0971-419a-bc04-0c4662d5ff3c\") " pod="kube-system/konnectivity-agent-mnljs"
Sep 11 23:01:17.592079 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.591690    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c27dbcc1-5027-44f3-9a8a-d1253a431256-etc-systemd\") pod \"tuned-mrmnp\" (UID: \"c27dbcc1-5027-44f3-9a8a-d1253a431256\") " pod="openshift-cluster-node-tuning-operator/tuned-mrmnp"
Sep 11 23:01:17.592079 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.591717    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c27dbcc1-5027-44f3-9a8a-d1253a431256-etc-sysconfig\") pod \"tuned-mrmnp\" (UID: \"c27dbcc1-5027-44f3-9a8a-d1253a431256\") " pod="openshift-cluster-node-tuning-operator/tuned-mrmnp"
Sep 11 23:01:17.592079 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.591696    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9dcaac8a-b529-4ba2-a3f3-a617817741c2-tmp-dir\") pod \"node-resolver-vqs5t\" (UID: \"9dcaac8a-b529-4ba2-a3f3-a617817741c2\") " pod="openshift-dns/node-resolver-vqs5t"
Sep 11 23:01:17.592079 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.591581    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a489f10b-bf16-40dc-aa5f-dc1d8228aa32-host-run-netns\") pod \"multus-lhmpr\" (UID: \"a489f10b-bf16-40dc-aa5f-dc1d8228aa32\") " pod="openshift-multus/multus-lhmpr"
Sep 11 23:01:17.592079 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.591755    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d2174c19-0971-419a-bc04-0c4662d5ff3c-konnectivity-ca\") pod \"konnectivity-agent-mnljs\" (UID: \"d2174c19-0971-419a-bc04-0c4662d5ff3c\") " pod="kube-system/konnectivity-agent-mnljs"
Sep 11 23:01:17.592079 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.591786    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdk7z\" (UniqueName: \"kubernetes.io/projected/6b0a8af3-932c-4091-b7a0-180fe1751a3a-kube-api-access-tdk7z\") pod \"ovnkube-node-p7244\" (UID: \"6b0a8af3-932c-4091-b7a0-180fe1751a3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7244"
Sep 11 23:01:17.592079 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.591810    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c27dbcc1-5027-44f3-9a8a-d1253a431256-etc-tuned\") pod \"tuned-mrmnp\" (UID: \"c27dbcc1-5027-44f3-9a8a-d1253a431256\") " pod="openshift-cluster-node-tuning-operator/tuned-mrmnp"
Sep 11 23:01:17.592079 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.591810    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c27dbcc1-5027-44f3-9a8a-d1253a431256-etc-systemd\") pod \"tuned-mrmnp\" (UID: \"c27dbcc1-5027-44f3-9a8a-d1253a431256\") " pod="openshift-cluster-node-tuning-operator/tuned-mrmnp"
Sep 11 23:01:17.592079 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.591832    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a489f10b-bf16-40dc-aa5f-dc1d8228aa32-system-cni-dir\") pod \"multus-lhmpr\" (UID: \"a489f10b-bf16-40dc-aa5f-dc1d8228aa32\") " pod="openshift-multus/multus-lhmpr"
Sep 11 23:01:17.592079 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.591860    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0155dec9-f8e2-4007-9861-b8c870223adb-cni-binary-copy\") pod \"multus-additional-cni-plugins-bv5gf\" (UID: \"0155dec9-f8e2-4007-9861-b8c870223adb\") " pod="openshift-multus/multus-additional-cni-plugins-bv5gf"
Sep 11 23:01:17.592079 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.591908    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3ad2cd03-8eae-49c4-8c94-ea45de3d3c7a-kubelet-config\") pod \"global-pull-secret-syncer-5n55v\" (UID: \"3ad2cd03-8eae-49c4-8c94-ea45de3d3c7a\") " pod="kube-system/global-pull-secret-syncer-5n55v"
Sep 11 23:01:17.592895 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.591929    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3ad2cd03-8eae-49c4-8c94-ea45de3d3c7a-dbus\") pod \"global-pull-secret-syncer-5n55v\" (UID: \"3ad2cd03-8eae-49c4-8c94-ea45de3d3c7a\") " pod="kube-system/global-pull-secret-syncer-5n55v"
Sep 11 23:01:17.592895 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.591954    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a489f10b-bf16-40dc-aa5f-dc1d8228aa32-multus-cni-dir\") pod \"multus-lhmpr\" (UID: \"a489f10b-bf16-40dc-aa5f-dc1d8228aa32\") " pod="openshift-multus/multus-lhmpr"
Sep 11 23:01:17.592895 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.591976    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8916993f-54e0-4346-bf2b-032ac759a2a6-socket-dir\") pod \"aws-ebs-csi-driver-node-kv48f\" (UID: \"8916993f-54e0-4346-bf2b-032ac759a2a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kv48f"
Sep 11 23:01:17.592895 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.592003    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8916993f-54e0-4346-bf2b-032ac759a2a6-etc-selinux\") pod \"aws-ebs-csi-driver-node-kv48f\" (UID: \"8916993f-54e0-4346-bf2b-032ac759a2a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kv48f"
Sep 11 23:01:17.592895 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.592026    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/aa21ad55-513d-46a1-89ce-78e88c3e0efa-host\") pod \"node-ca-zgtd5\" (UID: \"aa21ad55-513d-46a1-89ce-78e88c3e0efa\") " pod="openshift-image-registry/node-ca-zgtd5"
Sep 11 23:01:17.592895 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.592053    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b0a8af3-932c-4091-b7a0-180fe1751a3a-run-openvswitch\") pod \"ovnkube-node-p7244\" (UID: \"6b0a8af3-932c-4091-b7a0-180fe1751a3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7244"
Sep 11 23:01:17.592895 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.592077    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6b0a8af3-932c-4091-b7a0-180fe1751a3a-run-ovn\") pod \"ovnkube-node-p7244\" (UID: \"6b0a8af3-932c-4091-b7a0-180fe1751a3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7244"
Sep 11 23:01:17.592895 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.592092    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3ad2cd03-8eae-49c4-8c94-ea45de3d3c7a-kubelet-config\") pod \"global-pull-secret-syncer-5n55v\" (UID: \"3ad2cd03-8eae-49c4-8c94-ea45de3d3c7a\") " pod="kube-system/global-pull-secret-syncer-5n55v"
Sep 11 23:01:17.592895 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.592105    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6b0a8af3-932c-4091-b7a0-180fe1751a3a-ovnkube-config\") pod \"ovnkube-node-p7244\" (UID: \"6b0a8af3-932c-4091-b7a0-180fe1751a3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7244"
Sep 11 23:01:17.592895 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.592210    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6b0a8af3-932c-4091-b7a0-180fe1751a3a-ovnkube-script-lib\") pod \"ovnkube-node-p7244\" (UID: \"6b0a8af3-932c-4091-b7a0-180fe1751a3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7244"
Sep 11 23:01:17.592895 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.592234    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c27dbcc1-5027-44f3-9a8a-d1253a431256-lib-modules\") pod \"tuned-mrmnp\" (UID: \"c27dbcc1-5027-44f3-9a8a-d1253a431256\") " pod="openshift-cluster-node-tuning-operator/tuned-mrmnp"
Sep 11 23:01:17.592895 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.592234    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8916993f-54e0-4346-bf2b-032ac759a2a6-etc-selinux\") pod \"aws-ebs-csi-driver-node-kv48f\" (UID: \"8916993f-54e0-4346-bf2b-032ac759a2a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kv48f"
Sep 11 23:01:17.592895 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.592259    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8916993f-54e0-4346-bf2b-032ac759a2a6-registration-dir\") pod \"aws-ebs-csi-driver-node-kv48f\" (UID: \"8916993f-54e0-4346-bf2b-032ac759a2a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kv48f"
Sep 11 23:01:17.592895 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.592235    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a489f10b-bf16-40dc-aa5f-dc1d8228aa32-system-cni-dir\") pod \"multus-lhmpr\" (UID: \"a489f10b-bf16-40dc-aa5f-dc1d8228aa32\") " pod="openshift-multus/multus-lhmpr"
Sep 11 23:01:17.592895 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.592285    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9dcaac8a-b529-4ba2-a3f3-a617817741c2-hosts-file\") pod \"node-resolver-vqs5t\" (UID: \"9dcaac8a-b529-4ba2-a3f3-a617817741c2\") " pod="openshift-dns/node-resolver-vqs5t"
Sep 11 23:01:17.592895 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.591882    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6b0a8af3-932c-4091-b7a0-180fe1751a3a-env-overrides\") pod \"ovnkube-node-p7244\" (UID: \"6b0a8af3-932c-4091-b7a0-180fe1751a3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7244"
Sep 11 23:01:17.592895 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.592344    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3ad2cd03-8eae-49c4-8c94-ea45de3d3c7a-dbus\") pod \"global-pull-secret-syncer-5n55v\" (UID: \"3ad2cd03-8eae-49c4-8c94-ea45de3d3c7a\") " pod="kube-system/global-pull-secret-syncer-5n55v"
Sep 11 23:01:17.593755 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.592368    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/157e8f79-c688-49dd-a216-86c09ffb4992-iptables-alerter-script\") pod \"iptables-alerter-zdnn9\" (UID: \"157e8f79-c688-49dd-a216-86c09ffb4992\") " pod="openshift-network-operator/iptables-alerter-zdnn9"
Sep 11 23:01:17.593755 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.592398    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d2174c19-0971-419a-bc04-0c4662d5ff3c-konnectivity-ca\") pod \"konnectivity-agent-mnljs\" (UID: \"d2174c19-0971-419a-bc04-0c4662d5ff3c\") " pod="kube-system/konnectivity-agent-mnljs"
Sep 11 23:01:17.593755 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.592459    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c27dbcc1-5027-44f3-9a8a-d1253a431256-etc-sysconfig\") pod \"tuned-mrmnp\" (UID: \"c27dbcc1-5027-44f3-9a8a-d1253a431256\") " pod="openshift-cluster-node-tuning-operator/tuned-mrmnp"
Sep 11 23:01:17.593755 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.592467    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a489f10b-bf16-40dc-aa5f-dc1d8228aa32-multus-cni-dir\") pod \"multus-lhmpr\" (UID: \"a489f10b-bf16-40dc-aa5f-dc1d8228aa32\") " pod="openshift-multus/multus-lhmpr"
Sep 11 23:01:17.593755 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.592475    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8916993f-54e0-4346-bf2b-032ac759a2a6-socket-dir\") pod \"aws-ebs-csi-driver-node-kv48f\" (UID: \"8916993f-54e0-4346-bf2b-032ac759a2a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kv48f"
Sep 11 23:01:17.593755 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.592559    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9dcaac8a-b529-4ba2-a3f3-a617817741c2-hosts-file\") pod \"node-resolver-vqs5t\" (UID: \"9dcaac8a-b529-4ba2-a3f3-a617817741c2\") " pod="openshift-dns/node-resolver-vqs5t"
Sep 11 23:01:17.593755 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.592592    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b0a8af3-932c-4091-b7a0-180fe1751a3a-run-openvswitch\") pod \"ovnkube-node-p7244\" (UID: \"6b0a8af3-932c-4091-b7a0-180fe1751a3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7244"
Sep 11 23:01:17.593755 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.592619    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8916993f-54e0-4346-bf2b-032ac759a2a6-registration-dir\") pod \"aws-ebs-csi-driver-node-kv48f\" (UID: \"8916993f-54e0-4346-bf2b-032ac759a2a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kv48f"
Sep 11 23:01:17.593755 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.592666    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6b0a8af3-932c-4091-b7a0-180fe1751a3a-run-ovn\") pod \"ovnkube-node-p7244\" (UID: \"6b0a8af3-932c-4091-b7a0-180fe1751a3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7244"
Sep 11 23:01:17.593755 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.592710    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cn25v\" (UniqueName: \"kubernetes.io/projected/223f5b7c-785a-4cfb-9196-6f4adb77d14a-kube-api-access-cn25v\") pod \"network-check-target-jdr7s\" (UID: \"223f5b7c-785a-4cfb-9196-6f4adb77d14a\") " pod="openshift-network-diagnostics/network-check-target-jdr7s"
Sep 11 23:01:17.593755 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.592726    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c27dbcc1-5027-44f3-9a8a-d1253a431256-lib-modules\") pod \"tuned-mrmnp\" (UID: \"c27dbcc1-5027-44f3-9a8a-d1253a431256\") " pod="openshift-cluster-node-tuning-operator/tuned-mrmnp"
Sep 11 23:01:17.593755 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.592752    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c27dbcc1-5027-44f3-9a8a-d1253a431256-etc-sysctl-d\") pod \"tuned-mrmnp\" (UID: \"c27dbcc1-5027-44f3-9a8a-d1253a431256\") " pod="openshift-cluster-node-tuning-operator/tuned-mrmnp"
Sep 11 23:01:17.593755 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.593008    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a489f10b-bf16-40dc-aa5f-dc1d8228aa32-cnibin\") pod \"multus-lhmpr\" (UID: \"a489f10b-bf16-40dc-aa5f-dc1d8228aa32\") " pod="openshift-multus/multus-lhmpr"
Sep 11 23:01:17.593755 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.593041    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a489f10b-bf16-40dc-aa5f-dc1d8228aa32-host-run-multus-certs\") pod \"multus-lhmpr\" (UID: \"a489f10b-bf16-40dc-aa5f-dc1d8228aa32\") " pod="openshift-multus/multus-lhmpr"
Sep 11 23:01:17.593755 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.593049    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6b0a8af3-932c-4091-b7a0-180fe1751a3a-ovnkube-script-lib\") pod \"ovnkube-node-p7244\" (UID: \"6b0a8af3-932c-4091-b7a0-180fe1751a3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7244"
Sep 11 23:01:17.593755 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.593080    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0155dec9-f8e2-4007-9861-b8c870223adb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bv5gf\" (UID: \"0155dec9-f8e2-4007-9861-b8c870223adb\") " pod="openshift-multus/multus-additional-cni-plugins-bv5gf"
Sep 11 23:01:17.593755 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.593104    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6b0a8af3-932c-4091-b7a0-180fe1751a3a-ovn-node-metrics-cert\") pod \"ovnkube-node-p7244\" (UID: \"6b0a8af3-932c-4091-b7a0-180fe1751a3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7244"
Sep 11 23:01:17.594636 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.593129    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a489f10b-bf16-40dc-aa5f-dc1d8228aa32-cni-binary-copy\") pod \"multus-lhmpr\" (UID: \"a489f10b-bf16-40dc-aa5f-dc1d8228aa32\") " pod="openshift-multus/multus-lhmpr"
Sep 11 23:01:17.594636 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.593170    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c27dbcc1-5027-44f3-9a8a-d1253a431256-etc-sysctl-d\") pod \"tuned-mrmnp\" (UID: \"c27dbcc1-5027-44f3-9a8a-d1253a431256\") " pod="openshift-cluster-node-tuning-operator/tuned-mrmnp"
Sep 11 23:01:17.594636 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.593170    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0155dec9-f8e2-4007-9861-b8c870223adb-cni-binary-copy\") pod \"multus-additional-cni-plugins-bv5gf\" (UID: \"0155dec9-f8e2-4007-9861-b8c870223adb\") " pod="openshift-multus/multus-additional-cni-plugins-bv5gf"
Sep 11 23:01:17.594636 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.593217    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a489f10b-bf16-40dc-aa5f-dc1d8228aa32-etc-kubernetes\") pod \"multus-lhmpr\" (UID: \"a489f10b-bf16-40dc-aa5f-dc1d8228aa32\") " pod="openshift-multus/multus-lhmpr"
Sep 11 23:01:17.594636 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.593244    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c27dbcc1-5027-44f3-9a8a-d1253a431256-host\") pod \"tuned-mrmnp\" (UID: \"c27dbcc1-5027-44f3-9a8a-d1253a431256\") " pod="openshift-cluster-node-tuning-operator/tuned-mrmnp"
Sep 11 23:01:17.594636 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.593231    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/aa21ad55-513d-46a1-89ce-78e88c3e0efa-host\") pod \"node-ca-zgtd5\" (UID: \"aa21ad55-513d-46a1-89ce-78e88c3e0efa\") " pod="openshift-image-registry/node-ca-zgtd5"
Sep 11 23:01:17.594636 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.593273    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0155dec9-f8e2-4007-9861-b8c870223adb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bv5gf\" (UID: \"0155dec9-f8e2-4007-9861-b8c870223adb\") " pod="openshift-multus/multus-additional-cni-plugins-bv5gf"
Sep 11 23:01:17.594636 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.593290    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6b0a8af3-932c-4091-b7a0-180fe1751a3a-ovnkube-config\") pod \"ovnkube-node-p7244\" (UID: \"6b0a8af3-932c-4091-b7a0-180fe1751a3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7244"
Sep 11 23:01:17.594636 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.593327    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0155dec9-f8e2-4007-9861-b8c870223adb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bv5gf\" (UID: \"0155dec9-f8e2-4007-9861-b8c870223adb\") " pod="openshift-multus/multus-additional-cni-plugins-bv5gf"
Sep 11 23:01:17.594636 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.593350    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a489f10b-bf16-40dc-aa5f-dc1d8228aa32-cnibin\") pod \"multus-lhmpr\" (UID: \"a489f10b-bf16-40dc-aa5f-dc1d8228aa32\") " pod="openshift-multus/multus-lhmpr"
Sep 11 23:01:17.594636 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.593357    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c27dbcc1-5027-44f3-9a8a-d1253a431256-host\") pod \"tuned-mrmnp\" (UID: \"c27dbcc1-5027-44f3-9a8a-d1253a431256\") " pod="openshift-cluster-node-tuning-operator/tuned-mrmnp"
Sep 11 23:01:17.594636 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.593398    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsvr6\" (UniqueName: \"kubernetes.io/projected/9dcaac8a-b529-4ba2-a3f3-a617817741c2-kube-api-access-bsvr6\") pod \"node-resolver-vqs5t\" (UID: \"9dcaac8a-b529-4ba2-a3f3-a617817741c2\") " pod="openshift-dns/node-resolver-vqs5t"
Sep 11 23:01:17.594636 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.593454    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a489f10b-bf16-40dc-aa5f-dc1d8228aa32-hostroot\") pod \"multus-lhmpr\" (UID: \"a489f10b-bf16-40dc-aa5f-dc1d8228aa32\") " pod="openshift-multus/multus-lhmpr"
Sep 11 23:01:17.594636 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.593490    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a489f10b-bf16-40dc-aa5f-dc1d8228aa32-multus-conf-dir\") pod \"multus-lhmpr\" (UID: \"a489f10b-bf16-40dc-aa5f-dc1d8228aa32\") " pod="openshift-multus/multus-lhmpr"
Sep 11 23:01:17.594636 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.593523    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlp9l\" (UniqueName: \"kubernetes.io/projected/8916993f-54e0-4346-bf2b-032ac759a2a6-kube-api-access-hlp9l\") pod \"aws-ebs-csi-driver-node-kv48f\" (UID: \"8916993f-54e0-4346-bf2b-032ac759a2a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kv48f"
Sep 11 23:01:17.594636 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.593578    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a489f10b-bf16-40dc-aa5f-dc1d8228aa32-hostroot\") pod \"multus-lhmpr\" (UID: \"a489f10b-bf16-40dc-aa5f-dc1d8228aa32\") " pod="openshift-multus/multus-lhmpr"
Sep 11 23:01:17.594636 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.593603    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b0a8af3-932c-4091-b7a0-180fe1751a3a-host-kubelet\") pod \"ovnkube-node-p7244\" (UID: \"6b0a8af3-932c-4091-b7a0-180fe1751a3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7244"
Sep 11 23:01:17.594636 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.593615    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a489f10b-bf16-40dc-aa5f-dc1d8228aa32-multus-conf-dir\") pod \"multus-lhmpr\" (UID: \"a489f10b-bf16-40dc-aa5f-dc1d8228aa32\") " pod="openshift-multus/multus-lhmpr"
Sep 11 23:01:17.595535 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.593635    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a489f10b-bf16-40dc-aa5f-dc1d8228aa32-host-var-lib-kubelet\") pod \"multus-lhmpr\" (UID: \"a489f10b-bf16-40dc-aa5f-dc1d8228aa32\") " pod="openshift-multus/multus-lhmpr"
Sep 11 23:01:17.595535 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.593645    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b0a8af3-932c-4091-b7a0-180fe1751a3a-host-kubelet\") pod \"ovnkube-node-p7244\" (UID: \"6b0a8af3-932c-4091-b7a0-180fe1751a3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7244"
Sep 11 23:01:17.595535 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.593677    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b0a8af3-932c-4091-b7a0-180fe1751a3a-etc-openvswitch\") pod \"ovnkube-node-p7244\" (UID: \"6b0a8af3-932c-4091-b7a0-180fe1751a3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7244"
Sep 11 23:01:17.595535 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.593680    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a489f10b-bf16-40dc-aa5f-dc1d8228aa32-host-var-lib-kubelet\") pod \"multus-lhmpr\" (UID: \"a489f10b-bf16-40dc-aa5f-dc1d8228aa32\") " pod="openshift-multus/multus-lhmpr"
Sep 11 23:01:17.595535 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.593718    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6b0a8af3-932c-4091-b7a0-180fe1751a3a-log-socket\") pod \"ovnkube-node-p7244\" (UID: \"6b0a8af3-932c-4091-b7a0-180fe1751a3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7244"
Sep 11 23:01:17.595535 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.593723    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a489f10b-bf16-40dc-aa5f-dc1d8228aa32-host-run-multus-certs\") pod \"multus-lhmpr\" (UID: \"a489f10b-bf16-40dc-aa5f-dc1d8228aa32\") " pod="openshift-multus/multus-lhmpr"
Sep 11 23:01:17.595535 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.593736    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0155dec9-f8e2-4007-9861-b8c870223adb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bv5gf\" (UID: \"0155dec9-f8e2-4007-9861-b8c870223adb\") " pod="openshift-multus/multus-additional-cni-plugins-bv5gf"
Sep 11 23:01:17.595535 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.593765    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6b0a8af3-932c-4091-b7a0-180fe1751a3a-log-socket\") pod \"ovnkube-node-p7244\" (UID: \"6b0a8af3-932c-4091-b7a0-180fe1751a3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7244"
Sep 11 23:01:17.595535 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.593761    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b0a8af3-932c-4091-b7a0-180fe1751a3a-etc-openvswitch\") pod \"ovnkube-node-p7244\" (UID: \"6b0a8af3-932c-4091-b7a0-180fe1751a3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7244"
Sep 11 23:01:17.595535 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.593803    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6b0a8af3-932c-4091-b7a0-180fe1751a3a-host-cni-netd\") pod \"ovnkube-node-p7244\" (UID: \"6b0a8af3-932c-4091-b7a0-180fe1751a3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7244"
Sep 11 23:01:17.595535 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.593829    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a489f10b-bf16-40dc-aa5f-dc1d8228aa32-host-run-k8s-cni-cncf-io\") pod \"multus-lhmpr\" (UID: \"a489f10b-bf16-40dc-aa5f-dc1d8228aa32\") " pod="openshift-multus/multus-lhmpr"
Sep 11 23:01:17.595535 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.593815    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a489f10b-bf16-40dc-aa5f-dc1d8228aa32-etc-kubernetes\") pod \"multus-lhmpr\" (UID: \"a489f10b-bf16-40dc-aa5f-dc1d8228aa32\") " pod="openshift-multus/multus-lhmpr"
Sep 11 23:01:17.595535 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.593861    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c27dbcc1-5027-44f3-9a8a-d1253a431256-etc-sysctl-conf\") pod \"tuned-mrmnp\" (UID: \"c27dbcc1-5027-44f3-9a8a-d1253a431256\") " pod="openshift-cluster-node-tuning-operator/tuned-mrmnp"
Sep 11 23:01:17.595535 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.593887    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0155dec9-f8e2-4007-9861-b8c870223adb-cnibin\") pod \"multus-additional-cni-plugins-bv5gf\" (UID: \"0155dec9-f8e2-4007-9861-b8c870223adb\") " pod="openshift-multus/multus-additional-cni-plugins-bv5gf"
Sep 11 23:01:17.595535 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.593890    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a489f10b-bf16-40dc-aa5f-dc1d8228aa32-host-run-k8s-cni-cncf-io\") pod \"multus-lhmpr\" (UID: \"a489f10b-bf16-40dc-aa5f-dc1d8228aa32\") " pod="openshift-multus/multus-lhmpr"
Sep 11 23:01:17.595535 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.593917    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6b0a8af3-932c-4091-b7a0-180fe1751a3a-systemd-units\") pod \"ovnkube-node-p7244\" (UID: \"6b0a8af3-932c-4091-b7a0-180fe1751a3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7244"
Sep 11 23:01:17.595535 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.593863    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6b0a8af3-932c-4091-b7a0-180fe1751a3a-host-cni-netd\") pod \"ovnkube-node-p7244\" (UID: \"6b0a8af3-932c-4091-b7a0-180fe1751a3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7244"
Sep 11 23:01:17.595535 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.593942    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6b0a8af3-932c-4091-b7a0-180fe1751a3a-host-slash\") pod \"ovnkube-node-p7244\" (UID: \"6b0a8af3-932c-4091-b7a0-180fe1751a3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7244"
Sep 11 23:01:17.596254 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.593964    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6b0a8af3-932c-4091-b7a0-180fe1751a3a-run-systemd\") pod \"ovnkube-node-p7244\" (UID: \"6b0a8af3-932c-4091-b7a0-180fe1751a3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7244"
Sep 11 23:01:17.596254 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.593973    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0155dec9-f8e2-4007-9861-b8c870223adb-cnibin\") pod \"multus-additional-cni-plugins-bv5gf\" (UID: \"0155dec9-f8e2-4007-9861-b8c870223adb\") " pod="openshift-multus/multus-additional-cni-plugins-bv5gf"
Sep 11 23:01:17.596254 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.593988    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b0a8af3-932c-4091-b7a0-180fe1751a3a-var-lib-openvswitch\") pod \"ovnkube-node-p7244\" (UID: \"6b0a8af3-932c-4091-b7a0-180fe1751a3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7244"
Sep 11 23:01:17.596254 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.594013    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c27dbcc1-5027-44f3-9a8a-d1253a431256-etc-kubernetes\") pod \"tuned-mrmnp\" (UID: \"c27dbcc1-5027-44f3-9a8a-d1253a431256\") " pod="openshift-cluster-node-tuning-operator/tuned-mrmnp"
Sep 11 23:01:17.596254 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.594015    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c27dbcc1-5027-44f3-9a8a-d1253a431256-etc-sysctl-conf\") pod \"tuned-mrmnp\" (UID: \"c27dbcc1-5027-44f3-9a8a-d1253a431256\") " pod="openshift-cluster-node-tuning-operator/tuned-mrmnp"
Sep 11 23:01:17.596254 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.594017    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6b0a8af3-932c-4091-b7a0-180fe1751a3a-host-slash\") pod \"ovnkube-node-p7244\" (UID: \"6b0a8af3-932c-4091-b7a0-180fe1751a3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7244"
Sep 11 23:01:17.596254 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.594035    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a489f10b-bf16-40dc-aa5f-dc1d8228aa32-host-var-lib-cni-bin\") pod \"multus-lhmpr\" (UID: \"a489f10b-bf16-40dc-aa5f-dc1d8228aa32\") " pod="openshift-multus/multus-lhmpr"
Sep 11 23:01:17.596254 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.594048    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6b0a8af3-932c-4091-b7a0-180fe1751a3a-systemd-units\") pod \"ovnkube-node-p7244\" (UID: \"6b0a8af3-932c-4091-b7a0-180fe1751a3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7244"
Sep 11 23:01:17.596254 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.594063    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b0a8af3-932c-4091-b7a0-180fe1751a3a-var-lib-openvswitch\") pod \"ovnkube-node-p7244\" (UID: \"6b0a8af3-932c-4091-b7a0-180fe1751a3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7244"
Sep 11 23:01:17.596254 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.594062    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8916993f-54e0-4346-bf2b-032ac759a2a6-device-dir\") pod \"aws-ebs-csi-driver-node-kv48f\" (UID: \"8916993f-54e0-4346-bf2b-032ac759a2a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kv48f"
Sep 11 23:01:17.596254 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.594098    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c27dbcc1-5027-44f3-9a8a-d1253a431256-etc-kubernetes\") pod \"tuned-mrmnp\" (UID: \"c27dbcc1-5027-44f3-9a8a-d1253a431256\") " pod="openshift-cluster-node-tuning-operator/tuned-mrmnp"
Sep 11 23:01:17.596254 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.594105    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8916993f-54e0-4346-bf2b-032ac759a2a6-device-dir\") pod \"aws-ebs-csi-driver-node-kv48f\" (UID: \"8916993f-54e0-4346-bf2b-032ac759a2a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kv48f"
Sep 11 23:01:17.596254 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.594100    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6b0a8af3-932c-4091-b7a0-180fe1751a3a-host-run-netns\") pod \"ovnkube-node-p7244\" (UID: \"6b0a8af3-932c-4091-b7a0-180fe1751a3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7244"
Sep 11 23:01:17.596254 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.594124    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6b0a8af3-932c-4091-b7a0-180fe1751a3a-host-run-netns\") pod \"ovnkube-node-p7244\" (UID: \"6b0a8af3-932c-4091-b7a0-180fe1751a3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7244"
Sep 11 23:01:17.596254 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.594138    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6b0a8af3-932c-4091-b7a0-180fe1751a3a-host-cni-bin\") pod \"ovnkube-node-p7244\" (UID: \"6b0a8af3-932c-4091-b7a0-180fe1751a3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7244"
Sep 11 23:01:17.596254 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.594190    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b0a8af3-932c-4091-b7a0-180fe1751a3a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-p7244\" (UID: \"6b0a8af3-932c-4091-b7a0-180fe1751a3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7244"
Sep 11 23:01:17.596254 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.594198    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6b0a8af3-932c-4091-b7a0-180fe1751a3a-host-cni-bin\") pod \"ovnkube-node-p7244\" (UID: \"6b0a8af3-932c-4091-b7a0-180fe1751a3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7244"
Sep 11 23:01:17.596805 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.594215    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c27dbcc1-5027-44f3-9a8a-d1253a431256-sys\") pod \"tuned-mrmnp\" (UID: \"c27dbcc1-5027-44f3-9a8a-d1253a431256\") " pod="openshift-cluster-node-tuning-operator/tuned-mrmnp"
Sep 11 23:01:17.596805 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.594234    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a489f10b-bf16-40dc-aa5f-dc1d8228aa32-host-var-lib-cni-bin\") pod \"multus-lhmpr\" (UID: \"a489f10b-bf16-40dc-aa5f-dc1d8228aa32\") " pod="openshift-multus/multus-lhmpr"
Sep 11 23:01:17.596805 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.594140    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6b0a8af3-932c-4091-b7a0-180fe1751a3a-run-systemd\") pod \"ovnkube-node-p7244\" (UID: \"6b0a8af3-932c-4091-b7a0-180fe1751a3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7244"
Sep 11 23:01:17.596805 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.594286    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c27dbcc1-5027-44f3-9a8a-d1253a431256-etc-modprobe-d\") pod \"tuned-mrmnp\" (UID: \"c27dbcc1-5027-44f3-9a8a-d1253a431256\") " pod="openshift-cluster-node-tuning-operator/tuned-mrmnp"
Sep 11 23:01:17.596805 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.594266    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b0a8af3-932c-4091-b7a0-180fe1751a3a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-p7244\" (UID: \"6b0a8af3-932c-4091-b7a0-180fe1751a3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7244"
Sep 11 23:01:17.596805 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.594323    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a489f10b-bf16-40dc-aa5f-dc1d8228aa32-multus-daemon-config\") pod \"multus-lhmpr\" (UID: \"a489f10b-bf16-40dc-aa5f-dc1d8228aa32\") " pod="openshift-multus/multus-lhmpr"
Sep 11 23:01:17.596805 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.594337    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c27dbcc1-5027-44f3-9a8a-d1253a431256-sys\") pod \"tuned-mrmnp\" (UID: \"c27dbcc1-5027-44f3-9a8a-d1253a431256\") " pod="openshift-cluster-node-tuning-operator/tuned-mrmnp"
Sep 11 23:01:17.596805 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.594361    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx85g\" (UniqueName: \"kubernetes.io/projected/a489f10b-bf16-40dc-aa5f-dc1d8228aa32-kube-api-access-bx85g\") pod \"multus-lhmpr\" (UID: \"a489f10b-bf16-40dc-aa5f-dc1d8228aa32\") " pod="openshift-multus/multus-lhmpr"
Sep 11 23:01:17.596805 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.594376    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c27dbcc1-5027-44f3-9a8a-d1253a431256-etc-modprobe-d\") pod \"tuned-mrmnp\" (UID: \"c27dbcc1-5027-44f3-9a8a-d1253a431256\") " pod="openshift-cluster-node-tuning-operator/tuned-mrmnp"
Sep 11 23:01:17.596805 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.594436    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0155dec9-f8e2-4007-9861-b8c870223adb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bv5gf\" (UID: \"0155dec9-f8e2-4007-9861-b8c870223adb\") " pod="openshift-multus/multus-additional-cni-plugins-bv5gf"
Sep 11 23:01:17.596805 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.594548    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvp92\" (UniqueName: \"kubernetes.io/projected/157e8f79-c688-49dd-a216-86c09ffb4992-kube-api-access-cvp92\") pod \"iptables-alerter-zdnn9\" (UID: \"157e8f79-c688-49dd-a216-86c09ffb4992\") " pod="openshift-network-operator/iptables-alerter-zdnn9"
Sep 11 23:01:17.596805 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.594576    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8916993f-54e0-4346-bf2b-032ac759a2a6-sys-fs\") pod \"aws-ebs-csi-driver-node-kv48f\" (UID: \"8916993f-54e0-4346-bf2b-032ac759a2a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kv48f"
Sep 11 23:01:17.596805 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.594650    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a489f10b-bf16-40dc-aa5f-dc1d8228aa32-cni-binary-copy\") pod \"multus-lhmpr\" (UID: \"a489f10b-bf16-40dc-aa5f-dc1d8228aa32\") " pod="openshift-multus/multus-lhmpr"
Sep 11 23:01:17.596805 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.594724    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8916993f-54e0-4346-bf2b-032ac759a2a6-sys-fs\") pod \"aws-ebs-csi-driver-node-kv48f\" (UID: \"8916993f-54e0-4346-bf2b-032ac759a2a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kv48f"
Sep 11 23:01:17.596805 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.594759    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3ad2cd03-8eae-49c4-8c94-ea45de3d3c7a-original-pull-secret\") pod \"global-pull-secret-syncer-5n55v\" (UID: \"3ad2cd03-8eae-49c4-8c94-ea45de3d3c7a\") " pod="kube-system/global-pull-secret-syncer-5n55v"
Sep 11 23:01:17.596805 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.594781    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a489f10b-bf16-40dc-aa5f-dc1d8228aa32-multus-daemon-config\") pod \"multus-lhmpr\" (UID: \"a489f10b-bf16-40dc-aa5f-dc1d8228aa32\") " pod="openshift-multus/multus-lhmpr"
Sep 11 23:01:17.596805 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.594816    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a489f10b-bf16-40dc-aa5f-dc1d8228aa32-multus-socket-dir-parent\") pod \"multus-lhmpr\" (UID: \"a489f10b-bf16-40dc-aa5f-dc1d8228aa32\") " pod="openshift-multus/multus-lhmpr"
Sep 11 23:01:17.596805 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:17.594835    2546 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 23:01:17.597389 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.594845    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a489f10b-bf16-40dc-aa5f-dc1d8228aa32-host-var-lib-cni-multus\") pod \"multus-lhmpr\" (UID: \"a489f10b-bf16-40dc-aa5f-dc1d8228aa32\") " pod="openshift-multus/multus-lhmpr"
Sep 11 23:01:17.597389 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.594870    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/aa21ad55-513d-46a1-89ce-78e88c3e0efa-serviceca\") pod \"node-ca-zgtd5\" (UID: \"aa21ad55-513d-46a1-89ce-78e88c3e0efa\") " pod="openshift-image-registry/node-ca-zgtd5"
Sep 11 23:01:17.597389 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.594896    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbptk\" (UniqueName: \"kubernetes.io/projected/aa21ad55-513d-46a1-89ce-78e88c3e0efa-kube-api-access-xbptk\") pod \"node-ca-zgtd5\" (UID: \"aa21ad55-513d-46a1-89ce-78e88c3e0efa\") " pod="openshift-image-registry/node-ca-zgtd5"
Sep 11 23:01:17.597389 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.594925    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6b0a8af3-932c-4091-b7a0-180fe1751a3a-node-log\") pod \"ovnkube-node-p7244\" (UID: \"6b0a8af3-932c-4091-b7a0-180fe1751a3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7244"
Sep 11 23:01:17.597389 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.594956    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a489f10b-bf16-40dc-aa5f-dc1d8228aa32-host-var-lib-cni-multus\") pod \"multus-lhmpr\" (UID: \"a489f10b-bf16-40dc-aa5f-dc1d8228aa32\") " pod="openshift-multus/multus-lhmpr"
Sep 11 23:01:17.597389 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.595010    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6b0a8af3-932c-4091-b7a0-180fe1751a3a-node-log\") pod \"ovnkube-node-p7244\" (UID: \"6b0a8af3-932c-4091-b7a0-180fe1751a3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7244"
Sep 11 23:01:17.597389 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:17.595027    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ad2cd03-8eae-49c4-8c94-ea45de3d3c7a-original-pull-secret podName:3ad2cd03-8eae-49c4-8c94-ea45de3d3c7a nodeName:}" failed. No retries permitted until 2026-09-11 23:01:18.094996548 +0000 UTC m=+2.128125744 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3ad2cd03-8eae-49c4-8c94-ea45de3d3c7a-original-pull-secret") pod "global-pull-secret-syncer-5n55v" (UID: "3ad2cd03-8eae-49c4-8c94-ea45de3d3c7a") : object "kube-system"/"original-pull-secret" not registered
Sep 11 23:01:17.597389 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.595044    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a489f10b-bf16-40dc-aa5f-dc1d8228aa32-multus-socket-dir-parent\") pod \"multus-lhmpr\" (UID: \"a489f10b-bf16-40dc-aa5f-dc1d8228aa32\") " pod="openshift-multus/multus-lhmpr"
Sep 11 23:01:17.597389 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.595208    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c27dbcc1-5027-44f3-9a8a-d1253a431256-tmp\") pod \"tuned-mrmnp\" (UID: \"c27dbcc1-5027-44f3-9a8a-d1253a431256\") " pod="openshift-cluster-node-tuning-operator/tuned-mrmnp"
Sep 11 23:01:17.597389 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.595282    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c27dbcc1-5027-44f3-9a8a-d1253a431256-etc-tuned\") pod \"tuned-mrmnp\" (UID: \"c27dbcc1-5027-44f3-9a8a-d1253a431256\") " pod="openshift-cluster-node-tuning-operator/tuned-mrmnp"
Sep 11 23:01:17.597389 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.595286    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0155dec9-f8e2-4007-9861-b8c870223adb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bv5gf\" (UID: \"0155dec9-f8e2-4007-9861-b8c870223adb\") " pod="openshift-multus/multus-additional-cni-plugins-bv5gf"
Sep 11 23:01:17.597389 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.595560    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d2174c19-0971-419a-bc04-0c4662d5ff3c-agent-certs\") pod \"konnectivity-agent-mnljs\" (UID: \"d2174c19-0971-419a-bc04-0c4662d5ff3c\") " pod="kube-system/konnectivity-agent-mnljs"
Sep 11 23:01:17.597389 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.595859    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/aa21ad55-513d-46a1-89ce-78e88c3e0efa-serviceca\") pod \"node-ca-zgtd5\" (UID: \"aa21ad55-513d-46a1-89ce-78e88c3e0efa\") " pod="openshift-image-registry/node-ca-zgtd5"
Sep 11 23:01:17.597389 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.596891    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6b0a8af3-932c-4091-b7a0-180fe1751a3a-ovn-node-metrics-cert\") pod \"ovnkube-node-p7244\" (UID: \"6b0a8af3-932c-4091-b7a0-180fe1751a3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7244"
Sep 11 23:01:17.601404 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:17.601380    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 23:01:17.601404 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:17.601407    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 23:01:17.601565 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:17.601421    2546 projected.go:196] Error preparing data for projected volume kube-api-access-cn25v for pod openshift-network-diagnostics/network-check-target-jdr7s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 23:01:17.601565 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:17.601481    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/223f5b7c-785a-4cfb-9196-6f4adb77d14a-kube-api-access-cn25v podName:223f5b7c-785a-4cfb-9196-6f4adb77d14a nodeName:}" failed. No retries permitted until 2026-09-11 23:01:18.101466746 +0000 UTC m=+2.134595942 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-cn25v" (UniqueName: "kubernetes.io/projected/223f5b7c-785a-4cfb-9196-6f4adb77d14a-kube-api-access-cn25v") pod "network-check-target-jdr7s" (UID: "223f5b7c-785a-4cfb-9196-6f4adb77d14a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 23:01:17.604120 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.604097    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnhjx\" (UniqueName: \"kubernetes.io/projected/c27dbcc1-5027-44f3-9a8a-d1253a431256-kube-api-access-xnhjx\") pod \"tuned-mrmnp\" (UID: \"c27dbcc1-5027-44f3-9a8a-d1253a431256\") " pod="openshift-cluster-node-tuning-operator/tuned-mrmnp"
Sep 11 23:01:17.604285 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.604126    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qqt9\" (UniqueName: \"kubernetes.io/projected/0155dec9-f8e2-4007-9861-b8c870223adb-kube-api-access-9qqt9\") pod \"multus-additional-cni-plugins-bv5gf\" (UID: \"0155dec9-f8e2-4007-9861-b8c870223adb\") " pod="openshift-multus/multus-additional-cni-plugins-bv5gf"
Sep 11 23:01:17.604469 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.604451    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx85g\" (UniqueName: \"kubernetes.io/projected/a489f10b-bf16-40dc-aa5f-dc1d8228aa32-kube-api-access-bx85g\") pod \"multus-lhmpr\" (UID: \"a489f10b-bf16-40dc-aa5f-dc1d8228aa32\") " pod="openshift-multus/multus-lhmpr"
Sep 11 23:01:17.604658 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.604635    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvp92\" (UniqueName: \"kubernetes.io/projected/157e8f79-c688-49dd-a216-86c09ffb4992-kube-api-access-cvp92\") pod \"iptables-alerter-zdnn9\" (UID: \"157e8f79-c688-49dd-a216-86c09ffb4992\") " pod="openshift-network-operator/iptables-alerter-zdnn9"
Sep 11 23:01:17.605007 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.604984    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsvr6\" (UniqueName: \"kubernetes.io/projected/9dcaac8a-b529-4ba2-a3f3-a617817741c2-kube-api-access-bsvr6\") pod \"node-resolver-vqs5t\" (UID: \"9dcaac8a-b529-4ba2-a3f3-a617817741c2\") " pod="openshift-dns/node-resolver-vqs5t"
Sep 11 23:01:17.605062 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.605021    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlp9l\" (UniqueName: \"kubernetes.io/projected/8916993f-54e0-4346-bf2b-032ac759a2a6-kube-api-access-hlp9l\") pod \"aws-ebs-csi-driver-node-kv48f\" (UID: \"8916993f-54e0-4346-bf2b-032ac759a2a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kv48f"
Sep 11 23:01:17.605214 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.605199    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdk7z\" (UniqueName: \"kubernetes.io/projected/6b0a8af3-932c-4091-b7a0-180fe1751a3a-kube-api-access-tdk7z\") pod \"ovnkube-node-p7244\" (UID: \"6b0a8af3-932c-4091-b7a0-180fe1751a3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7244"
Sep 11 23:01:17.605798 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.605784    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbptk\" (UniqueName: \"kubernetes.io/projected/aa21ad55-513d-46a1-89ce-78e88c3e0efa-kube-api-access-xbptk\") pod \"node-ca-zgtd5\" (UID: \"aa21ad55-513d-46a1-89ce-78e88c3e0efa\") " pod="openshift-image-registry/node-ca-zgtd5"
Sep 11 23:01:17.621812 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.621761    2546 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 23:01:17.716072 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.716038    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mrmnp"
Sep 11 23:01:17.727682 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:17.727654    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc27dbcc1_5027_44f3_9a8a_d1253a431256.slice/crio-9dad3707f405638e79fc90f7359489d20c6273ae6a0524b6f65ed57e1e2b59ac WatchSource:0}: Error finding container 9dad3707f405638e79fc90f7359489d20c6273ae6a0524b6f65ed57e1e2b59ac: Status 404 returned error can't find the container with id 9dad3707f405638e79fc90f7359489d20c6273ae6a0524b6f65ed57e1e2b59ac
Sep 11 23:01:17.743268 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.743241    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mnljs"
Sep 11 23:01:17.751250 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.751226    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zgtd5"
Sep 11 23:01:17.752937 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:17.752915    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2174c19_0971_419a_bc04_0c4662d5ff3c.slice/crio-a2e3152cabaefa4d3c4b69c87e09f709084a26bb0b00d44ac7a7dc03d3ac3c21 WatchSource:0}: Error finding container a2e3152cabaefa4d3c4b69c87e09f709084a26bb0b00d44ac7a7dc03d3ac3c21: Status 404 returned error can't find the container with id a2e3152cabaefa4d3c4b69c87e09f709084a26bb0b00d44ac7a7dc03d3ac3c21
Sep 11 23:01:17.756122 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.755897    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-lhmpr"
Sep 11 23:01:17.760499 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.760483    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bv5gf"
Sep 11 23:01:17.761984 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:17.761964    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa21ad55_513d_46a1_89ce_78e88c3e0efa.slice/crio-612d2bb5e6646d8cf745d1d2b3fe2f73fcee01c1bbdacfd4965a0f22fdf55ab6 WatchSource:0}: Error finding container 612d2bb5e6646d8cf745d1d2b3fe2f73fcee01c1bbdacfd4965a0f22fdf55ab6: Status 404 returned error can't find the container with id 612d2bb5e6646d8cf745d1d2b3fe2f73fcee01c1bbdacfd4965a0f22fdf55ab6
Sep 11 23:01:17.764717 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:17.764694    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda489f10b_bf16_40dc_aa5f_dc1d8228aa32.slice/crio-749148ceb0e619824254d76727a717c85207bbed8787ebe208282f9cc75cf05d WatchSource:0}: Error finding container 749148ceb0e619824254d76727a717c85207bbed8787ebe208282f9cc75cf05d: Status 404 returned error can't find the container with id 749148ceb0e619824254d76727a717c85207bbed8787ebe208282f9cc75cf05d
Sep 11 23:01:17.767838 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.767813    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zdnn9"
Sep 11 23:01:17.769552 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:17.769534    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0155dec9_f8e2_4007_9861_b8c870223adb.slice/crio-e1384ee41b26d85ddb957eb710d32fa8fa3d08e0773a07f8696a63a1978b2d8b WatchSource:0}: Error finding container e1384ee41b26d85ddb957eb710d32fa8fa3d08e0773a07f8696a63a1978b2d8b: Status 404 returned error can't find the container with id e1384ee41b26d85ddb957eb710d32fa8fa3d08e0773a07f8696a63a1978b2d8b
Sep 11 23:01:17.773609 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.773592    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-p7244"
Sep 11 23:01:17.776643 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:17.776610    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod157e8f79_c688_49dd_a216_86c09ffb4992.slice/crio-901bb8b4fd7db3cd07d44cd202058dd2918bb5e00de43531c1f4dac40398b859 WatchSource:0}: Error finding container 901bb8b4fd7db3cd07d44cd202058dd2918bb5e00de43531c1f4dac40398b859: Status 404 returned error can't find the container with id 901bb8b4fd7db3cd07d44cd202058dd2918bb5e00de43531c1f4dac40398b859
Sep 11 23:01:17.777915 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.777850    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kv48f"
Sep 11 23:01:17.782341 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.782325    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vqs5t"
Sep 11 23:01:17.782679 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:17.782649    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b0a8af3_932c_4091_b7a0_180fe1751a3a.slice/crio-04ff54982ed9240d067ab1312f8a21e759359955321675e64f06d9ee7bfa3f1c WatchSource:0}: Error finding container 04ff54982ed9240d067ab1312f8a21e759359955321675e64f06d9ee7bfa3f1c: Status 404 returned error can't find the container with id 04ff54982ed9240d067ab1312f8a21e759359955321675e64f06d9ee7bfa3f1c
Sep 11 23:01:17.788513 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:17.788365    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8916993f_54e0_4346_bf2b_032ac759a2a6.slice/crio-49a0b33bad51870f07867bb13d48e4017e13986d21795f39de6a19eeb8af6409 WatchSource:0}: Error finding container 49a0b33bad51870f07867bb13d48e4017e13986d21795f39de6a19eeb8af6409: Status 404 returned error can't find the container with id 49a0b33bad51870f07867bb13d48e4017e13986d21795f39de6a19eeb8af6409
Sep 11 23:01:17.792184 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:17.792142    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dcaac8a_b529_4ba2_a3f3_a617817741c2.slice/crio-967e120fdae9e4dafb96046e0dc2874a86aeac4bf7bbf608caa591aa53a87f1d WatchSource:0}: Error finding container 967e120fdae9e4dafb96046e0dc2874a86aeac4bf7bbf608caa591aa53a87f1d: Status 404 returned error can't find the container with id 967e120fdae9e4dafb96046e0dc2874a86aeac4bf7bbf608caa591aa53a87f1d
Sep 11 23:01:17.888816 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.888729    2546 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 23:01:17.997864 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:17.997837    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50607ccd-47f9-463a-bdc7-181484a00caa-metrics-certs\") pod \"network-metrics-daemon-vbgsz\" (UID: \"50607ccd-47f9-463a-bdc7-181484a00caa\") " pod="openshift-multus/network-metrics-daemon-vbgsz"
Sep 11 23:01:17.998010 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:17.997944    2546 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 23:01:17.998010 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:17.997990    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50607ccd-47f9-463a-bdc7-181484a00caa-metrics-certs podName:50607ccd-47f9-463a-bdc7-181484a00caa nodeName:}" failed. No retries permitted until 2026-09-11 23:01:18.997978069 +0000 UTC m=+3.031107268 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/50607ccd-47f9-463a-bdc7-181484a00caa-metrics-certs") pod "network-metrics-daemon-vbgsz" (UID: "50607ccd-47f9-463a-bdc7-181484a00caa") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 23:01:18.099015 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:18.098988    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3ad2cd03-8eae-49c4-8c94-ea45de3d3c7a-original-pull-secret\") pod \"global-pull-secret-syncer-5n55v\" (UID: \"3ad2cd03-8eae-49c4-8c94-ea45de3d3c7a\") " pod="kube-system/global-pull-secret-syncer-5n55v"
Sep 11 23:01:18.099168 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:18.099088    2546 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 23:01:18.099168 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:18.099138    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ad2cd03-8eae-49c4-8c94-ea45de3d3c7a-original-pull-secret podName:3ad2cd03-8eae-49c4-8c94-ea45de3d3c7a nodeName:}" failed. No retries permitted until 2026-09-11 23:01:19.099125193 +0000 UTC m=+3.132254388 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3ad2cd03-8eae-49c4-8c94-ea45de3d3c7a-original-pull-secret") pod "global-pull-secret-syncer-5n55v" (UID: "3ad2cd03-8eae-49c4-8c94-ea45de3d3c7a") : object "kube-system"/"original-pull-secret" not registered
Sep 11 23:01:18.181344 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:18.181264    2546 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 23:01:18.200017 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:18.199985    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cn25v\" (UniqueName: \"kubernetes.io/projected/223f5b7c-785a-4cfb-9196-6f4adb77d14a-kube-api-access-cn25v\") pod \"network-check-target-jdr7s\" (UID: \"223f5b7c-785a-4cfb-9196-6f4adb77d14a\") " pod="openshift-network-diagnostics/network-check-target-jdr7s"
Sep 11 23:01:18.200209 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:18.200172    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 23:01:18.200209 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:18.200202    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 23:01:18.200337 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:18.200220    2546 projected.go:196] Error preparing data for projected volume kube-api-access-cn25v for pod openshift-network-diagnostics/network-check-target-jdr7s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 23:01:18.200337 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:18.200278    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/223f5b7c-785a-4cfb-9196-6f4adb77d14a-kube-api-access-cn25v podName:223f5b7c-785a-4cfb-9196-6f4adb77d14a nodeName:}" failed. No retries permitted until 2026-09-11 23:01:19.200260213 +0000 UTC m=+3.233389412 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-cn25v" (UniqueName: "kubernetes.io/projected/223f5b7c-785a-4cfb-9196-6f4adb77d14a-kube-api-access-cn25v") pod "network-check-target-jdr7s" (UID: "223f5b7c-785a-4cfb-9196-6f4adb77d14a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 23:01:18.432579 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:18.432469    2546 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 22:56:17 +0000 UTC" deadline="2028-05-27 14:08:32.153845752 +0000 UTC"
Sep 11 23:01:18.432579 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:18.432532    2546 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14967h7m13.721317755s"
Sep 11 23:01:18.487703 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:18.487672    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5n55v"
Sep 11 23:01:18.487879 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:18.487805    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5n55v" podUID="3ad2cd03-8eae-49c4-8c94-ea45de3d3c7a"
Sep 11 23:01:18.488259 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:18.488236    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vbgsz"
Sep 11 23:01:18.488375 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:18.488347    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vbgsz" podUID="50607ccd-47f9-463a-bdc7-181484a00caa"
Sep 11 23:01:18.510692 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:18.510523    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kv48f" event={"ID":"8916993f-54e0-4346-bf2b-032ac759a2a6","Type":"ContainerStarted","Data":"49a0b33bad51870f07867bb13d48e4017e13986d21795f39de6a19eeb8af6409"}
Sep 11 23:01:18.518032 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:18.517995    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p7244" event={"ID":"6b0a8af3-932c-4091-b7a0-180fe1751a3a","Type":"ContainerStarted","Data":"04ff54982ed9240d067ab1312f8a21e759359955321675e64f06d9ee7bfa3f1c"}
Sep 11 23:01:18.522396 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:18.522340    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zdnn9" event={"ID":"157e8f79-c688-49dd-a216-86c09ffb4992","Type":"ContainerStarted","Data":"901bb8b4fd7db3cd07d44cd202058dd2918bb5e00de43531c1f4dac40398b859"}
Sep 11 23:01:18.529503 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:18.529473    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bv5gf" event={"ID":"0155dec9-f8e2-4007-9861-b8c870223adb","Type":"ContainerStarted","Data":"e1384ee41b26d85ddb957eb710d32fa8fa3d08e0773a07f8696a63a1978b2d8b"}
Sep 11 23:01:18.533059 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:18.533032    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vqs5t" event={"ID":"9dcaac8a-b529-4ba2-a3f3-a617817741c2","Type":"ContainerStarted","Data":"967e120fdae9e4dafb96046e0dc2874a86aeac4bf7bbf608caa591aa53a87f1d"}
Sep 11 23:01:18.536621 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:18.536564    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-lhmpr" event={"ID":"a489f10b-bf16-40dc-aa5f-dc1d8228aa32","Type":"ContainerStarted","Data":"749148ceb0e619824254d76727a717c85207bbed8787ebe208282f9cc75cf05d"}
Sep 11 23:01:18.541010 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:18.540935    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zgtd5" event={"ID":"aa21ad55-513d-46a1-89ce-78e88c3e0efa","Type":"ContainerStarted","Data":"612d2bb5e6646d8cf745d1d2b3fe2f73fcee01c1bbdacfd4965a0f22fdf55ab6"}
Sep 11 23:01:18.552075 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:18.552025    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mnljs" event={"ID":"d2174c19-0971-419a-bc04-0c4662d5ff3c","Type":"ContainerStarted","Data":"a2e3152cabaefa4d3c4b69c87e09f709084a26bb0b00d44ac7a7dc03d3ac3c21"}
Sep 11 23:01:18.556090 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:18.556064    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mrmnp" event={"ID":"c27dbcc1-5027-44f3-9a8a-d1253a431256","Type":"ContainerStarted","Data":"9dad3707f405638e79fc90f7359489d20c6273ae6a0524b6f65ed57e1e2b59ac"}
Sep 11 23:01:19.006706 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:19.006669    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50607ccd-47f9-463a-bdc7-181484a00caa-metrics-certs\") pod \"network-metrics-daemon-vbgsz\" (UID: \"50607ccd-47f9-463a-bdc7-181484a00caa\") " pod="openshift-multus/network-metrics-daemon-vbgsz"
Sep 11 23:01:19.006896 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:19.006851    2546 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 23:01:19.006964 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:19.006918    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50607ccd-47f9-463a-bdc7-181484a00caa-metrics-certs podName:50607ccd-47f9-463a-bdc7-181484a00caa nodeName:}" failed. No retries permitted until 2026-09-11 23:01:21.00689944 +0000 UTC m=+5.040028648 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/50607ccd-47f9-463a-bdc7-181484a00caa-metrics-certs") pod "network-metrics-daemon-vbgsz" (UID: "50607ccd-47f9-463a-bdc7-181484a00caa") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 23:01:19.108521 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:19.108482    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3ad2cd03-8eae-49c4-8c94-ea45de3d3c7a-original-pull-secret\") pod \"global-pull-secret-syncer-5n55v\" (UID: \"3ad2cd03-8eae-49c4-8c94-ea45de3d3c7a\") " pod="kube-system/global-pull-secret-syncer-5n55v"
Sep 11 23:01:19.108684 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:19.108664    2546 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 23:01:19.108753 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:19.108729    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ad2cd03-8eae-49c4-8c94-ea45de3d3c7a-original-pull-secret podName:3ad2cd03-8eae-49c4-8c94-ea45de3d3c7a nodeName:}" failed. No retries permitted until 2026-09-11 23:01:21.108710858 +0000 UTC m=+5.141840065 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3ad2cd03-8eae-49c4-8c94-ea45de3d3c7a-original-pull-secret") pod "global-pull-secret-syncer-5n55v" (UID: "3ad2cd03-8eae-49c4-8c94-ea45de3d3c7a") : object "kube-system"/"original-pull-secret" not registered
Sep 11 23:01:19.198398 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:19.198367    2546 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 23:01:19.209228 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:19.209195    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cn25v\" (UniqueName: \"kubernetes.io/projected/223f5b7c-785a-4cfb-9196-6f4adb77d14a-kube-api-access-cn25v\") pod \"network-check-target-jdr7s\" (UID: \"223f5b7c-785a-4cfb-9196-6f4adb77d14a\") " pod="openshift-network-diagnostics/network-check-target-jdr7s"
Sep 11 23:01:19.209408 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:19.209389    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 23:01:19.209486 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:19.209416    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 23:01:19.209486 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:19.209429    2546 projected.go:196] Error preparing data for projected volume kube-api-access-cn25v for pod openshift-network-diagnostics/network-check-target-jdr7s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 23:01:19.209587 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:19.209487    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/223f5b7c-785a-4cfb-9196-6f4adb77d14a-kube-api-access-cn25v podName:223f5b7c-785a-4cfb-9196-6f4adb77d14a nodeName:}" failed. No retries permitted until 2026-09-11 23:01:21.209467915 +0000 UTC m=+5.242597130 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-cn25v" (UniqueName: "kubernetes.io/projected/223f5b7c-785a-4cfb-9196-6f4adb77d14a-kube-api-access-cn25v") pod "network-check-target-jdr7s" (UID: "223f5b7c-785a-4cfb-9196-6f4adb77d14a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 23:01:19.433508 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:19.433360    2546 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 22:56:17 +0000 UTC" deadline="2028-06-26 16:58:52.847354844 +0000 UTC"
Sep 11 23:01:19.433508 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:19.433425    2546 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15689h57m33.413934008s"
Sep 11 23:01:19.487987 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:19.487937    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jdr7s"
Sep 11 23:01:19.488202 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:19.488092    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jdr7s" podUID="223f5b7c-785a-4cfb-9196-6f4adb77d14a"
Sep 11 23:01:20.490413 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:20.490374    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5n55v"
Sep 11 23:01:20.490880 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:20.490531    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5n55v" podUID="3ad2cd03-8eae-49c4-8c94-ea45de3d3c7a"
Sep 11 23:01:20.490975 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:20.490955    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vbgsz"
Sep 11 23:01:20.491215 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:20.491069    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vbgsz" podUID="50607ccd-47f9-463a-bdc7-181484a00caa"
Sep 11 23:01:21.029243 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:21.029208    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50607ccd-47f9-463a-bdc7-181484a00caa-metrics-certs\") pod \"network-metrics-daemon-vbgsz\" (UID: \"50607ccd-47f9-463a-bdc7-181484a00caa\") " pod="openshift-multus/network-metrics-daemon-vbgsz"
Sep 11 23:01:21.029422 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:21.029372    2546 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 23:01:21.029496 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:21.029438    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50607ccd-47f9-463a-bdc7-181484a00caa-metrics-certs podName:50607ccd-47f9-463a-bdc7-181484a00caa nodeName:}" failed. No retries permitted until 2026-09-11 23:01:25.029418542 +0000 UTC m=+9.062547751 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/50607ccd-47f9-463a-bdc7-181484a00caa-metrics-certs") pod "network-metrics-daemon-vbgsz" (UID: "50607ccd-47f9-463a-bdc7-181484a00caa") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 23:01:21.130778 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:21.130203    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3ad2cd03-8eae-49c4-8c94-ea45de3d3c7a-original-pull-secret\") pod \"global-pull-secret-syncer-5n55v\" (UID: \"3ad2cd03-8eae-49c4-8c94-ea45de3d3c7a\") " pod="kube-system/global-pull-secret-syncer-5n55v"
Sep 11 23:01:21.130778 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:21.130380    2546 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 23:01:21.130778 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:21.130441    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ad2cd03-8eae-49c4-8c94-ea45de3d3c7a-original-pull-secret podName:3ad2cd03-8eae-49c4-8c94-ea45de3d3c7a nodeName:}" failed. No retries permitted until 2026-09-11 23:01:25.130424376 +0000 UTC m=+9.163553574 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3ad2cd03-8eae-49c4-8c94-ea45de3d3c7a-original-pull-secret") pod "global-pull-secret-syncer-5n55v" (UID: "3ad2cd03-8eae-49c4-8c94-ea45de3d3c7a") : object "kube-system"/"original-pull-secret" not registered
Sep 11 23:01:21.230818 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:21.230777    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cn25v\" (UniqueName: \"kubernetes.io/projected/223f5b7c-785a-4cfb-9196-6f4adb77d14a-kube-api-access-cn25v\") pod \"network-check-target-jdr7s\" (UID: \"223f5b7c-785a-4cfb-9196-6f4adb77d14a\") " pod="openshift-network-diagnostics/network-check-target-jdr7s"
Sep 11 23:01:21.231027 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:21.230995    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 23:01:21.231027 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:21.231021    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 23:01:21.231118 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:21.231033    2546 projected.go:196] Error preparing data for projected volume kube-api-access-cn25v for pod openshift-network-diagnostics/network-check-target-jdr7s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 23:01:21.231118 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:21.231096    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/223f5b7c-785a-4cfb-9196-6f4adb77d14a-kube-api-access-cn25v podName:223f5b7c-785a-4cfb-9196-6f4adb77d14a nodeName:}" failed. No retries permitted until 2026-09-11 23:01:25.231078317 +0000 UTC m=+9.264207529 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-cn25v" (UniqueName: "kubernetes.io/projected/223f5b7c-785a-4cfb-9196-6f4adb77d14a-kube-api-access-cn25v") pod "network-check-target-jdr7s" (UID: "223f5b7c-785a-4cfb-9196-6f4adb77d14a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 23:01:21.488658 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:21.488577    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jdr7s"
Sep 11 23:01:21.488826 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:21.488726    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jdr7s" podUID="223f5b7c-785a-4cfb-9196-6f4adb77d14a"
Sep 11 23:01:22.489914 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:22.489881    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vbgsz"
Sep 11 23:01:22.490394 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:22.490008    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vbgsz" podUID="50607ccd-47f9-463a-bdc7-181484a00caa"
Sep 11 23:01:22.490394 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:22.490385    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5n55v"
Sep 11 23:01:22.490517 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:22.490475    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5n55v" podUID="3ad2cd03-8eae-49c4-8c94-ea45de3d3c7a"
Sep 11 23:01:23.488625 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:23.488588    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jdr7s"
Sep 11 23:01:23.488858 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:23.488736    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jdr7s" podUID="223f5b7c-785a-4cfb-9196-6f4adb77d14a"
Sep 11 23:01:24.490243 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:24.490207    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5n55v"
Sep 11 23:01:24.490672 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:24.490336    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5n55v" podUID="3ad2cd03-8eae-49c4-8c94-ea45de3d3c7a"
Sep 11 23:01:24.504493 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:24.504468    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vbgsz"
Sep 11 23:01:24.505866 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:24.505832    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vbgsz" podUID="50607ccd-47f9-463a-bdc7-181484a00caa"
Sep 11 23:01:25.066437 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:25.066347    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50607ccd-47f9-463a-bdc7-181484a00caa-metrics-certs\") pod \"network-metrics-daemon-vbgsz\" (UID: \"50607ccd-47f9-463a-bdc7-181484a00caa\") " pod="openshift-multus/network-metrics-daemon-vbgsz"
Sep 11 23:01:25.066573 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:25.066523    2546 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 23:01:25.066611 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:25.066583    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50607ccd-47f9-463a-bdc7-181484a00caa-metrics-certs podName:50607ccd-47f9-463a-bdc7-181484a00caa nodeName:}" failed. No retries permitted until 2026-09-11 23:01:33.066567412 +0000 UTC m=+17.099696610 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/50607ccd-47f9-463a-bdc7-181484a00caa-metrics-certs") pod "network-metrics-daemon-vbgsz" (UID: "50607ccd-47f9-463a-bdc7-181484a00caa") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 23:01:25.167166 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:25.167127    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3ad2cd03-8eae-49c4-8c94-ea45de3d3c7a-original-pull-secret\") pod \"global-pull-secret-syncer-5n55v\" (UID: \"3ad2cd03-8eae-49c4-8c94-ea45de3d3c7a\") " pod="kube-system/global-pull-secret-syncer-5n55v"
Sep 11 23:01:25.167322 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:25.167285    2546 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 23:01:25.167377 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:25.167343    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ad2cd03-8eae-49c4-8c94-ea45de3d3c7a-original-pull-secret podName:3ad2cd03-8eae-49c4-8c94-ea45de3d3c7a nodeName:}" failed. No retries permitted until 2026-09-11 23:01:33.167324906 +0000 UTC m=+17.200454118 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3ad2cd03-8eae-49c4-8c94-ea45de3d3c7a-original-pull-secret") pod "global-pull-secret-syncer-5n55v" (UID: "3ad2cd03-8eae-49c4-8c94-ea45de3d3c7a") : object "kube-system"/"original-pull-secret" not registered
Sep 11 23:01:25.267577 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:25.267534    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cn25v\" (UniqueName: \"kubernetes.io/projected/223f5b7c-785a-4cfb-9196-6f4adb77d14a-kube-api-access-cn25v\") pod \"network-check-target-jdr7s\" (UID: \"223f5b7c-785a-4cfb-9196-6f4adb77d14a\") " pod="openshift-network-diagnostics/network-check-target-jdr7s"
Sep 11 23:01:25.267748 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:25.267729    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 23:01:25.267799 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:25.267760    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 23:01:25.267799 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:25.267774    2546 projected.go:196] Error preparing data for projected volume kube-api-access-cn25v for pod openshift-network-diagnostics/network-check-target-jdr7s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 23:01:25.267862 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:25.267841    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/223f5b7c-785a-4cfb-9196-6f4adb77d14a-kube-api-access-cn25v podName:223f5b7c-785a-4cfb-9196-6f4adb77d14a nodeName:}" failed. No retries permitted until 2026-09-11 23:01:33.267820678 +0000 UTC m=+17.300949893 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-cn25v" (UniqueName: "kubernetes.io/projected/223f5b7c-785a-4cfb-9196-6f4adb77d14a-kube-api-access-cn25v") pod "network-check-target-jdr7s" (UID: "223f5b7c-785a-4cfb-9196-6f4adb77d14a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 23:01:25.488040 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:25.487950    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jdr7s"
Sep 11 23:01:25.488236 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:25.488203    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jdr7s" podUID="223f5b7c-785a-4cfb-9196-6f4adb77d14a"
Sep 11 23:01:26.488782 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:26.488743    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5n55v"
Sep 11 23:01:26.489376 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:26.488853    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5n55v" podUID="3ad2cd03-8eae-49c4-8c94-ea45de3d3c7a"
Sep 11 23:01:26.489376 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:26.489248    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vbgsz"
Sep 11 23:01:26.489459 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:26.489379    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vbgsz" podUID="50607ccd-47f9-463a-bdc7-181484a00caa"
Sep 11 23:01:27.488241 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:27.488210    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jdr7s"
Sep 11 23:01:27.488419 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:27.488336    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jdr7s" podUID="223f5b7c-785a-4cfb-9196-6f4adb77d14a"
Sep 11 23:01:28.489139 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:28.488732    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5n55v"
Sep 11 23:01:28.489139 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:28.488875    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5n55v" podUID="3ad2cd03-8eae-49c4-8c94-ea45de3d3c7a"
Sep 11 23:01:28.489139 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:28.488929    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vbgsz"
Sep 11 23:01:28.489139 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:28.489089    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vbgsz" podUID="50607ccd-47f9-463a-bdc7-181484a00caa"
Sep 11 23:01:29.488118 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:29.488078    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jdr7s"
Sep 11 23:01:29.488291 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:29.488200    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jdr7s" podUID="223f5b7c-785a-4cfb-9196-6f4adb77d14a"
Sep 11 23:01:30.488075 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:30.487988    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5n55v"
Sep 11 23:01:30.488075 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:30.488040    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vbgsz"
Sep 11 23:01:30.488552 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:30.488134    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5n55v" podUID="3ad2cd03-8eae-49c4-8c94-ea45de3d3c7a"
Sep 11 23:01:30.488552 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:30.488268    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vbgsz" podUID="50607ccd-47f9-463a-bdc7-181484a00caa"
Sep 11 23:01:31.488484 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:31.488451    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jdr7s"
Sep 11 23:01:31.488884 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:31.488595    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jdr7s" podUID="223f5b7c-785a-4cfb-9196-6f4adb77d14a"
Sep 11 23:01:32.488620 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:32.488582    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5n55v"
Sep 11 23:01:32.489079 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:32.488715    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5n55v" podUID="3ad2cd03-8eae-49c4-8c94-ea45de3d3c7a"
Sep 11 23:01:32.489079 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:32.488774    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vbgsz"
Sep 11 23:01:32.489079 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:32.488919    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vbgsz" podUID="50607ccd-47f9-463a-bdc7-181484a00caa"
Sep 11 23:01:33.126429 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:33.126398    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50607ccd-47f9-463a-bdc7-181484a00caa-metrics-certs\") pod \"network-metrics-daemon-vbgsz\" (UID: \"50607ccd-47f9-463a-bdc7-181484a00caa\") " pod="openshift-multus/network-metrics-daemon-vbgsz"
Sep 11 23:01:33.126655 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:33.126574    2546 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 23:01:33.126710 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:33.126656    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50607ccd-47f9-463a-bdc7-181484a00caa-metrics-certs podName:50607ccd-47f9-463a-bdc7-181484a00caa nodeName:}" failed. No retries permitted until 2026-09-11 23:01:49.126636704 +0000 UTC m=+33.159765904 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/50607ccd-47f9-463a-bdc7-181484a00caa-metrics-certs") pod "network-metrics-daemon-vbgsz" (UID: "50607ccd-47f9-463a-bdc7-181484a00caa") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 23:01:33.226958 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:33.226924    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3ad2cd03-8eae-49c4-8c94-ea45de3d3c7a-original-pull-secret\") pod \"global-pull-secret-syncer-5n55v\" (UID: \"3ad2cd03-8eae-49c4-8c94-ea45de3d3c7a\") " pod="kube-system/global-pull-secret-syncer-5n55v"
Sep 11 23:01:33.227131 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:33.227074    2546 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 23:01:33.227209 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:33.227143    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ad2cd03-8eae-49c4-8c94-ea45de3d3c7a-original-pull-secret podName:3ad2cd03-8eae-49c4-8c94-ea45de3d3c7a nodeName:}" failed. No retries permitted until 2026-09-11 23:01:49.227124414 +0000 UTC m=+33.260253611 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3ad2cd03-8eae-49c4-8c94-ea45de3d3c7a-original-pull-secret") pod "global-pull-secret-syncer-5n55v" (UID: "3ad2cd03-8eae-49c4-8c94-ea45de3d3c7a") : object "kube-system"/"original-pull-secret" not registered
Sep 11 23:01:33.327708 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:33.327676    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cn25v\" (UniqueName: \"kubernetes.io/projected/223f5b7c-785a-4cfb-9196-6f4adb77d14a-kube-api-access-cn25v\") pod \"network-check-target-jdr7s\" (UID: \"223f5b7c-785a-4cfb-9196-6f4adb77d14a\") " pod="openshift-network-diagnostics/network-check-target-jdr7s"
Sep 11 23:01:33.327893 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:33.327872    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 23:01:33.327982 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:33.327903    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 23:01:33.327982 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:33.327917    2546 projected.go:196] Error preparing data for projected volume kube-api-access-cn25v for pod openshift-network-diagnostics/network-check-target-jdr7s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 23:01:33.328090 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:33.327989    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/223f5b7c-785a-4cfb-9196-6f4adb77d14a-kube-api-access-cn25v podName:223f5b7c-785a-4cfb-9196-6f4adb77d14a nodeName:}" failed. No retries permitted until 2026-09-11 23:01:49.327969039 +0000 UTC m=+33.361098257 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-cn25v" (UniqueName: "kubernetes.io/projected/223f5b7c-785a-4cfb-9196-6f4adb77d14a-kube-api-access-cn25v") pod "network-check-target-jdr7s" (UID: "223f5b7c-785a-4cfb-9196-6f4adb77d14a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 23:01:33.488097 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:33.488018    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jdr7s"
Sep 11 23:01:33.488259 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:33.488128    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jdr7s" podUID="223f5b7c-785a-4cfb-9196-6f4adb77d14a"
Sep 11 23:01:34.488741 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:34.488704    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5n55v"
Sep 11 23:01:34.489243 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:34.488746    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vbgsz"
Sep 11 23:01:34.489243 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:34.488838    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5n55v" podUID="3ad2cd03-8eae-49c4-8c94-ea45de3d3c7a"
Sep 11 23:01:34.489243 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:34.488980    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vbgsz" podUID="50607ccd-47f9-463a-bdc7-181484a00caa"
Sep 11 23:01:35.488271 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:35.488234    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jdr7s"
Sep 11 23:01:35.488456 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:35.488354    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jdr7s" podUID="223f5b7c-785a-4cfb-9196-6f4adb77d14a"
Sep 11 23:01:36.488210 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:36.488176    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vbgsz"
Sep 11 23:01:36.488796 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:36.488299    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vbgsz" podUID="50607ccd-47f9-463a-bdc7-181484a00caa"
Sep 11 23:01:36.488796 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:36.488336    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5n55v"
Sep 11 23:01:36.488796 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:36.488400    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5n55v" podUID="3ad2cd03-8eae-49c4-8c94-ea45de3d3c7a"
Sep 11 23:01:37.488579 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:37.488549    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jdr7s"
Sep 11 23:01:37.488905 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:37.488657    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jdr7s" podUID="223f5b7c-785a-4cfb-9196-6f4adb77d14a"
Sep 11 23:01:38.490233 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:38.488563    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5n55v"
Sep 11 23:01:38.490233 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:38.488894    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5n55v" podUID="3ad2cd03-8eae-49c4-8c94-ea45de3d3c7a"
Sep 11 23:01:38.490233 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:38.488972    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vbgsz"
Sep 11 23:01:38.490233 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:38.489058    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vbgsz" podUID="50607ccd-47f9-463a-bdc7-181484a00caa"
Sep 11 23:01:38.612966 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:38.612930    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mrmnp" event={"ID":"c27dbcc1-5027-44f3-9a8a-d1253a431256","Type":"ContainerStarted","Data":"f11392d710a331bf2b0ae7dc62d4ef83b6a7a58614eacc9858309d7225e59ee9"}
Sep 11 23:01:38.615106 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:38.615076    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-161.ec2.internal" event={"ID":"fbf860b5b0097ab5f709024d70f9c3d5","Type":"ContainerStarted","Data":"9addbd7e9acb6990e32cc0e0c15038c8eef8817c95b42ad72a078d943a22ce46"}
Sep 11 23:01:38.620035 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:38.619673    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7244_6b0a8af3-932c-4091-b7a0-180fe1751a3a/ovn-acl-logging/0.log"
Sep 11 23:01:38.621618 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:38.621277    2546 generic.go:356] "Generic (PLEG): container finished" podID="6b0a8af3-932c-4091-b7a0-180fe1751a3a" containerID="8a9eb4d6227247516147aa10b1cd11a8adc5a1173f0e1a0acd55c86430c1a2ce" exitCode=1
Sep 11 23:01:38.621618 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:38.621358    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p7244" event={"ID":"6b0a8af3-932c-4091-b7a0-180fe1751a3a","Type":"ContainerStarted","Data":"2173c949db7bef4c823a5cba0afb1e10deebc933079b70e796bd1e8935e0956b"}
Sep 11 23:01:38.621618 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:38.621425    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p7244" event={"ID":"6b0a8af3-932c-4091-b7a0-180fe1751a3a","Type":"ContainerStarted","Data":"07a738c8bdba322eec9ac4057f63b4defffdcefa0a524b391a8d917991bc2876"}
Sep 11 23:01:38.621618 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:38.621470    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p7244" event={"ID":"6b0a8af3-932c-4091-b7a0-180fe1751a3a","Type":"ContainerStarted","Data":"46b285a73cd4530df56090c5ba9d0e0cef3cb821c772722c3588d5a9f7ba57f7"}
Sep 11 23:01:38.621618 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:38.621512    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p7244" event={"ID":"6b0a8af3-932c-4091-b7a0-180fe1751a3a","Type":"ContainerDied","Data":"8a9eb4d6227247516147aa10b1cd11a8adc5a1173f0e1a0acd55c86430c1a2ce"}
Sep 11 23:01:38.621618 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:38.621555    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p7244" event={"ID":"6b0a8af3-932c-4091-b7a0-180fe1751a3a","Type":"ContainerStarted","Data":"209567e19e521435682e77afcb0b35ce3ea2cd63a3beb202755578ebba331b5b"}
Sep 11 23:01:38.623599 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:38.623578    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-lhmpr" event={"ID":"a489f10b-bf16-40dc-aa5f-dc1d8228aa32","Type":"ContainerStarted","Data":"738fc70cc104d282ce4bc71225c7efd64f7c4a713e6259bd414e77dcf3c45086"}
Sep 11 23:01:38.633549 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:38.633489    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-mrmnp" podStartSLOduration=2.4116247570000002 podStartE2EDuration="22.633471187s" totalImagesPullingTime="20.22184643s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 23:01:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:01:38.633328315 +0000 UTC m=+22.666457531" watchObservedRunningTime="2026-09-11 23:01:38.633471187 +0000 UTC m=+22.666600400"
Sep 11 23:01:38.653195 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:38.650769    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-161.ec2.internal" podStartSLOduration=21.650755089 podStartE2EDuration="21.650755089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 23:01:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:01:38.650045663 +0000 UTC m=+22.683174880" watchObservedRunningTime="2026-09-11 23:01:38.650755089 +0000 UTC m=+22.683884306"
Sep 11 23:01:38.686946 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:38.686760    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-lhmpr" podStartSLOduration=2.03808236 podStartE2EDuration="22.686737714s" totalImagesPullingTime="20.648655354s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 23:01:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:01:38.685584562 +0000 UTC m=+22.718713781" watchObservedRunningTime="2026-09-11 23:01:38.686737714 +0000 UTC m=+22.719866932"
Sep 11 23:01:39.487844 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:39.487806    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jdr7s"
Sep 11 23:01:39.488014 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:39.487936    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jdr7s" podUID="223f5b7c-785a-4cfb-9196-6f4adb77d14a"
Sep 11 23:01:39.628342 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:39.628261    2546 generic.go:356] "Generic (PLEG): container finished" podID="e80b9e5ab597f3b17c8733c311f97f1f" containerID="539edc15cf469163d29ffa31d03a1841f1fda90be89bcf9701b4e078055b7e1f" exitCode=0
Sep 11 23:01:39.629025 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:39.628347    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-161.ec2.internal" event={"ID":"e80b9e5ab597f3b17c8733c311f97f1f","Type":"ContainerDied","Data":"539edc15cf469163d29ffa31d03a1841f1fda90be89bcf9701b4e078055b7e1f"}
Sep 11 23:01:39.630015 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:39.629988    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vqs5t" event={"ID":"9dcaac8a-b529-4ba2-a3f3-a617817741c2","Type":"ContainerStarted","Data":"6fb7f5719b9cb701d8a0568df0f6e6ac7845e34120043e43dbfee251a0f2d5eb"}
Sep 11 23:01:39.631671 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:39.631641    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zgtd5" event={"ID":"aa21ad55-513d-46a1-89ce-78e88c3e0efa","Type":"ContainerStarted","Data":"4bc5ba600bb679ea66038f255169b1812088c29115f9e840838fc2e8a9436ebc"}
Sep 11 23:01:39.634829 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:39.634801    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mnljs" event={"ID":"d2174c19-0971-419a-bc04-0c4662d5ff3c","Type":"ContainerStarted","Data":"ca6be5211bbb9d1ce4c3abc75303d7fdf3676613c8bbad59975a2ea6cb534398"}
Sep 11 23:01:39.636965 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:39.636939    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kv48f" event={"ID":"8916993f-54e0-4346-bf2b-032ac759a2a6","Type":"ContainerStarted","Data":"2a683771c494d7d5c26709e30620f230d139fa0052487fc68d46a7df31d87a1a"}
Sep 11 23:01:39.639878 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:39.639862    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7244_6b0a8af3-932c-4091-b7a0-180fe1751a3a/ovn-acl-logging/0.log"
Sep 11 23:01:39.640302 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:39.640275    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p7244" event={"ID":"6b0a8af3-932c-4091-b7a0-180fe1751a3a","Type":"ContainerStarted","Data":"22bdb31b3cfcfcdbdd477d9c062b4238bd8b059bde451d851bd35bdcd8a18c02"}
Sep 11 23:01:39.641866 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:39.641846    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zdnn9" event={"ID":"157e8f79-c688-49dd-a216-86c09ffb4992","Type":"ContainerStarted","Data":"b2c1906d9734c7ff6bbefa59ff32c2cc3ad23c7d3db99f53d84380e40071b7e9"}
Sep 11 23:01:39.643468 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:39.643445    2546 generic.go:356] "Generic (PLEG): container finished" podID="0155dec9-f8e2-4007-9861-b8c870223adb" containerID="7c1c1a7d8a4d6d7ef3fa974a1098bb6958bbe9a1b825df94a83d88401ca3011b" exitCode=0
Sep 11 23:01:39.643561 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:39.643476    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bv5gf" event={"ID":"0155dec9-f8e2-4007-9861-b8c870223adb","Type":"ContainerDied","Data":"7c1c1a7d8a4d6d7ef3fa974a1098bb6958bbe9a1b825df94a83d88401ca3011b"}
Sep 11 23:01:39.661310 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:39.661258    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-mnljs" podStartSLOduration=7.89599292 podStartE2EDuration="23.661242803s" totalImagesPullingTime="15.765249883s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 23:01:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:01:39.66089714 +0000 UTC m=+23.694026353" watchObservedRunningTime="2026-09-11 23:01:39.661242803 +0000 UTC m=+23.694372020"
Sep 11 23:01:39.693411 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:39.693363    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-vqs5t" podStartSLOduration=4.064094428 podStartE2EDuration="23.693348466s" totalImagesPullingTime="19.629254038s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 23:01:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:01:39.677388908 +0000 UTC m=+23.710518150" watchObservedRunningTime="2026-09-11 23:01:39.693348466 +0000 UTC m=+23.726477682"
Sep 11 23:01:39.693773 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:39.693751    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-zgtd5" podStartSLOduration=7.936900202 podStartE2EDuration="23.693742076s" totalImagesPullingTime="15.756841874s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 23:01:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:01:39.693184352 +0000 UTC m=+23.726313569" watchObservedRunningTime="2026-09-11 23:01:39.693742076 +0000 UTC m=+23.726871292"
Sep 11 23:01:39.708082 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:39.708038    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-zdnn9" podStartSLOduration=3.5348505279999998 podStartE2EDuration="23.708019187s" totalImagesPullingTime="20.173168659s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 23:01:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:01:39.707557435 +0000 UTC m=+23.740686651" watchObservedRunningTime="2026-09-11 23:01:39.708019187 +0000 UTC m=+23.741148405"
Sep 11 23:01:39.843829 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:39.843805    2546 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 11 23:01:39.987951 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:39.987872    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-mnljs"
Sep 11 23:01:39.988548 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:39.988531    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-mnljs"
Sep 11 23:01:40.457007 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:40.456794    2546 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-11T23:01:39.843824926Z","UUID":"61653c13-ce4b-450e-895f-0812c9375c11","Handler":null,"Name":"","Endpoint":""}
Sep 11 23:01:40.459171 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:40.459137    2546 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 11 23:01:40.459304 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:40.459206    2546 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 11 23:01:40.487909 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:40.487881    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vbgsz"
Sep 11 23:01:40.488050 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:40.488010    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vbgsz" podUID="50607ccd-47f9-463a-bdc7-181484a00caa"
Sep 11 23:01:40.488286 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:40.488250    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5n55v"
Sep 11 23:01:40.488384 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:40.488366    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5n55v" podUID="3ad2cd03-8eae-49c4-8c94-ea45de3d3c7a"
Sep 11 23:01:40.646789 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:40.646746    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kv48f" event={"ID":"8916993f-54e0-4346-bf2b-032ac759a2a6","Type":"ContainerStarted","Data":"4b7fba0231b2a74ee92bca6bc700b7952d6fddae12662d465913140f05db9822"}
Sep 11 23:01:40.648427 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:40.648399    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-161.ec2.internal" event={"ID":"e80b9e5ab597f3b17c8733c311f97f1f","Type":"ContainerStarted","Data":"18937f8754a5feb16291227af9d1ad6a693b30bcdae75b6a74a5ca049c5327a7"}
Sep 11 23:01:40.648792 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:40.648774    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-mnljs"
Sep 11 23:01:40.649224 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:40.649201    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-mnljs"
Sep 11 23:01:40.680427 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:40.680382    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-161.ec2.internal" podStartSLOduration=23.680362741 podStartE2EDuration="23.680362741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 23:01:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:01:40.665564483 +0000 UTC m=+24.698693700" watchObservedRunningTime="2026-09-11 23:01:40.680362741 +0000 UTC m=+24.713491959"
Sep 11 23:01:41.488043 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:41.487860    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jdr7s"
Sep 11 23:01:41.488236 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:41.488123    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jdr7s" podUID="223f5b7c-785a-4cfb-9196-6f4adb77d14a"
Sep 11 23:01:41.655574 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:41.655547    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7244_6b0a8af3-932c-4091-b7a0-180fe1751a3a/ovn-acl-logging/0.log"
Sep 11 23:01:41.655985 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:41.655914    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p7244" event={"ID":"6b0a8af3-932c-4091-b7a0-180fe1751a3a","Type":"ContainerStarted","Data":"ac8b795389a1fce3e4c1a2e322004c9a8ea4c1b43db6b7e5240d18ba9eb0ba58"}
Sep 11 23:01:42.488485 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:42.488452    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5n55v"
Sep 11 23:01:42.488667 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:42.488600    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5n55v" podUID="3ad2cd03-8eae-49c4-8c94-ea45de3d3c7a"
Sep 11 23:01:42.488667 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:42.488648    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vbgsz"
Sep 11 23:01:42.488793 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:42.488775    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vbgsz" podUID="50607ccd-47f9-463a-bdc7-181484a00caa"
Sep 11 23:01:43.488071 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:43.488035    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jdr7s"
Sep 11 23:01:43.488523 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:43.488170    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jdr7s" podUID="223f5b7c-785a-4cfb-9196-6f4adb77d14a"
Sep 11 23:01:44.491949 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:44.491906    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vbgsz"
Sep 11 23:01:44.492393 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:44.492040    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vbgsz" podUID="50607ccd-47f9-463a-bdc7-181484a00caa"
Sep 11 23:01:44.492393 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:44.492070    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5n55v"
Sep 11 23:01:44.492393 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:44.492130    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5n55v" podUID="3ad2cd03-8eae-49c4-8c94-ea45de3d3c7a"
Sep 11 23:01:45.488009 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:45.487838    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jdr7s"
Sep 11 23:01:45.488201 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:45.488086    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jdr7s" podUID="223f5b7c-785a-4cfb-9196-6f4adb77d14a"
Sep 11 23:01:45.668803 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:45.668766    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kv48f" event={"ID":"8916993f-54e0-4346-bf2b-032ac759a2a6","Type":"ContainerStarted","Data":"7e42e53b4ff5c2fab05a5a66ea01c76ab85ffa91402abaddcb7c8b90ce33fd75"}
Sep 11 23:01:45.672200 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:45.672181    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7244_6b0a8af3-932c-4091-b7a0-180fe1751a3a/ovn-acl-logging/0.log"
Sep 11 23:01:45.672519 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:45.672499    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p7244" event={"ID":"6b0a8af3-932c-4091-b7a0-180fe1751a3a","Type":"ContainerStarted","Data":"469cf7d7c04df2aeb5295fd05ff4cc130536ad44b6ffd6bfddc3f289f26b1226"}
Sep 11 23:01:45.672768 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:45.672748    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-p7244"
Sep 11 23:01:45.672845 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:45.672825    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-p7244"
Sep 11 23:01:45.672919 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:45.672906    2546 scope.go:117] "RemoveContainer" containerID="8a9eb4d6227247516147aa10b1cd11a8adc5a1173f0e1a0acd55c86430c1a2ce"
Sep 11 23:01:45.673004 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:45.672986    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-p7244"
Sep 11 23:01:45.674941 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:45.674916    2546 generic.go:356] "Generic (PLEG): container finished" podID="0155dec9-f8e2-4007-9861-b8c870223adb" containerID="4d87ff02cb74ebc434016e48d54eca1b9ba208381b005634a8b3514e34ed647d" exitCode=0
Sep 11 23:01:45.675025 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:45.674966    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bv5gf" event={"ID":"0155dec9-f8e2-4007-9861-b8c870223adb","Type":"ContainerDied","Data":"4d87ff02cb74ebc434016e48d54eca1b9ba208381b005634a8b3514e34ed647d"}
Sep 11 23:01:45.690066 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:45.690042    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-p7244"
Sep 11 23:01:45.690299 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:45.690194    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-p7244"
Sep 11 23:01:45.691861 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:45.691813    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kv48f" podStartSLOduration=3.7987479669999997 podStartE2EDuration="29.691796713s" totalImagesPullingTime="25.893048746s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 23:01:16 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:01:45.690709871 +0000 UTC m=+29.723839087" watchObservedRunningTime="2026-09-11 23:01:45.691796713 +0000 UTC m=+29.724925931"
Sep 11 23:01:46.489452 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:46.489290    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vbgsz"
Sep 11 23:01:46.489581 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:46.489354    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5n55v"
Sep 11 23:01:46.489581 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:46.489507    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vbgsz" podUID="50607ccd-47f9-463a-bdc7-181484a00caa"
Sep 11 23:01:46.489698 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:46.489602    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5n55v" podUID="3ad2cd03-8eae-49c4-8c94-ea45de3d3c7a"
Sep 11 23:01:46.536814 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:46.536290    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5n55v"]
Sep 11 23:01:46.537281 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:46.537053    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vbgsz"]
Sep 11 23:01:46.537705 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:46.537673    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jdr7s"]
Sep 11 23:01:46.537801 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:46.537786    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jdr7s"
Sep 11 23:01:46.537919 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:46.537897    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jdr7s" podUID="223f5b7c-785a-4cfb-9196-6f4adb77d14a"
Sep 11 23:01:46.681126 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:46.681096    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7244_6b0a8af3-932c-4091-b7a0-180fe1751a3a/ovn-acl-logging/0.log"
Sep 11 23:01:46.681898 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:46.681533    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p7244" event={"ID":"6b0a8af3-932c-4091-b7a0-180fe1751a3a","Type":"ContainerStarted","Data":"1e87534869f0d917584125a906fce838488bfcf0be8603bbc4fa559daa16f968"}
Sep 11 23:01:46.683761 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:46.683737    2546 generic.go:356] "Generic (PLEG): container finished" podID="0155dec9-f8e2-4007-9861-b8c870223adb" containerID="07e7c286bf49e8d5c19d9926de97f10b0321ce29d13360524e95ee7fb563add6" exitCode=0
Sep 11 23:01:46.683865 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:46.683813    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bv5gf" event={"ID":"0155dec9-f8e2-4007-9861-b8c870223adb","Type":"ContainerDied","Data":"07e7c286bf49e8d5c19d9926de97f10b0321ce29d13360524e95ee7fb563add6"}
Sep 11 23:01:46.683865 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:46.683838    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vbgsz"
Sep 11 23:01:46.683865 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:46.683864    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5n55v"
Sep 11 23:01:46.683978 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:46.683944    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5n55v" podUID="3ad2cd03-8eae-49c4-8c94-ea45de3d3c7a"
Sep 11 23:01:46.684190 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:46.684165    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vbgsz" podUID="50607ccd-47f9-463a-bdc7-181484a00caa"
Sep 11 23:01:46.739132 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:46.739080    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-p7244" podStartSLOduration=10.566032908 podStartE2EDuration="30.7390646s" totalImagesPullingTime="20.173031692s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 23:01:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:01:46.715492072 +0000 UTC m=+30.748621291" watchObservedRunningTime="2026-09-11 23:01:46.7390646 +0000 UTC m=+30.772193807"
Sep 11 23:01:47.688831 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:47.688798    2546 generic.go:356] "Generic (PLEG): container finished" podID="0155dec9-f8e2-4007-9861-b8c870223adb" containerID="d6f993f9b8dbde264353935ea6935c2b773ae6befb025575e030a704c79cbe0b" exitCode=0
Sep 11 23:01:47.689201 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:47.688888    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bv5gf" event={"ID":"0155dec9-f8e2-4007-9861-b8c870223adb","Type":"ContainerDied","Data":"d6f993f9b8dbde264353935ea6935c2b773ae6befb025575e030a704c79cbe0b"}
Sep 11 23:01:48.491747 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:48.491717    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5n55v"
Sep 11 23:01:48.491747 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:48.491744    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vbgsz"
Sep 11 23:01:48.491990 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:48.491845    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5n55v" podUID="3ad2cd03-8eae-49c4-8c94-ea45de3d3c7a"
Sep 11 23:01:48.491990 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:48.491910    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jdr7s"
Sep 11 23:01:48.492103 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:48.492045    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vbgsz" podUID="50607ccd-47f9-463a-bdc7-181484a00caa"
Sep 11 23:01:48.492183 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:48.492142    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jdr7s" podUID="223f5b7c-785a-4cfb-9196-6f4adb77d14a"
Sep 11 23:01:49.137976 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:49.137941    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50607ccd-47f9-463a-bdc7-181484a00caa-metrics-certs\") pod \"network-metrics-daemon-vbgsz\" (UID: \"50607ccd-47f9-463a-bdc7-181484a00caa\") " pod="openshift-multus/network-metrics-daemon-vbgsz"
Sep 11 23:01:49.138592 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:49.138124    2546 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 23:01:49.138592 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:49.138219    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50607ccd-47f9-463a-bdc7-181484a00caa-metrics-certs podName:50607ccd-47f9-463a-bdc7-181484a00caa nodeName:}" failed. No retries permitted until 2026-09-11 23:02:21.138197899 +0000 UTC m=+65.171327094 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/50607ccd-47f9-463a-bdc7-181484a00caa-metrics-certs") pod "network-metrics-daemon-vbgsz" (UID: "50607ccd-47f9-463a-bdc7-181484a00caa") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 23:01:49.238673 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:49.238644    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3ad2cd03-8eae-49c4-8c94-ea45de3d3c7a-original-pull-secret\") pod \"global-pull-secret-syncer-5n55v\" (UID: \"3ad2cd03-8eae-49c4-8c94-ea45de3d3c7a\") " pod="kube-system/global-pull-secret-syncer-5n55v"
Sep 11 23:01:49.238836 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:49.238792    2546 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 23:01:49.238896 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:49.238859    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ad2cd03-8eae-49c4-8c94-ea45de3d3c7a-original-pull-secret podName:3ad2cd03-8eae-49c4-8c94-ea45de3d3c7a nodeName:}" failed. No retries permitted until 2026-09-11 23:02:21.238839734 +0000 UTC m=+65.271968942 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3ad2cd03-8eae-49c4-8c94-ea45de3d3c7a-original-pull-secret") pod "global-pull-secret-syncer-5n55v" (UID: "3ad2cd03-8eae-49c4-8c94-ea45de3d3c7a") : object "kube-system"/"original-pull-secret" not registered
Sep 11 23:01:49.340085 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:49.340040    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cn25v\" (UniqueName: \"kubernetes.io/projected/223f5b7c-785a-4cfb-9196-6f4adb77d14a-kube-api-access-cn25v\") pod \"network-check-target-jdr7s\" (UID: \"223f5b7c-785a-4cfb-9196-6f4adb77d14a\") " pod="openshift-network-diagnostics/network-check-target-jdr7s"
Sep 11 23:01:49.340289 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:49.340265    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 23:01:49.340289 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:49.340288    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 23:01:49.340398 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:49.340303    2546 projected.go:196] Error preparing data for projected volume kube-api-access-cn25v for pod openshift-network-diagnostics/network-check-target-jdr7s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 23:01:49.340446 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:49.340421    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/223f5b7c-785a-4cfb-9196-6f4adb77d14a-kube-api-access-cn25v podName:223f5b7c-785a-4cfb-9196-6f4adb77d14a nodeName:}" failed. No retries permitted until 2026-09-11 23:02:21.34040062 +0000 UTC m=+65.373529832 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-cn25v" (UniqueName: "kubernetes.io/projected/223f5b7c-785a-4cfb-9196-6f4adb77d14a-kube-api-access-cn25v") pod "network-check-target-jdr7s" (UID: "223f5b7c-785a-4cfb-9196-6f4adb77d14a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 23:01:50.488176 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:50.488124    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jdr7s"
Sep 11 23:01:50.488628 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:50.488124    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5n55v"
Sep 11 23:01:50.488628 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:50.488280    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jdr7s" podUID="223f5b7c-785a-4cfb-9196-6f4adb77d14a"
Sep 11 23:01:50.488628 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:50.488134    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vbgsz"
Sep 11 23:01:50.488628 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:50.488331    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5n55v" podUID="3ad2cd03-8eae-49c4-8c94-ea45de3d3c7a"
Sep 11 23:01:50.488628 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:50.488414    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vbgsz" podUID="50607ccd-47f9-463a-bdc7-181484a00caa"
Sep 11 23:01:51.217935 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:51.217904    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-161.ec2.internal" event="NodeReady"
Sep 11 23:01:51.218110 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:51.218077    2546 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 11 23:01:51.253042 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:51.253010    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-7q8hx"]
Sep 11 23:01:51.277236 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:51.277210    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5c8f46447c-cdpct"]
Sep 11 23:01:51.277399 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:51.277341    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-7q8hx"
Sep 11 23:01:51.280857 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:51.280832    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 11 23:01:51.281435 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:51.281415    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 11 23:01:51.281537 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:51.281468    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-t54dv\""
Sep 11 23:01:51.292418 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:51.292094    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-7q8hx"]
Sep 11 23:01:51.292418 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:51.292138    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-glt22"]
Sep 11 23:01:51.292418 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:51.292233    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c8f46447c-cdpct"
Sep 11 23:01:51.295552 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:51.295529    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 11 23:01:51.295685 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:51.295659    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 11 23:01:51.295766 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:51.295689    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-jqw7w\""
Sep 11 23:01:51.295766 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:51.295696    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 11 23:01:51.304046 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:51.304018    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 11 23:01:51.308249 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:51.308225    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-fzfcr"]
Sep 11 23:01:51.308380 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:51.308349    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-glt22"
Sep 11 23:01:51.311395 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:51.311366    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 11 23:01:51.311641 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:51.311620    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-z767f\""
Sep 11 23:01:51.311741 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:51.311641    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 11 23:01:51.311905 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:51.311887    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 11 23:01:51.320714 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:51.320691    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5c8f46447c-cdpct"]
Sep 11 23:01:51.320816 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:51.320726    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-glt22"]
Sep 11 23:01:51.320816 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:51.320747    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fzfcr"]
Sep 11 23:01:51.320930 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:51.320817    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fzfcr"
Sep 11 23:01:51.323899 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:51.323881    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 11 23:01:51.324695 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:51.324546    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-x84tr\""
Sep 11 23:01:51.324695 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:51.324625    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 11 23:01:51.355944 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:51.355905    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-987rm\" (UniqueName: \"kubernetes.io/projected/8ea4ca4e-b321-4324-b380-dab72513b02a-kube-api-access-987rm\") pod \"image-registry-5c8f46447c-cdpct\" (UID: \"8ea4ca4e-b321-4324-b380-dab72513b02a\") " pod="openshift-image-registry/image-registry-5c8f46447c-cdpct"
Sep 11 23:01:51.356111 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:51.355953    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8ea4ca4e-b321-4324-b380-dab72513b02a-trusted-ca\") pod \"image-registry-5c8f46447c-cdpct\" (UID: \"8ea4ca4e-b321-4324-b380-dab72513b02a\") " pod="openshift-image-registry/image-registry-5c8f46447c-cdpct"
Sep 11 23:01:51.356111 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:51.355988    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8ea4ca4e-b321-4324-b380-dab72513b02a-image-registry-private-configuration\") pod \"image-registry-5c8f46447c-cdpct\" (UID: \"8ea4ca4e-b321-4324-b380-dab72513b02a\") " pod="openshift-image-registry/image-registry-5c8f46447c-cdpct"
Sep 11 23:01:51.356111 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:51.356017    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8ea4ca4e-b321-4324-b380-dab72513b02a-installation-pull-secrets\") pod \"image-registry-5c8f46447c-cdpct\" (UID: \"8ea4ca4e-b321-4324-b380-dab72513b02a\") " pod="openshift-image-registry/image-registry-5c8f46447c-cdpct"
Sep 11 23:01:51.356111 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:51.356054    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9783f5ec-7c57-4a7a-a688-3afad6cda55d-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-7q8hx\" (UID: \"9783f5ec-7c57-4a7a-a688-3afad6cda55d\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-7q8hx"
Sep 11 23:01:51.356111 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:51.356094    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8ea4ca4e-b321-4324-b380-dab72513b02a-bound-sa-token\") pod \"image-registry-5c8f46447c-cdpct\" (UID: \"8ea4ca4e-b321-4324-b380-dab72513b02a\") " pod="openshift-image-registry/image-registry-5c8f46447c-cdpct"
Sep 11 23:01:51.356373 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:51.356178    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8ea4ca4e-b321-4324-b380-dab72513b02a-registry-certificates\") pod \"image-registry-5c8f46447c-cdpct\" (UID: \"8ea4ca4e-b321-4324-b380-dab72513b02a\") " pod="openshift-image-registry/image-registry-5c8f46447c-cdpct"
Sep 11 23:01:51.356373 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:51.356211    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8ea4ca4e-b321-4324-b380-dab72513b02a-registry-tls\") pod \"image-registry-5c8f46447c-cdpct\" (UID: \"8ea4ca4e-b321-4324-b380-dab72513b02a\") " pod="openshift-image-registry/image-registry-5c8f46447c-cdpct"
Sep 11 23:01:51.356373 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:51.356238    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8ea4ca4e-b321-4324-b380-dab72513b02a-ca-trust-extracted\") pod \"image-registry-5c8f46447c-cdpct\" (UID: \"8ea4ca4e-b321-4324-b380-dab72513b02a\") " pod="openshift-image-registry/image-registry-5c8f46447c-cdpct"
Sep 11 23:01:51.356373 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:51.356272    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9783f5ec-7c57-4a7a-a688-3afad6cda55d-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-7q8hx\" (UID: \"9783f5ec-7c57-4a7a-a688-3afad6cda55d\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-7q8hx"
Sep 11 23:01:51.456989 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:51.456957    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8ea4ca4e-b321-4324-b380-dab72513b02a-registry-tls\") pod \"image-registry-5c8f46447c-cdpct\" (UID: \"8ea4ca4e-b321-4324-b380-dab72513b02a\") " pod="openshift-image-registry/image-registry-5c8f46447c-cdpct"
Sep 11 23:01:51.456989 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:51.456992    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8ea4ca4e-b321-4324-b380-dab72513b02a-ca-trust-extracted\") pod \"image-registry-5c8f46447c-cdpct\" (UID: \"8ea4ca4e-b321-4324-b380-dab72513b02a\") " pod="openshift-image-registry/image-registry-5c8f46447c-cdpct"
Sep 11 23:01:51.457257 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:51.457024    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j67z6\" (UniqueName: \"kubernetes.io/projected/efcb43e4-c096-422f-bc0b-2c42795b0adc-kube-api-access-j67z6\") pod \"ingress-canary-glt22\" (UID: \"efcb43e4-c096-422f-bc0b-2c42795b0adc\") " pod="openshift-ingress-canary/ingress-canary-glt22"
Sep 11 23:01:51.457257 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:51.457057    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9783f5ec-7c57-4a7a-a688-3afad6cda55d-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-7q8hx\" (UID: \"9783f5ec-7c57-4a7a-a688-3afad6cda55d\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-7q8hx"
Sep 11 23:01:51.457257 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:51.457080    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98962ebe-ac82-4d92-a979-022ad4873bc9-tmp-dir\") pod \"dns-default-fzfcr\" (UID: \"98962ebe-ac82-4d92-a979-022ad4873bc9\") " pod="openshift-dns/dns-default-fzfcr"
Sep 11 23:01:51.457257 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:51.457137    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/efcb43e4-c096-422f-bc0b-2c42795b0adc-cert\") pod \"ingress-canary-glt22\" (UID: \"efcb43e4-c096-422f-bc0b-2c42795b0adc\") " pod="openshift-ingress-canary/ingress-canary-glt22"
Sep 11 23:01:51.457257 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:51.457197    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-987rm\" (UniqueName: \"kubernetes.io/projected/8ea4ca4e-b321-4324-b380-dab72513b02a-kube-api-access-987rm\") pod \"image-registry-5c8f46447c-cdpct\" (UID: \"8ea4ca4e-b321-4324-b380-dab72513b02a\") " pod="openshift-image-registry/image-registry-5c8f46447c-cdpct"
Sep 11 23:01:51.457257 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:51.457139    2546 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 23:01:51.457257 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:51.457233    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr574\" (UniqueName: \"kubernetes.io/projected/98962ebe-ac82-4d92-a979-022ad4873bc9-kube-api-access-nr574\") pod \"dns-default-fzfcr\" (UID: \"98962ebe-ac82-4d92-a979-022ad4873bc9\") " pod="openshift-dns/dns-default-fzfcr"
Sep 11 23:01:51.457257 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:51.457254    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8ea4ca4e-b321-4324-b380-dab72513b02a-trusted-ca\") pod \"image-registry-5c8f46447c-cdpct\" (UID: \"8ea4ca4e-b321-4324-b380-dab72513b02a\") " pod="openshift-image-registry/image-registry-5c8f46447c-cdpct"
Sep 11 23:01:51.457615 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:51.457283    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8ea4ca4e-b321-4324-b380-dab72513b02a-image-registry-private-configuration\") pod \"image-registry-5c8f46447c-cdpct\" (UID: \"8ea4ca4e-b321-4324-b380-dab72513b02a\") " pod="openshift-image-registry/image-registry-5c8f46447c-cdpct"
Sep 11 23:01:51.457615 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:51.457308    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8ea4ca4e-b321-4324-b380-dab72513b02a-installation-pull-secrets\") pod \"image-registry-5c8f46447c-cdpct\" (UID: \"8ea4ca4e-b321-4324-b380-dab72513b02a\") " pod="openshift-image-registry/image-registry-5c8f46447c-cdpct"
Sep 11 23:01:51.457615 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:51.457326    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9783f5ec-7c57-4a7a-a688-3afad6cda55d-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-7q8hx\" (UID: \"9783f5ec-7c57-4a7a-a688-3afad6cda55d\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-7q8hx"
Sep 11 23:01:51.457615 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:51.457349    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8ea4ca4e-b321-4324-b380-dab72513b02a-bound-sa-token\") pod \"image-registry-5c8f46447c-cdpct\" (UID: \"8ea4ca4e-b321-4324-b380-dab72513b02a\") " pod="openshift-image-registry/image-registry-5c8f46447c-cdpct"
Sep 11 23:01:51.457615 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:51.457365    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/98962ebe-ac82-4d92-a979-022ad4873bc9-metrics-tls\") pod \"dns-default-fzfcr\" (UID: \"98962ebe-ac82-4d92-a979-022ad4873bc9\") " pod="openshift-dns/dns-default-fzfcr"
Sep 11 23:01:51.457615 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:51.457395    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/98962ebe-ac82-4d92-a979-022ad4873bc9-config-volume\") pod \"dns-default-fzfcr\" (UID: \"98962ebe-ac82-4d92-a979-022ad4873bc9\") " pod="openshift-dns/dns-default-fzfcr"
Sep 11 23:01:51.457615 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:51.457417    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8ea4ca4e-b321-4324-b380-dab72513b02a-registry-certificates\") pod \"image-registry-5c8f46447c-cdpct\" (UID: \"8ea4ca4e-b321-4324-b380-dab72513b02a\") " pod="openshift-image-registry/image-registry-5c8f46447c-cdpct"
Sep 11 23:01:51.457615 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:51.457467    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8ea4ca4e-b321-4324-b380-dab72513b02a-ca-trust-extracted\") pod \"image-registry-5c8f46447c-cdpct\" (UID: \"8ea4ca4e-b321-4324-b380-dab72513b02a\") " pod="openshift-image-registry/image-registry-5c8f46447c-cdpct"
Sep 11 23:01:51.457973 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:51.457820    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9783f5ec-7c57-4a7a-a688-3afad6cda55d-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-7q8hx\" (UID: \"9783f5ec-7c57-4a7a-a688-3afad6cda55d\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-7q8hx"
Sep 11 23:01:51.457973 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:51.457237    2546 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c8f46447c-cdpct: secret "image-registry-tls" not found
Sep 11 23:01:51.457973 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:51.457960    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8ea4ca4e-b321-4324-b380-dab72513b02a-registry-certificates\") pod \"image-registry-5c8f46447c-cdpct\" (UID: \"8ea4ca4e-b321-4324-b380-dab72513b02a\") " pod="openshift-image-registry/image-registry-5c8f46447c-cdpct"
Sep 11 23:01:51.458110 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:51.457978    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8ea4ca4e-b321-4324-b380-dab72513b02a-registry-tls podName:8ea4ca4e-b321-4324-b380-dab72513b02a nodeName:}" failed. No retries permitted until 2026-09-11 23:01:51.957945521 +0000 UTC m=+35.991074729 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8ea4ca4e-b321-4324-b380-dab72513b02a-registry-tls") pod "image-registry-5c8f46447c-cdpct" (UID: "8ea4ca4e-b321-4324-b380-dab72513b02a") : secret "image-registry-tls" not found
Sep 11 23:01:51.458275 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:51.458256    2546 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 11 23:01:51.458401 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:51.458312    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9783f5ec-7c57-4a7a-a688-3afad6cda55d-networking-console-plugin-cert podName:9783f5ec-7c57-4a7a-a688-3afad6cda55d nodeName:}" failed. No retries permitted until 2026-09-11 23:01:51.958293908 +0000 UTC m=+35.991423103 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9783f5ec-7c57-4a7a-a688-3afad6cda55d-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-7q8hx" (UID: "9783f5ec-7c57-4a7a-a688-3afad6cda55d") : secret "networking-console-plugin-cert" not found
Sep 11 23:01:51.462460 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:51.462436    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8ea4ca4e-b321-4324-b380-dab72513b02a-installation-pull-secrets\") pod \"image-registry-5c8f46447c-cdpct\" (UID: \"8ea4ca4e-b321-4324-b380-dab72513b02a\") " pod="openshift-image-registry/image-registry-5c8f46447c-cdpct"
Sep 11 23:01:51.462569 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:51.462465    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8ea4ca4e-b321-4324-b380-dab72513b02a-image-registry-private-configuration\") pod \"image-registry-5c8f46447c-cdpct\" (UID: \"8ea4ca4e-b321-4324-b380-dab72513b02a\") " pod="openshift-image-registry/image-registry-5c8f46447c-cdpct"
Sep 11 23:01:51.463516 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:51.463493    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8ea4ca4e-b321-4324-b380-dab72513b02a-trusted-ca\") pod \"image-registry-5c8f46447c-cdpct\" (UID: \"8ea4ca4e-b321-4324-b380-dab72513b02a\") " pod="openshift-image-registry/image-registry-5c8f46447c-cdpct"
Sep 11 23:01:51.469968 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:51.469909    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8ea4ca4e-b321-4324-b380-dab72513b02a-bound-sa-token\") pod \"image-registry-5c8f46447c-cdpct\" (UID: \"8ea4ca4e-b321-4324-b380-dab72513b02a\") " pod="openshift-image-registry/image-registry-5c8f46447c-cdpct"
Sep 11 23:01:51.470312 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:51.470260    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-987rm\" (UniqueName: \"kubernetes.io/projected/8ea4ca4e-b321-4324-b380-dab72513b02a-kube-api-access-987rm\") pod \"image-registry-5c8f46447c-cdpct\" (UID: \"8ea4ca4e-b321-4324-b380-dab72513b02a\") " pod="openshift-image-registry/image-registry-5c8f46447c-cdpct"
Sep 11 23:01:51.558250 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:51.558209    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/efcb43e4-c096-422f-bc0b-2c42795b0adc-cert\") pod \"ingress-canary-glt22\" (UID: \"efcb43e4-c096-422f-bc0b-2c42795b0adc\") " pod="openshift-ingress-canary/ingress-canary-glt22"
Sep 11 23:01:51.558250 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:51.558257    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr574\" (UniqueName: \"kubernetes.io/projected/98962ebe-ac82-4d92-a979-022ad4873bc9-kube-api-access-nr574\") pod \"dns-default-fzfcr\" (UID: \"98962ebe-ac82-4d92-a979-022ad4873bc9\") " pod="openshift-dns/dns-default-fzfcr"
Sep 11 23:01:51.558761 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:51.558423    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/98962ebe-ac82-4d92-a979-022ad4873bc9-metrics-tls\") pod \"dns-default-fzfcr\" (UID: \"98962ebe-ac82-4d92-a979-022ad4873bc9\") " pod="openshift-dns/dns-default-fzfcr"
Sep 11 23:01:51.558761 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:51.558481    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/98962ebe-ac82-4d92-a979-022ad4873bc9-config-volume\") pod \"dns-default-fzfcr\" (UID: \"98962ebe-ac82-4d92-a979-022ad4873bc9\") " pod="openshift-dns/dns-default-fzfcr"
Sep 11 23:01:51.558761 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:51.558531    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j67z6\" (UniqueName: \"kubernetes.io/projected/efcb43e4-c096-422f-bc0b-2c42795b0adc-kube-api-access-j67z6\") pod \"ingress-canary-glt22\" (UID: \"efcb43e4-c096-422f-bc0b-2c42795b0adc\") " pod="openshift-ingress-canary/ingress-canary-glt22"
Sep 11 23:01:51.558761 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:51.558555    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98962ebe-ac82-4d92-a979-022ad4873bc9-tmp-dir\") pod \"dns-default-fzfcr\" (UID: \"98962ebe-ac82-4d92-a979-022ad4873bc9\") " pod="openshift-dns/dns-default-fzfcr"
Sep 11 23:01:51.558761 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:51.558582    2546 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 23:01:51.558761 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:51.558654    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98962ebe-ac82-4d92-a979-022ad4873bc9-metrics-tls podName:98962ebe-ac82-4d92-a979-022ad4873bc9 nodeName:}" failed. No retries permitted until 2026-09-11 23:01:52.058627961 +0000 UTC m=+36.091757160 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/98962ebe-ac82-4d92-a979-022ad4873bc9-metrics-tls") pod "dns-default-fzfcr" (UID: "98962ebe-ac82-4d92-a979-022ad4873bc9") : secret "dns-default-metrics-tls" not found
Sep 11 23:01:51.559085 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:51.558872    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98962ebe-ac82-4d92-a979-022ad4873bc9-tmp-dir\") pod \"dns-default-fzfcr\" (UID: \"98962ebe-ac82-4d92-a979-022ad4873bc9\") " pod="openshift-dns/dns-default-fzfcr"
Sep 11 23:01:51.559199 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:51.559171    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/98962ebe-ac82-4d92-a979-022ad4873bc9-config-volume\") pod \"dns-default-fzfcr\" (UID: \"98962ebe-ac82-4d92-a979-022ad4873bc9\") " pod="openshift-dns/dns-default-fzfcr"
Sep 11 23:01:51.561573 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:51.561550    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/efcb43e4-c096-422f-bc0b-2c42795b0adc-cert\") pod \"ingress-canary-glt22\" (UID: \"efcb43e4-c096-422f-bc0b-2c42795b0adc\") " pod="openshift-ingress-canary/ingress-canary-glt22"
Sep 11 23:01:51.568403 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:51.568372    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr574\" (UniqueName: \"kubernetes.io/projected/98962ebe-ac82-4d92-a979-022ad4873bc9-kube-api-access-nr574\") pod \"dns-default-fzfcr\" (UID: \"98962ebe-ac82-4d92-a979-022ad4873bc9\") " pod="openshift-dns/dns-default-fzfcr"
Sep 11 23:01:51.568919 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:51.568898    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j67z6\" (UniqueName: \"kubernetes.io/projected/efcb43e4-c096-422f-bc0b-2c42795b0adc-kube-api-access-j67z6\") pod \"ingress-canary-glt22\" (UID: \"efcb43e4-c096-422f-bc0b-2c42795b0adc\") " pod="openshift-ingress-canary/ingress-canary-glt22"
Sep 11 23:01:51.619010 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:51.618976    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-glt22"
Sep 11 23:01:51.962789 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:51.962754    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8ea4ca4e-b321-4324-b380-dab72513b02a-registry-tls\") pod \"image-registry-5c8f46447c-cdpct\" (UID: \"8ea4ca4e-b321-4324-b380-dab72513b02a\") " pod="openshift-image-registry/image-registry-5c8f46447c-cdpct"
Sep 11 23:01:51.962985 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:51.962858    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9783f5ec-7c57-4a7a-a688-3afad6cda55d-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-7q8hx\" (UID: \"9783f5ec-7c57-4a7a-a688-3afad6cda55d\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-7q8hx"
Sep 11 23:01:51.962985 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:51.962923    2546 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 23:01:51.962985 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:51.962945    2546 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c8f46447c-cdpct: secret "image-registry-tls" not found
Sep 11 23:01:51.963127 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:51.962986    2546 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 11 23:01:51.963127 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:51.963011    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8ea4ca4e-b321-4324-b380-dab72513b02a-registry-tls podName:8ea4ca4e-b321-4324-b380-dab72513b02a nodeName:}" failed. No retries permitted until 2026-09-11 23:01:52.962990122 +0000 UTC m=+36.996119317 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8ea4ca4e-b321-4324-b380-dab72513b02a-registry-tls") pod "image-registry-5c8f46447c-cdpct" (UID: "8ea4ca4e-b321-4324-b380-dab72513b02a") : secret "image-registry-tls" not found
Sep 11 23:01:51.963127 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:51.963040    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9783f5ec-7c57-4a7a-a688-3afad6cda55d-networking-console-plugin-cert podName:9783f5ec-7c57-4a7a-a688-3afad6cda55d nodeName:}" failed. No retries permitted until 2026-09-11 23:01:52.963024192 +0000 UTC m=+36.996153403 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9783f5ec-7c57-4a7a-a688-3afad6cda55d-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-7q8hx" (UID: "9783f5ec-7c57-4a7a-a688-3afad6cda55d") : secret "networking-console-plugin-cert" not found
Sep 11 23:01:52.063899 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:52.063860    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/98962ebe-ac82-4d92-a979-022ad4873bc9-metrics-tls\") pod \"dns-default-fzfcr\" (UID: \"98962ebe-ac82-4d92-a979-022ad4873bc9\") " pod="openshift-dns/dns-default-fzfcr"
Sep 11 23:01:52.064062 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:52.064017    2546 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 23:01:52.064117 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:52.064083    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98962ebe-ac82-4d92-a979-022ad4873bc9-metrics-tls podName:98962ebe-ac82-4d92-a979-022ad4873bc9 nodeName:}" failed. No retries permitted until 2026-09-11 23:01:53.064068819 +0000 UTC m=+37.097198014 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/98962ebe-ac82-4d92-a979-022ad4873bc9-metrics-tls") pod "dns-default-fzfcr" (UID: "98962ebe-ac82-4d92-a979-022ad4873bc9") : secret "dns-default-metrics-tls" not found
Sep 11 23:01:52.370444 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:52.370137    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-glt22"]
Sep 11 23:01:52.373508 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:01:52.373480    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefcb43e4_c096_422f_bc0b_2c42795b0adc.slice/crio-008fd7906131e523d8a5b1937630798cae5e5a78a94b7ec69f55a80cc5cbdd76 WatchSource:0}: Error finding container 008fd7906131e523d8a5b1937630798cae5e5a78a94b7ec69f55a80cc5cbdd76: Status 404 returned error can't find the container with id 008fd7906131e523d8a5b1937630798cae5e5a78a94b7ec69f55a80cc5cbdd76
Sep 11 23:01:52.488279 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:52.488252    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jdr7s"
Sep 11 23:01:52.488406 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:52.488305    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vbgsz"
Sep 11 23:01:52.488406 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:52.488320    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5n55v"
Sep 11 23:01:52.493519 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:52.493496    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 23:01:52.493632 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:52.493501    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 23:01:52.493632 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:52.493551    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-rtlm9\""
Sep 11 23:01:52.493858 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:52.493843    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 23:01:52.493858 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:52.493852    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 23:01:52.493993 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:52.493910    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qjwfh\""
Sep 11 23:01:52.700252 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:52.700216    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-glt22" event={"ID":"efcb43e4-c096-422f-bc0b-2c42795b0adc","Type":"ContainerStarted","Data":"008fd7906131e523d8a5b1937630798cae5e5a78a94b7ec69f55a80cc5cbdd76"}
Sep 11 23:01:52.703025 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:52.702990    2546 generic.go:356] "Generic (PLEG): container finished" podID="0155dec9-f8e2-4007-9861-b8c870223adb" containerID="aa6024359f43da945b8363f4f09c9f8a4e9d2170be9aa26be09707ed8495401e" exitCode=0
Sep 11 23:01:52.703129 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:52.703027    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bv5gf" event={"ID":"0155dec9-f8e2-4007-9861-b8c870223adb","Type":"ContainerDied","Data":"aa6024359f43da945b8363f4f09c9f8a4e9d2170be9aa26be09707ed8495401e"}
Sep 11 23:01:52.971698 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:52.971647    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8ea4ca4e-b321-4324-b380-dab72513b02a-registry-tls\") pod \"image-registry-5c8f46447c-cdpct\" (UID: \"8ea4ca4e-b321-4324-b380-dab72513b02a\") " pod="openshift-image-registry/image-registry-5c8f46447c-cdpct"
Sep 11 23:01:52.971885 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:52.971765    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9783f5ec-7c57-4a7a-a688-3afad6cda55d-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-7q8hx\" (UID: \"9783f5ec-7c57-4a7a-a688-3afad6cda55d\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-7q8hx"
Sep 11 23:01:52.971885 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:52.971829    2546 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 23:01:52.971885 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:52.971852    2546 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c8f46447c-cdpct: secret "image-registry-tls" not found
Sep 11 23:01:52.972034 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:52.971899    2546 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 11 23:01:52.972034 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:52.971909    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8ea4ca4e-b321-4324-b380-dab72513b02a-registry-tls podName:8ea4ca4e-b321-4324-b380-dab72513b02a nodeName:}" failed. No retries permitted until 2026-09-11 23:01:54.971893844 +0000 UTC m=+39.005023042 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8ea4ca4e-b321-4324-b380-dab72513b02a-registry-tls") pod "image-registry-5c8f46447c-cdpct" (UID: "8ea4ca4e-b321-4324-b380-dab72513b02a") : secret "image-registry-tls" not found
Sep 11 23:01:52.972034 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:52.971954    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9783f5ec-7c57-4a7a-a688-3afad6cda55d-networking-console-plugin-cert podName:9783f5ec-7c57-4a7a-a688-3afad6cda55d nodeName:}" failed. No retries permitted until 2026-09-11 23:01:54.971937661 +0000 UTC m=+39.005066873 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9783f5ec-7c57-4a7a-a688-3afad6cda55d-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-7q8hx" (UID: "9783f5ec-7c57-4a7a-a688-3afad6cda55d") : secret "networking-console-plugin-cert" not found
Sep 11 23:01:53.072891 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:53.072843    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/98962ebe-ac82-4d92-a979-022ad4873bc9-metrics-tls\") pod \"dns-default-fzfcr\" (UID: \"98962ebe-ac82-4d92-a979-022ad4873bc9\") " pod="openshift-dns/dns-default-fzfcr"
Sep 11 23:01:53.073080 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:53.073024    2546 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 23:01:53.073144 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:53.073127    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98962ebe-ac82-4d92-a979-022ad4873bc9-metrics-tls podName:98962ebe-ac82-4d92-a979-022ad4873bc9 nodeName:}" failed. No retries permitted until 2026-09-11 23:01:55.073104011 +0000 UTC m=+39.106233211 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/98962ebe-ac82-4d92-a979-022ad4873bc9-metrics-tls") pod "dns-default-fzfcr" (UID: "98962ebe-ac82-4d92-a979-022ad4873bc9") : secret "dns-default-metrics-tls" not found
Sep 11 23:01:53.709055 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:53.709015    2546 generic.go:356] "Generic (PLEG): container finished" podID="0155dec9-f8e2-4007-9861-b8c870223adb" containerID="9a86fa94b3c40ff17133eb2492ab031ae8b4fe37f182d4cb559c35f49693a1c9" exitCode=0
Sep 11 23:01:53.709055 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:53.709057    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bv5gf" event={"ID":"0155dec9-f8e2-4007-9861-b8c870223adb","Type":"ContainerDied","Data":"9a86fa94b3c40ff17133eb2492ab031ae8b4fe37f182d4cb559c35f49693a1c9"}
Sep 11 23:01:54.715420 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:54.715372    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bv5gf" event={"ID":"0155dec9-f8e2-4007-9861-b8c870223adb","Type":"ContainerStarted","Data":"b0625c2101d7a04e47c39b92ab523b57ac54642a22eba7af6bead00a3f7fe708"}
Sep 11 23:01:54.739701 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:54.739635    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-bv5gf" podStartSLOduration=7.582979187 podStartE2EDuration="38.739614498s" totalImagesPullingTime="31.156635311s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 23:01:16 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:01:54.737250325 +0000 UTC m=+38.770379550" watchObservedRunningTime="2026-09-11 23:01:54.739614498 +0000 UTC m=+38.772743718"
Sep 11 23:01:54.988503 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:54.988425    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9783f5ec-7c57-4a7a-a688-3afad6cda55d-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-7q8hx\" (UID: \"9783f5ec-7c57-4a7a-a688-3afad6cda55d\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-7q8hx"
Sep 11 23:01:54.988503 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:54.988487    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8ea4ca4e-b321-4324-b380-dab72513b02a-registry-tls\") pod \"image-registry-5c8f46447c-cdpct\" (UID: \"8ea4ca4e-b321-4324-b380-dab72513b02a\") " pod="openshift-image-registry/image-registry-5c8f46447c-cdpct"
Sep 11 23:01:54.988671 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:54.988572    2546 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 11 23:01:54.988671 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:54.988599    2546 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 23:01:54.988671 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:54.988612    2546 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c8f46447c-cdpct: secret "image-registry-tls" not found
Sep 11 23:01:54.988671 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:54.988648    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9783f5ec-7c57-4a7a-a688-3afad6cda55d-networking-console-plugin-cert podName:9783f5ec-7c57-4a7a-a688-3afad6cda55d nodeName:}" failed. No retries permitted until 2026-09-11 23:01:58.988632353 +0000 UTC m=+43.021761567 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9783f5ec-7c57-4a7a-a688-3afad6cda55d-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-7q8hx" (UID: "9783f5ec-7c57-4a7a-a688-3afad6cda55d") : secret "networking-console-plugin-cert" not found
Sep 11 23:01:54.988671 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:54.988665    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8ea4ca4e-b321-4324-b380-dab72513b02a-registry-tls podName:8ea4ca4e-b321-4324-b380-dab72513b02a nodeName:}" failed. No retries permitted until 2026-09-11 23:01:58.988658545 +0000 UTC m=+43.021787740 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8ea4ca4e-b321-4324-b380-dab72513b02a-registry-tls") pod "image-registry-5c8f46447c-cdpct" (UID: "8ea4ca4e-b321-4324-b380-dab72513b02a") : secret "image-registry-tls" not found
Sep 11 23:01:55.089246 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:55.089194    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/98962ebe-ac82-4d92-a979-022ad4873bc9-metrics-tls\") pod \"dns-default-fzfcr\" (UID: \"98962ebe-ac82-4d92-a979-022ad4873bc9\") " pod="openshift-dns/dns-default-fzfcr"
Sep 11 23:01:55.089394 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:55.089350    2546 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 23:01:55.089432 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:55.089419    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98962ebe-ac82-4d92-a979-022ad4873bc9-metrics-tls podName:98962ebe-ac82-4d92-a979-022ad4873bc9 nodeName:}" failed. No retries permitted until 2026-09-11 23:01:59.089403681 +0000 UTC m=+43.122532880 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/98962ebe-ac82-4d92-a979-022ad4873bc9-metrics-tls") pod "dns-default-fzfcr" (UID: "98962ebe-ac82-4d92-a979-022ad4873bc9") : secret "dns-default-metrics-tls" not found
Sep 11 23:01:55.719748 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:55.719537    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-glt22" event={"ID":"efcb43e4-c096-422f-bc0b-2c42795b0adc","Type":"ContainerStarted","Data":"0bde422962ffeba065f7b8e957903433be0aa7d01bb13bb7bf5c3ae5f41e02b8"}
Sep 11 23:01:55.738552 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:55.738500    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-glt22" podStartSLOduration=2.017222403 podStartE2EDuration="4.738483266s" totalImagesPullingTime="2.721260863s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 23:01:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:01:55.736753312 +0000 UTC m=+39.769882547" watchObservedRunningTime="2026-09-11 23:01:55.738483266 +0000 UTC m=+39.771612482"
Sep 11 23:01:59.019956 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:59.019919    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9783f5ec-7c57-4a7a-a688-3afad6cda55d-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-7q8hx\" (UID: \"9783f5ec-7c57-4a7a-a688-3afad6cda55d\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-7q8hx"
Sep 11 23:01:59.020357 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:59.019979    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8ea4ca4e-b321-4324-b380-dab72513b02a-registry-tls\") pod \"image-registry-5c8f46447c-cdpct\" (UID: \"8ea4ca4e-b321-4324-b380-dab72513b02a\") " pod="openshift-image-registry/image-registry-5c8f46447c-cdpct"
Sep 11 23:01:59.020357 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:59.020072    2546 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 23:01:59.020357 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:59.020076    2546 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 11 23:01:59.020357 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:59.020082    2546 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c8f46447c-cdpct: secret "image-registry-tls" not found
Sep 11 23:01:59.020357 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:59.020140    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9783f5ec-7c57-4a7a-a688-3afad6cda55d-networking-console-plugin-cert podName:9783f5ec-7c57-4a7a-a688-3afad6cda55d nodeName:}" failed. No retries permitted until 2026-09-11 23:02:07.020125041 +0000 UTC m=+51.053254236 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9783f5ec-7c57-4a7a-a688-3afad6cda55d-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-7q8hx" (UID: "9783f5ec-7c57-4a7a-a688-3afad6cda55d") : secret "networking-console-plugin-cert" not found
Sep 11 23:01:59.020357 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:59.020195    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8ea4ca4e-b321-4324-b380-dab72513b02a-registry-tls podName:8ea4ca4e-b321-4324-b380-dab72513b02a nodeName:}" failed. No retries permitted until 2026-09-11 23:02:07.020178463 +0000 UTC m=+51.053307675 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8ea4ca4e-b321-4324-b380-dab72513b02a-registry-tls") pod "image-registry-5c8f46447c-cdpct" (UID: "8ea4ca4e-b321-4324-b380-dab72513b02a") : secret "image-registry-tls" not found
Sep 11 23:01:59.121175 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:01:59.121129    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/98962ebe-ac82-4d92-a979-022ad4873bc9-metrics-tls\") pod \"dns-default-fzfcr\" (UID: \"98962ebe-ac82-4d92-a979-022ad4873bc9\") " pod="openshift-dns/dns-default-fzfcr"
Sep 11 23:01:59.121329 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:59.121273    2546 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 23:01:59.121367 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:01:59.121335    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98962ebe-ac82-4d92-a979-022ad4873bc9-metrics-tls podName:98962ebe-ac82-4d92-a979-022ad4873bc9 nodeName:}" failed. No retries permitted until 2026-09-11 23:02:07.121321165 +0000 UTC m=+51.154450360 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/98962ebe-ac82-4d92-a979-022ad4873bc9-metrics-tls") pod "dns-default-fzfcr" (UID: "98962ebe-ac82-4d92-a979-022ad4873bc9") : secret "dns-default-metrics-tls" not found
Sep 11 23:02:07.080197 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:02:07.080136    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8ea4ca4e-b321-4324-b380-dab72513b02a-registry-tls\") pod \"image-registry-5c8f46447c-cdpct\" (UID: \"8ea4ca4e-b321-4324-b380-dab72513b02a\") " pod="openshift-image-registry/image-registry-5c8f46447c-cdpct"
Sep 11 23:02:07.080644 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:02:07.080266    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9783f5ec-7c57-4a7a-a688-3afad6cda55d-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-7q8hx\" (UID: \"9783f5ec-7c57-4a7a-a688-3afad6cda55d\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-7q8hx"
Sep 11 23:02:07.080644 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:02:07.080318    2546 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 23:02:07.080644 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:02:07.080335    2546 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c8f46447c-cdpct: secret "image-registry-tls" not found
Sep 11 23:02:07.080644 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:02:07.080395    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8ea4ca4e-b321-4324-b380-dab72513b02a-registry-tls podName:8ea4ca4e-b321-4324-b380-dab72513b02a nodeName:}" failed. No retries permitted until 2026-09-11 23:02:23.080380231 +0000 UTC m=+67.113509425 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8ea4ca4e-b321-4324-b380-dab72513b02a-registry-tls") pod "image-registry-5c8f46447c-cdpct" (UID: "8ea4ca4e-b321-4324-b380-dab72513b02a") : secret "image-registry-tls" not found
Sep 11 23:02:07.080644 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:02:07.080415    2546 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 11 23:02:07.080644 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:02:07.080491    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9783f5ec-7c57-4a7a-a688-3afad6cda55d-networking-console-plugin-cert podName:9783f5ec-7c57-4a7a-a688-3afad6cda55d nodeName:}" failed. No retries permitted until 2026-09-11 23:02:23.080474704 +0000 UTC m=+67.113603919 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9783f5ec-7c57-4a7a-a688-3afad6cda55d-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-7q8hx" (UID: "9783f5ec-7c57-4a7a-a688-3afad6cda55d") : secret "networking-console-plugin-cert" not found
Sep 11 23:02:07.180861 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:02:07.180833    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/98962ebe-ac82-4d92-a979-022ad4873bc9-metrics-tls\") pod \"dns-default-fzfcr\" (UID: \"98962ebe-ac82-4d92-a979-022ad4873bc9\") " pod="openshift-dns/dns-default-fzfcr"
Sep 11 23:02:07.181008 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:02:07.180990    2546 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 23:02:07.181059 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:02:07.181048    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98962ebe-ac82-4d92-a979-022ad4873bc9-metrics-tls podName:98962ebe-ac82-4d92-a979-022ad4873bc9 nodeName:}" failed. No retries permitted until 2026-09-11 23:02:23.181035051 +0000 UTC m=+67.214164247 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/98962ebe-ac82-4d92-a979-022ad4873bc9-metrics-tls") pod "dns-default-fzfcr" (UID: "98962ebe-ac82-4d92-a979-022ad4873bc9") : secret "dns-default-metrics-tls" not found
Sep 11 23:02:17.699840 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:02:17.699812    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-p7244"
Sep 11 23:02:21.179955 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:02:21.179910    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50607ccd-47f9-463a-bdc7-181484a00caa-metrics-certs\") pod \"network-metrics-daemon-vbgsz\" (UID: \"50607ccd-47f9-463a-bdc7-181484a00caa\") " pod="openshift-multus/network-metrics-daemon-vbgsz"
Sep 11 23:02:21.183060 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:02:21.183043    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 23:02:21.190793 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:02:21.190769    2546 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 11 23:02:21.190859 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:02:21.190849    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50607ccd-47f9-463a-bdc7-181484a00caa-metrics-certs podName:50607ccd-47f9-463a-bdc7-181484a00caa nodeName:}" failed. No retries permitted until 2026-09-11 23:03:25.190827303 +0000 UTC m=+129.223956499 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/50607ccd-47f9-463a-bdc7-181484a00caa-metrics-certs") pod "network-metrics-daemon-vbgsz" (UID: "50607ccd-47f9-463a-bdc7-181484a00caa") : secret "metrics-daemon-secret" not found
Sep 11 23:02:21.281171 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:02:21.281119    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3ad2cd03-8eae-49c4-8c94-ea45de3d3c7a-original-pull-secret\") pod \"global-pull-secret-syncer-5n55v\" (UID: \"3ad2cd03-8eae-49c4-8c94-ea45de3d3c7a\") " pod="kube-system/global-pull-secret-syncer-5n55v"
Sep 11 23:02:21.284626 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:02:21.284600    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 23:02:21.294720 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:02:21.294695    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3ad2cd03-8eae-49c4-8c94-ea45de3d3c7a-original-pull-secret\") pod \"global-pull-secret-syncer-5n55v\" (UID: \"3ad2cd03-8eae-49c4-8c94-ea45de3d3c7a\") " pod="kube-system/global-pull-secret-syncer-5n55v"
Sep 11 23:02:21.327298 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:02:21.327276    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5n55v"
Sep 11 23:02:21.381720 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:02:21.381690    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cn25v\" (UniqueName: \"kubernetes.io/projected/223f5b7c-785a-4cfb-9196-6f4adb77d14a-kube-api-access-cn25v\") pod \"network-check-target-jdr7s\" (UID: \"223f5b7c-785a-4cfb-9196-6f4adb77d14a\") " pod="openshift-network-diagnostics/network-check-target-jdr7s"
Sep 11 23:02:21.384886 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:02:21.384863    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 23:02:21.397092 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:02:21.396799    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 23:02:21.405748 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:02:21.405719    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cn25v\" (UniqueName: \"kubernetes.io/projected/223f5b7c-785a-4cfb-9196-6f4adb77d14a-kube-api-access-cn25v\") pod \"network-check-target-jdr7s\" (UID: \"223f5b7c-785a-4cfb-9196-6f4adb77d14a\") " pod="openshift-network-diagnostics/network-check-target-jdr7s"
Sep 11 23:02:21.601738 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:02:21.601665    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5n55v"]
Sep 11 23:02:21.605070 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:02:21.605046    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ad2cd03_8eae_49c4_8c94_ea45de3d3c7a.slice/crio-4723d59618c4e807b7088f56484189786c8cc210afb43c5708451fae213de60b WatchSource:0}: Error finding container 4723d59618c4e807b7088f56484189786c8cc210afb43c5708451fae213de60b: Status 404 returned error can't find the container with id 4723d59618c4e807b7088f56484189786c8cc210afb43c5708451fae213de60b
Sep 11 23:02:21.626181 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:02:21.626143    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qjwfh\""
Sep 11 23:02:21.634227 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:02:21.634206    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jdr7s"
Sep 11 23:02:21.783510 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:02:21.783468    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5n55v" event={"ID":"3ad2cd03-8eae-49c4-8c94-ea45de3d3c7a","Type":"ContainerStarted","Data":"4723d59618c4e807b7088f56484189786c8cc210afb43c5708451fae213de60b"}
Sep 11 23:02:21.929189 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:02:21.929139    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jdr7s"]
Sep 11 23:02:21.931644 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:02:21.931608    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod223f5b7c_785a_4cfb_9196_6f4adb77d14a.slice/crio-6c9291e144b30a26329539a58306a7455e61923390daa32d3840e412f082df23 WatchSource:0}: Error finding container 6c9291e144b30a26329539a58306a7455e61923390daa32d3840e412f082df23: Status 404 returned error can't find the container with id 6c9291e144b30a26329539a58306a7455e61923390daa32d3840e412f082df23
Sep 11 23:02:22.788334 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:02:22.788293    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jdr7s" event={"ID":"223f5b7c-785a-4cfb-9196-6f4adb77d14a","Type":"ContainerStarted","Data":"6c9291e144b30a26329539a58306a7455e61923390daa32d3840e412f082df23"}
Sep 11 23:02:23.097685 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:02:23.097600    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9783f5ec-7c57-4a7a-a688-3afad6cda55d-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-7q8hx\" (UID: \"9783f5ec-7c57-4a7a-a688-3afad6cda55d\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-7q8hx"
Sep 11 23:02:23.097685 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:02:23.097668    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8ea4ca4e-b321-4324-b380-dab72513b02a-registry-tls\") pod \"image-registry-5c8f46447c-cdpct\" (UID: \"8ea4ca4e-b321-4324-b380-dab72513b02a\") " pod="openshift-image-registry/image-registry-5c8f46447c-cdpct"
Sep 11 23:02:23.098045 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:02:23.097785    2546 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 11 23:02:23.098045 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:02:23.097815    2546 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 23:02:23.098045 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:02:23.097831    2546 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c8f46447c-cdpct: secret "image-registry-tls" not found
Sep 11 23:02:23.098045 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:02:23.097866    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9783f5ec-7c57-4a7a-a688-3afad6cda55d-networking-console-plugin-cert podName:9783f5ec-7c57-4a7a-a688-3afad6cda55d nodeName:}" failed. No retries permitted until 2026-09-11 23:02:55.097844865 +0000 UTC m=+99.130974064 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9783f5ec-7c57-4a7a-a688-3afad6cda55d-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-7q8hx" (UID: "9783f5ec-7c57-4a7a-a688-3afad6cda55d") : secret "networking-console-plugin-cert" not found
Sep 11 23:02:23.098045 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:02:23.097886    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8ea4ca4e-b321-4324-b380-dab72513b02a-registry-tls podName:8ea4ca4e-b321-4324-b380-dab72513b02a nodeName:}" failed. No retries permitted until 2026-09-11 23:02:55.097876777 +0000 UTC m=+99.131005978 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8ea4ca4e-b321-4324-b380-dab72513b02a-registry-tls") pod "image-registry-5c8f46447c-cdpct" (UID: "8ea4ca4e-b321-4324-b380-dab72513b02a") : secret "image-registry-tls" not found
Sep 11 23:02:23.198577 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:02:23.198473    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/98962ebe-ac82-4d92-a979-022ad4873bc9-metrics-tls\") pod \"dns-default-fzfcr\" (UID: \"98962ebe-ac82-4d92-a979-022ad4873bc9\") " pod="openshift-dns/dns-default-fzfcr"
Sep 11 23:02:23.198759 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:02:23.198627    2546 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 23:02:23.198759 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:02:23.198705    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98962ebe-ac82-4d92-a979-022ad4873bc9-metrics-tls podName:98962ebe-ac82-4d92-a979-022ad4873bc9 nodeName:}" failed. No retries permitted until 2026-09-11 23:02:55.198684164 +0000 UTC m=+99.231813360 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/98962ebe-ac82-4d92-a979-022ad4873bc9-metrics-tls") pod "dns-default-fzfcr" (UID: "98962ebe-ac82-4d92-a979-022ad4873bc9") : secret "dns-default-metrics-tls" not found
Sep 11 23:02:26.800262 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:02:26.800149    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jdr7s" event={"ID":"223f5b7c-785a-4cfb-9196-6f4adb77d14a","Type":"ContainerStarted","Data":"b76469f4f73afb17a473b62cb56c5add2bedd6c3be519071ee4f6fe3cefe2d67"}
Sep 11 23:02:26.800683 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:02:26.800280    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-jdr7s"
Sep 11 23:02:26.801666 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:02:26.801645    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5n55v" event={"ID":"3ad2cd03-8eae-49c4-8c94-ea45de3d3c7a","Type":"ContainerStarted","Data":"cf36f5695210ca5766171df3cb9c35cdc9128d09b5f0798644508afc5e9bd1f0"}
Sep 11 23:02:26.817668 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:02:26.817595    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-jdr7s" podStartSLOduration=66.191668363 podStartE2EDuration="1m10.817583409s" totalImagesPullingTime="4.625915046s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 23:01:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:02:26.815826656 +0000 UTC m=+70.848955874" watchObservedRunningTime="2026-09-11 23:02:26.817583409 +0000 UTC m=+70.850712640"
Sep 11 23:02:26.830715 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:02:26.830670    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-5n55v" podStartSLOduration=65.875303034 podStartE2EDuration="1m10.830657777s" totalImagesPullingTime="4.955354743s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 23:01:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:02:26.829993382 +0000 UTC m=+70.863122599" watchObservedRunningTime="2026-09-11 23:02:26.830657777 +0000 UTC m=+70.863786994"
Sep 11 23:02:55.116811 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:02:55.116762    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9783f5ec-7c57-4a7a-a688-3afad6cda55d-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-7q8hx\" (UID: \"9783f5ec-7c57-4a7a-a688-3afad6cda55d\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-7q8hx"
Sep 11 23:02:55.117401 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:02:55.116831    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8ea4ca4e-b321-4324-b380-dab72513b02a-registry-tls\") pod \"image-registry-5c8f46447c-cdpct\" (UID: \"8ea4ca4e-b321-4324-b380-dab72513b02a\") " pod="openshift-image-registry/image-registry-5c8f46447c-cdpct"
Sep 11 23:02:55.117401 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:02:55.116918    2546 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 11 23:02:55.117401 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:02:55.116945    2546 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 23:02:55.117401 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:02:55.116969    2546 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c8f46447c-cdpct: secret "image-registry-tls" not found
Sep 11 23:02:55.117401 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:02:55.117006    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9783f5ec-7c57-4a7a-a688-3afad6cda55d-networking-console-plugin-cert podName:9783f5ec-7c57-4a7a-a688-3afad6cda55d nodeName:}" failed. No retries permitted until 2026-09-11 23:03:59.116990005 +0000 UTC m=+163.150119201 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9783f5ec-7c57-4a7a-a688-3afad6cda55d-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-7q8hx" (UID: "9783f5ec-7c57-4a7a-a688-3afad6cda55d") : secret "networking-console-plugin-cert" not found
Sep 11 23:02:55.117401 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:02:55.117035    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8ea4ca4e-b321-4324-b380-dab72513b02a-registry-tls podName:8ea4ca4e-b321-4324-b380-dab72513b02a nodeName:}" failed. No retries permitted until 2026-09-11 23:03:59.117019745 +0000 UTC m=+163.150148959 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8ea4ca4e-b321-4324-b380-dab72513b02a-registry-tls") pod "image-registry-5c8f46447c-cdpct" (UID: "8ea4ca4e-b321-4324-b380-dab72513b02a") : secret "image-registry-tls" not found
Sep 11 23:02:55.217614 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:02:55.217580    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/98962ebe-ac82-4d92-a979-022ad4873bc9-metrics-tls\") pod \"dns-default-fzfcr\" (UID: \"98962ebe-ac82-4d92-a979-022ad4873bc9\") " pod="openshift-dns/dns-default-fzfcr"
Sep 11 23:02:55.217741 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:02:55.217722    2546 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 23:02:55.217793 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:02:55.217784    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98962ebe-ac82-4d92-a979-022ad4873bc9-metrics-tls podName:98962ebe-ac82-4d92-a979-022ad4873bc9 nodeName:}" failed. No retries permitted until 2026-09-11 23:03:59.217770254 +0000 UTC m=+163.250899448 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/98962ebe-ac82-4d92-a979-022ad4873bc9-metrics-tls") pod "dns-default-fzfcr" (UID: "98962ebe-ac82-4d92-a979-022ad4873bc9") : secret "dns-default-metrics-tls" not found
Sep 11 23:02:57.807310 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:02:57.807277    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-jdr7s"
Sep 11 23:03:25.228749 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:03:25.228696    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50607ccd-47f9-463a-bdc7-181484a00caa-metrics-certs\") pod \"network-metrics-daemon-vbgsz\" (UID: \"50607ccd-47f9-463a-bdc7-181484a00caa\") " pod="openshift-multus/network-metrics-daemon-vbgsz"
Sep 11 23:03:25.229278 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:03:25.228851    2546 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 11 23:03:25.229278 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:03:25.228933    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50607ccd-47f9-463a-bdc7-181484a00caa-metrics-certs podName:50607ccd-47f9-463a-bdc7-181484a00caa nodeName:}" failed. No retries permitted until 2026-09-11 23:05:27.228916517 +0000 UTC m=+251.262045712 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/50607ccd-47f9-463a-bdc7-181484a00caa-metrics-certs") pod "network-metrics-daemon-vbgsz" (UID: "50607ccd-47f9-463a-bdc7-181484a00caa") : secret "metrics-daemon-secret" not found
Sep 11 23:03:54.288749 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:03:54.288686    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-7q8hx" podUID="9783f5ec-7c57-4a7a-a688-3afad6cda55d"
Sep 11 23:03:54.302877 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:03:54.302849    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-5c8f46447c-cdpct" podUID="8ea4ca4e-b321-4324-b380-dab72513b02a"
Sep 11 23:03:54.331332 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:03:54.331300    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-fzfcr" podUID="98962ebe-ac82-4d92-a979-022ad4873bc9"
Sep 11 23:03:55.012834 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:03:55.012801    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fzfcr"
Sep 11 23:03:55.012834 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:03:55.012830    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c8f46447c-cdpct"
Sep 11 23:03:55.013046 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:03:55.012938    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-7q8hx"
Sep 11 23:03:55.517256 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:03:55.517218    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-vbgsz" podUID="50607ccd-47f9-463a-bdc7-181484a00caa"
Sep 11 23:03:59.166753 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:03:59.166696    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8ea4ca4e-b321-4324-b380-dab72513b02a-registry-tls\") pod \"image-registry-5c8f46447c-cdpct\" (UID: \"8ea4ca4e-b321-4324-b380-dab72513b02a\") " pod="openshift-image-registry/image-registry-5c8f46447c-cdpct"
Sep 11 23:03:59.167229 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:03:59.166811    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9783f5ec-7c57-4a7a-a688-3afad6cda55d-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-7q8hx\" (UID: \"9783f5ec-7c57-4a7a-a688-3afad6cda55d\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-7q8hx"
Sep 11 23:03:59.167229 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:03:59.166859    2546 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 23:03:59.167229 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:03:59.166877    2546 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c8f46447c-cdpct: secret "image-registry-tls" not found
Sep 11 23:03:59.167229 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:03:59.166937    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8ea4ca4e-b321-4324-b380-dab72513b02a-registry-tls podName:8ea4ca4e-b321-4324-b380-dab72513b02a nodeName:}" failed. No retries permitted until 2026-09-11 23:06:01.16692152 +0000 UTC m=+285.200050719 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8ea4ca4e-b321-4324-b380-dab72513b02a-registry-tls") pod "image-registry-5c8f46447c-cdpct" (UID: "8ea4ca4e-b321-4324-b380-dab72513b02a") : secret "image-registry-tls" not found
Sep 11 23:03:59.167229 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:03:59.166938    2546 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 11 23:03:59.167229 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:03:59.167006    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9783f5ec-7c57-4a7a-a688-3afad6cda55d-networking-console-plugin-cert podName:9783f5ec-7c57-4a7a-a688-3afad6cda55d nodeName:}" failed. No retries permitted until 2026-09-11 23:06:01.16698891 +0000 UTC m=+285.200118106 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9783f5ec-7c57-4a7a-a688-3afad6cda55d-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-7q8hx" (UID: "9783f5ec-7c57-4a7a-a688-3afad6cda55d") : secret "networking-console-plugin-cert" not found
Sep 11 23:03:59.267436 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:03:59.267394    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/98962ebe-ac82-4d92-a979-022ad4873bc9-metrics-tls\") pod \"dns-default-fzfcr\" (UID: \"98962ebe-ac82-4d92-a979-022ad4873bc9\") " pod="openshift-dns/dns-default-fzfcr"
Sep 11 23:03:59.267638 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:03:59.267582    2546 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 23:03:59.267708 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:03:59.267680    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98962ebe-ac82-4d92-a979-022ad4873bc9-metrics-tls podName:98962ebe-ac82-4d92-a979-022ad4873bc9 nodeName:}" failed. No retries permitted until 2026-09-11 23:06:01.267656457 +0000 UTC m=+285.300785653 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/98962ebe-ac82-4d92-a979-022ad4873bc9-metrics-tls") pod "dns-default-fzfcr" (UID: "98962ebe-ac82-4d92-a979-022ad4873bc9") : secret "dns-default-metrics-tls" not found
Sep 11 23:04:07.488141 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:04:07.488088    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vbgsz"
Sep 11 23:05:27.310294 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:27.310253    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50607ccd-47f9-463a-bdc7-181484a00caa-metrics-certs\") pod \"network-metrics-daemon-vbgsz\" (UID: \"50607ccd-47f9-463a-bdc7-181484a00caa\") " pod="openshift-multus/network-metrics-daemon-vbgsz"
Sep 11 23:05:27.310767 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:05:27.310367    2546 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 11 23:05:27.310767 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:05:27.310421    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50607ccd-47f9-463a-bdc7-181484a00caa-metrics-certs podName:50607ccd-47f9-463a-bdc7-181484a00caa nodeName:}" failed. No retries permitted until 2026-09-11 23:07:29.310405902 +0000 UTC m=+373.343535097 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/50607ccd-47f9-463a-bdc7-181484a00caa-metrics-certs") pod "network-metrics-daemon-vbgsz" (UID: "50607ccd-47f9-463a-bdc7-181484a00caa") : secret "metrics-daemon-secret" not found
Sep 11 23:05:41.978811 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:41.978732    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-5tht9"]
Sep 11 23:05:41.981625 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:41.981607    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-5tht9"
Sep 11 23:05:41.984052 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:41.984029    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-8psvk"]
Sep 11 23:05:41.985613 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:41.985590    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-8qvz9\""
Sep 11 23:05:41.986888 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:41.986810    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-8psvk"
Sep 11 23:05:41.987009 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:41.986984    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 11 23:05:41.987103 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:41.987016    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 11 23:05:41.992524 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:41.992501    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 11 23:05:41.997424 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:41.996944    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-jj8kb\""
Sep 11 23:05:41.997424 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:41.996941    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 11 23:05:41.997424 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:41.997268    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 11 23:05:41.997424 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:41.997340    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 11 23:05:41.998262 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:41.997941    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-5tht9"]
Sep 11 23:05:42.007327 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.007304    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-8psvk"]
Sep 11 23:05:42.013096 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.013072    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pl5v9\" (UniqueName: \"kubernetes.io/projected/8b564021-1e62-4eac-a86f-7a3a92980aa3-kube-api-access-pl5v9\") pod \"volume-data-source-validator-86695b7cc7-5tht9\" (UID: \"8b564021-1e62-4eac-a86f-7a3a92980aa3\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-5tht9"
Sep 11 23:05:42.013213 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.013125    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/23f6c775-d2ea-4c5f-8bd4-da89b88950c0-config\") pod \"service-ca-operator-cf7c776b7-8psvk\" (UID: \"23f6c775-d2ea-4c5f-8bd4-da89b88950c0\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-8psvk"
Sep 11 23:05:42.013292 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.013268    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsmtc\" (UniqueName: \"kubernetes.io/projected/23f6c775-d2ea-4c5f-8bd4-da89b88950c0-kube-api-access-qsmtc\") pod \"service-ca-operator-cf7c776b7-8psvk\" (UID: \"23f6c775-d2ea-4c5f-8bd4-da89b88950c0\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-8psvk"
Sep 11 23:05:42.013341 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.013314    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/23f6c775-d2ea-4c5f-8bd4-da89b88950c0-serving-cert\") pod \"service-ca-operator-cf7c776b7-8psvk\" (UID: \"23f6c775-d2ea-4c5f-8bd4-da89b88950c0\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-8psvk"
Sep 11 23:05:42.080025 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.079992    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-5qpm8"]
Sep 11 23:05:42.082841 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.082819    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-qtf5g"]
Sep 11 23:05:42.082954 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.082933    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-5qpm8"
Sep 11 23:05:42.085539 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.085522    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-h6w6f"]
Sep 11 23:05:42.085685 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.085670    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-qtf5g"
Sep 11 23:05:42.086670 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.086650    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-vs5wf\""
Sep 11 23:05:42.088480 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.088461    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-xr6z6"]
Sep 11 23:05:42.088571 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.088513    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-h6w6f"
Sep 11 23:05:42.088626 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.088462    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 11 23:05:42.088809 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.088790    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 11 23:05:42.089214 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.089188    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 11 23:05:42.089337 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.089323    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-59kcp\""
Sep 11 23:05:42.091219 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.091201    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-xr6z6"
Sep 11 23:05:42.091633 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.091613    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 11 23:05:42.093857 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.093836    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 11 23:05:42.094059 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.093848    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-xc2jx\""
Sep 11 23:05:42.094181 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.093889    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 11 23:05:42.094181 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.093894    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 11 23:05:42.094565 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.094540    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 11 23:05:42.094672 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.094638    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-4xgzp\""
Sep 11 23:05:42.094760 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.094715    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 11 23:05:42.094760 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.094639    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 11 23:05:42.095295 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.095276    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 11 23:05:42.098056 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.098035    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-c8fff6cd4-tr742"]
Sep 11 23:05:42.100817 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.100795    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 11 23:05:42.100932 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.100912    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-c8fff6cd4-tr742"
Sep 11 23:05:42.101987 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.101932    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-5qpm8"]
Sep 11 23:05:42.103766 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.103746    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-qtf5g"]
Sep 11 23:05:42.104386 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.104367    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 11 23:05:42.104537 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.104517    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-xr6z6"]
Sep 11 23:05:42.104600 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.104571    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 11 23:05:42.104660 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.104647    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 11 23:05:42.104711 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.104694    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 11 23:05:42.104834 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.104816    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-wvsll\""
Sep 11 23:05:42.104931 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.104826    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 11 23:05:42.104931 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.104891    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 11 23:05:42.113594 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.113573    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a6dd2771-bd9e-4e48-976e-d2f4c14eba11-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-h6w6f\" (UID: \"a6dd2771-bd9e-4e48-976e-d2f4c14eba11\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-h6w6f"
Sep 11 23:05:42.113690 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.113601    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a6dd2771-bd9e-4e48-976e-d2f4c14eba11-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-h6w6f\" (UID: \"a6dd2771-bd9e-4e48-976e-d2f4c14eba11\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-h6w6f"
Sep 11 23:05:42.113690 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.113624    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/68e4123a-25ea-4c58-a934-d44495740042-tmp\") pod \"insights-operator-5844b856db-xr6z6\" (UID: \"68e4123a-25ea-4c58-a934-d44495740042\") " pod="openshift-insights/insights-operator-5844b856db-xr6z6"
Sep 11 23:05:42.113690 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.113640    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/68e4123a-25ea-4c58-a934-d44495740042-serving-cert\") pod \"insights-operator-5844b856db-xr6z6\" (UID: \"68e4123a-25ea-4c58-a934-d44495740042\") " pod="openshift-insights/insights-operator-5844b856db-xr6z6"
Sep 11 23:05:42.113813 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.113695    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/42864299-d484-4b6b-a2c7-dea808adb9f6-stats-auth\") pod \"router-default-c8fff6cd4-tr742\" (UID: \"42864299-d484-4b6b-a2c7-dea808adb9f6\") " pod="openshift-ingress/router-default-c8fff6cd4-tr742"
Sep 11 23:05:42.113813 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.113758    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/23f6c775-d2ea-4c5f-8bd4-da89b88950c0-config\") pod \"service-ca-operator-cf7c776b7-8psvk\" (UID: \"23f6c775-d2ea-4c5f-8bd4-da89b88950c0\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-8psvk"
Sep 11 23:05:42.113813 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.113788    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68e4123a-25ea-4c58-a934-d44495740042-service-ca-bundle\") pod \"insights-operator-5844b856db-xr6z6\" (UID: \"68e4123a-25ea-4c58-a934-d44495740042\") " pod="openshift-insights/insights-operator-5844b856db-xr6z6"
Sep 11 23:05:42.113921 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.113817    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fdb719e6-5372-4e41-b059-d6cc84575f67-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-qtf5g\" (UID: \"fdb719e6-5372-4e41-b059-d6cc84575f67\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-qtf5g"
Sep 11 23:05:42.113921 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.113843    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42864299-d484-4b6b-a2c7-dea808adb9f6-metrics-certs\") pod \"router-default-c8fff6cd4-tr742\" (UID: \"42864299-d484-4b6b-a2c7-dea808adb9f6\") " pod="openshift-ingress/router-default-c8fff6cd4-tr742"
Sep 11 23:05:42.113921 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.113877    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdf6g\" (UniqueName: \"kubernetes.io/projected/68e4123a-25ea-4c58-a934-d44495740042-kube-api-access-vdf6g\") pod \"insights-operator-5844b856db-xr6z6\" (UID: \"68e4123a-25ea-4c58-a934-d44495740042\") " pod="openshift-insights/insights-operator-5844b856db-xr6z6"
Sep 11 23:05:42.113921 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.113904    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/23f6c775-d2ea-4c5f-8bd4-da89b88950c0-serving-cert\") pod \"service-ca-operator-cf7c776b7-8psvk\" (UID: \"23f6c775-d2ea-4c5f-8bd4-da89b88950c0\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-8psvk"
Sep 11 23:05:42.114099 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.113940    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42864299-d484-4b6b-a2c7-dea808adb9f6-service-ca-bundle\") pod \"router-default-c8fff6cd4-tr742\" (UID: \"42864299-d484-4b6b-a2c7-dea808adb9f6\") " pod="openshift-ingress/router-default-c8fff6cd4-tr742"
Sep 11 23:05:42.114099 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.114037    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/42864299-d484-4b6b-a2c7-dea808adb9f6-default-certificate\") pod \"router-default-c8fff6cd4-tr742\" (UID: \"42864299-d484-4b6b-a2c7-dea808adb9f6\") " pod="openshift-ingress/router-default-c8fff6cd4-tr742"
Sep 11 23:05:42.114099 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.114088    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pl5v9\" (UniqueName: \"kubernetes.io/projected/8b564021-1e62-4eac-a86f-7a3a92980aa3-kube-api-access-pl5v9\") pod \"volume-data-source-validator-86695b7cc7-5tht9\" (UID: \"8b564021-1e62-4eac-a86f-7a3a92980aa3\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-5tht9"
Sep 11 23:05:42.114289 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.114117    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ds5f\" (UniqueName: \"kubernetes.io/projected/42864299-d484-4b6b-a2c7-dea808adb9f6-kube-api-access-9ds5f\") pod \"router-default-c8fff6cd4-tr742\" (UID: \"42864299-d484-4b6b-a2c7-dea808adb9f6\") " pod="openshift-ingress/router-default-c8fff6cd4-tr742"
Sep 11 23:05:42.114289 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.114143    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9k6xx\" (UniqueName: \"kubernetes.io/projected/fdb719e6-5372-4e41-b059-d6cc84575f67-kube-api-access-9k6xx\") pod \"cluster-samples-operator-86499c89c4-qtf5g\" (UID: \"fdb719e6-5372-4e41-b059-d6cc84575f67\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-qtf5g"
Sep 11 23:05:42.114289 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.114188    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/68e4123a-25ea-4c58-a934-d44495740042-snapshots\") pod \"insights-operator-5844b856db-xr6z6\" (UID: \"68e4123a-25ea-4c58-a934-d44495740042\") " pod="openshift-insights/insights-operator-5844b856db-xr6z6"
Sep 11 23:05:42.114289 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.114224    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6b4gs\" (UniqueName: \"kubernetes.io/projected/3604600b-6950-442f-a4cc-1120274abc96-kube-api-access-6b4gs\") pod \"network-check-source-6f8b768455-5qpm8\" (UID: \"3604600b-6950-442f-a4cc-1120274abc96\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-5qpm8"
Sep 11 23:05:42.114289 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.114262    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsmtc\" (UniqueName: \"kubernetes.io/projected/23f6c775-d2ea-4c5f-8bd4-da89b88950c0-kube-api-access-qsmtc\") pod \"service-ca-operator-cf7c776b7-8psvk\" (UID: \"23f6c775-d2ea-4c5f-8bd4-da89b88950c0\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-8psvk"
Sep 11 23:05:42.114289 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.114290    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlcrn\" (UniqueName: \"kubernetes.io/projected/a6dd2771-bd9e-4e48-976e-d2f4c14eba11-kube-api-access-nlcrn\") pod \"cluster-monitoring-operator-6cd954b897-h6w6f\" (UID: \"a6dd2771-bd9e-4e48-976e-d2f4c14eba11\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-h6w6f"
Sep 11 23:05:42.114548 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.114307    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68e4123a-25ea-4c58-a934-d44495740042-trusted-ca-bundle\") pod \"insights-operator-5844b856db-xr6z6\" (UID: \"68e4123a-25ea-4c58-a934-d44495740042\") " pod="openshift-insights/insights-operator-5844b856db-xr6z6"
Sep 11 23:05:42.114548 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.114343    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/23f6c775-d2ea-4c5f-8bd4-da89b88950c0-config\") pod \"service-ca-operator-cf7c776b7-8psvk\" (UID: \"23f6c775-d2ea-4c5f-8bd4-da89b88950c0\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-8psvk"
Sep 11 23:05:42.116586 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.116568    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/23f6c775-d2ea-4c5f-8bd4-da89b88950c0-serving-cert\") pod \"service-ca-operator-cf7c776b7-8psvk\" (UID: \"23f6c775-d2ea-4c5f-8bd4-da89b88950c0\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-8psvk"
Sep 11 23:05:42.120678 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.120636    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-c8fff6cd4-tr742"]
Sep 11 23:05:42.123350 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.123331    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-h6w6f"]
Sep 11 23:05:42.131740 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.131715    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsmtc\" (UniqueName: \"kubernetes.io/projected/23f6c775-d2ea-4c5f-8bd4-da89b88950c0-kube-api-access-qsmtc\") pod \"service-ca-operator-cf7c776b7-8psvk\" (UID: \"23f6c775-d2ea-4c5f-8bd4-da89b88950c0\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-8psvk"
Sep 11 23:05:42.132304 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.132283    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pl5v9\" (UniqueName: \"kubernetes.io/projected/8b564021-1e62-4eac-a86f-7a3a92980aa3-kube-api-access-pl5v9\") pod \"volume-data-source-validator-86695b7cc7-5tht9\" (UID: \"8b564021-1e62-4eac-a86f-7a3a92980aa3\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-5tht9"
Sep 11 23:05:42.215029 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.214992    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/68e4123a-25ea-4c58-a934-d44495740042-tmp\") pod \"insights-operator-5844b856db-xr6z6\" (UID: \"68e4123a-25ea-4c58-a934-d44495740042\") " pod="openshift-insights/insights-operator-5844b856db-xr6z6"
Sep 11 23:05:42.215029 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.215025    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/68e4123a-25ea-4c58-a934-d44495740042-serving-cert\") pod \"insights-operator-5844b856db-xr6z6\" (UID: \"68e4123a-25ea-4c58-a934-d44495740042\") " pod="openshift-insights/insights-operator-5844b856db-xr6z6"
Sep 11 23:05:42.215317 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.215045    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/42864299-d484-4b6b-a2c7-dea808adb9f6-stats-auth\") pod \"router-default-c8fff6cd4-tr742\" (UID: \"42864299-d484-4b6b-a2c7-dea808adb9f6\") " pod="openshift-ingress/router-default-c8fff6cd4-tr742"
Sep 11 23:05:42.215317 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.215071    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68e4123a-25ea-4c58-a934-d44495740042-service-ca-bundle\") pod \"insights-operator-5844b856db-xr6z6\" (UID: \"68e4123a-25ea-4c58-a934-d44495740042\") " pod="openshift-insights/insights-operator-5844b856db-xr6z6"
Sep 11 23:05:42.215317 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.215091    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fdb719e6-5372-4e41-b059-d6cc84575f67-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-qtf5g\" (UID: \"fdb719e6-5372-4e41-b059-d6cc84575f67\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-qtf5g"
Sep 11 23:05:42.215317 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.215111    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42864299-d484-4b6b-a2c7-dea808adb9f6-metrics-certs\") pod \"router-default-c8fff6cd4-tr742\" (UID: \"42864299-d484-4b6b-a2c7-dea808adb9f6\") " pod="openshift-ingress/router-default-c8fff6cd4-tr742"
Sep 11 23:05:42.215317 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.215140    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdf6g\" (UniqueName: \"kubernetes.io/projected/68e4123a-25ea-4c58-a934-d44495740042-kube-api-access-vdf6g\") pod \"insights-operator-5844b856db-xr6z6\" (UID: \"68e4123a-25ea-4c58-a934-d44495740042\") " pod="openshift-insights/insights-operator-5844b856db-xr6z6"
Sep 11 23:05:42.215317 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.215209    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42864299-d484-4b6b-a2c7-dea808adb9f6-service-ca-bundle\") pod \"router-default-c8fff6cd4-tr742\" (UID: \"42864299-d484-4b6b-a2c7-dea808adb9f6\") " pod="openshift-ingress/router-default-c8fff6cd4-tr742"
Sep 11 23:05:42.215317 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.215230    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/42864299-d484-4b6b-a2c7-dea808adb9f6-default-certificate\") pod \"router-default-c8fff6cd4-tr742\" (UID: \"42864299-d484-4b6b-a2c7-dea808adb9f6\") " pod="openshift-ingress/router-default-c8fff6cd4-tr742"
Sep 11 23:05:42.215317 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.215282    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ds5f\" (UniqueName: \"kubernetes.io/projected/42864299-d484-4b6b-a2c7-dea808adb9f6-kube-api-access-9ds5f\") pod \"router-default-c8fff6cd4-tr742\" (UID: \"42864299-d484-4b6b-a2c7-dea808adb9f6\") " pod="openshift-ingress/router-default-c8fff6cd4-tr742"
Sep 11 23:05:42.215317 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.215307    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k6xx\" (UniqueName: \"kubernetes.io/projected/fdb719e6-5372-4e41-b059-d6cc84575f67-kube-api-access-9k6xx\") pod \"cluster-samples-operator-86499c89c4-qtf5g\" (UID: \"fdb719e6-5372-4e41-b059-d6cc84575f67\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-qtf5g"
Sep 11 23:05:42.215743 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:05:42.215230    2546 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 11 23:05:42.215743 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:05:42.215356    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/42864299-d484-4b6b-a2c7-dea808adb9f6-service-ca-bundle podName:42864299-d484-4b6b-a2c7-dea808adb9f6 nodeName:}" failed. No retries permitted until 2026-09-11 23:05:42.715335471 +0000 UTC m=+266.748464666 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/42864299-d484-4b6b-a2c7-dea808adb9f6-service-ca-bundle") pod "router-default-c8fff6cd4-tr742" (UID: "42864299-d484-4b6b-a2c7-dea808adb9f6") : configmap references non-existent config key: service-ca.crt
Sep 11 23:05:42.215743 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:05:42.215383    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdb719e6-5372-4e41-b059-d6cc84575f67-samples-operator-tls podName:fdb719e6-5372-4e41-b059-d6cc84575f67 nodeName:}" failed. No retries permitted until 2026-09-11 23:05:42.715369893 +0000 UTC m=+266.748499091 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/fdb719e6-5372-4e41-b059-d6cc84575f67-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-qtf5g" (UID: "fdb719e6-5372-4e41-b059-d6cc84575f67") : secret "samples-operator-tls" not found
Sep 11 23:05:42.215743 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.215404    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/68e4123a-25ea-4c58-a934-d44495740042-snapshots\") pod \"insights-operator-5844b856db-xr6z6\" (UID: \"68e4123a-25ea-4c58-a934-d44495740042\") " pod="openshift-insights/insights-operator-5844b856db-xr6z6"
Sep 11 23:05:42.215743 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.215437    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b4gs\" (UniqueName: \"kubernetes.io/projected/3604600b-6950-442f-a4cc-1120274abc96-kube-api-access-6b4gs\") pod \"network-check-source-6f8b768455-5qpm8\" (UID: \"3604600b-6950-442f-a4cc-1120274abc96\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-5qpm8"
Sep 11 23:05:42.215743 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.215467    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlcrn\" (UniqueName: \"kubernetes.io/projected/a6dd2771-bd9e-4e48-976e-d2f4c14eba11-kube-api-access-nlcrn\") pod \"cluster-monitoring-operator-6cd954b897-h6w6f\" (UID: \"a6dd2771-bd9e-4e48-976e-d2f4c14eba11\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-h6w6f"
Sep 11 23:05:42.215743 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.215494    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68e4123a-25ea-4c58-a934-d44495740042-trusted-ca-bundle\") pod \"insights-operator-5844b856db-xr6z6\" (UID: \"68e4123a-25ea-4c58-a934-d44495740042\") " pod="openshift-insights/insights-operator-5844b856db-xr6z6"
Sep 11 23:05:42.215743 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.215577    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a6dd2771-bd9e-4e48-976e-d2f4c14eba11-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-h6w6f\" (UID: \"a6dd2771-bd9e-4e48-976e-d2f4c14eba11\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-h6w6f"
Sep 11 23:05:42.215743 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.215582    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/68e4123a-25ea-4c58-a934-d44495740042-tmp\") pod \"insights-operator-5844b856db-xr6z6\" (UID: \"68e4123a-25ea-4c58-a934-d44495740042\") " pod="openshift-insights/insights-operator-5844b856db-xr6z6"
Sep 11 23:05:42.215743 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.215607    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a6dd2771-bd9e-4e48-976e-d2f4c14eba11-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-h6w6f\" (UID: \"a6dd2771-bd9e-4e48-976e-d2f4c14eba11\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-h6w6f"
Sep 11 23:05:42.215743 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:05:42.215239    2546 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 11 23:05:42.215743 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:05:42.215684    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42864299-d484-4b6b-a2c7-dea808adb9f6-metrics-certs podName:42864299-d484-4b6b-a2c7-dea808adb9f6 nodeName:}" failed. No retries permitted until 2026-09-11 23:05:42.715665718 +0000 UTC m=+266.748794930 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/42864299-d484-4b6b-a2c7-dea808adb9f6-metrics-certs") pod "router-default-c8fff6cd4-tr742" (UID: "42864299-d484-4b6b-a2c7-dea808adb9f6") : secret "router-metrics-certs-default" not found
Sep 11 23:05:42.215743 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.215728    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68e4123a-25ea-4c58-a934-d44495740042-service-ca-bundle\") pod \"insights-operator-5844b856db-xr6z6\" (UID: \"68e4123a-25ea-4c58-a934-d44495740042\") " pod="openshift-insights/insights-operator-5844b856db-xr6z6"
Sep 11 23:05:42.216398 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:05:42.216021    2546 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 11 23:05:42.216398 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:05:42.216080    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6dd2771-bd9e-4e48-976e-d2f4c14eba11-cluster-monitoring-operator-tls podName:a6dd2771-bd9e-4e48-976e-d2f4c14eba11 nodeName:}" failed. No retries permitted until 2026-09-11 23:05:42.716059856 +0000 UTC m=+266.749189055 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a6dd2771-bd9e-4e48-976e-d2f4c14eba11-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-h6w6f" (UID: "a6dd2771-bd9e-4e48-976e-d2f4c14eba11") : secret "cluster-monitoring-operator-tls" not found
Sep 11 23:05:42.216398 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.216341    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/68e4123a-25ea-4c58-a934-d44495740042-snapshots\") pod \"insights-operator-5844b856db-xr6z6\" (UID: \"68e4123a-25ea-4c58-a934-d44495740042\") " pod="openshift-insights/insights-operator-5844b856db-xr6z6"
Sep 11 23:05:42.216635 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.216613    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a6dd2771-bd9e-4e48-976e-d2f4c14eba11-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-h6w6f\" (UID: \"a6dd2771-bd9e-4e48-976e-d2f4c14eba11\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-h6w6f"
Sep 11 23:05:42.217303 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.217272    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68e4123a-25ea-4c58-a934-d44495740042-trusted-ca-bundle\") pod \"insights-operator-5844b856db-xr6z6\" (UID: \"68e4123a-25ea-4c58-a934-d44495740042\") " pod="openshift-insights/insights-operator-5844b856db-xr6z6"
Sep 11 23:05:42.217888 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.217864    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/42864299-d484-4b6b-a2c7-dea808adb9f6-stats-auth\") pod \"router-default-c8fff6cd4-tr742\" (UID: \"42864299-d484-4b6b-a2c7-dea808adb9f6\") " pod="openshift-ingress/router-default-c8fff6cd4-tr742"
Sep 11 23:05:42.218104 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.218079    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/42864299-d484-4b6b-a2c7-dea808adb9f6-default-certificate\") pod \"router-default-c8fff6cd4-tr742\" (UID: \"42864299-d484-4b6b-a2c7-dea808adb9f6\") " pod="openshift-ingress/router-default-c8fff6cd4-tr742"
Sep 11 23:05:42.218226 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.218209    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/68e4123a-25ea-4c58-a934-d44495740042-serving-cert\") pod \"insights-operator-5844b856db-xr6z6\" (UID: \"68e4123a-25ea-4c58-a934-d44495740042\") " pod="openshift-insights/insights-operator-5844b856db-xr6z6"
Sep 11 23:05:42.228178 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.228129    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ds5f\" (UniqueName: \"kubernetes.io/projected/42864299-d484-4b6b-a2c7-dea808adb9f6-kube-api-access-9ds5f\") pod \"router-default-c8fff6cd4-tr742\" (UID: \"42864299-d484-4b6b-a2c7-dea808adb9f6\") " pod="openshift-ingress/router-default-c8fff6cd4-tr742"
Sep 11 23:05:42.228374 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.228350    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlcrn\" (UniqueName: \"kubernetes.io/projected/a6dd2771-bd9e-4e48-976e-d2f4c14eba11-kube-api-access-nlcrn\") pod \"cluster-monitoring-operator-6cd954b897-h6w6f\" (UID: \"a6dd2771-bd9e-4e48-976e-d2f4c14eba11\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-h6w6f"
Sep 11 23:05:42.228512 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.228496    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdf6g\" (UniqueName: \"kubernetes.io/projected/68e4123a-25ea-4c58-a934-d44495740042-kube-api-access-vdf6g\") pod \"insights-operator-5844b856db-xr6z6\" (UID: \"68e4123a-25ea-4c58-a934-d44495740042\") " pod="openshift-insights/insights-operator-5844b856db-xr6z6"
Sep 11 23:05:42.228750 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.228729    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k6xx\" (UniqueName: \"kubernetes.io/projected/fdb719e6-5372-4e41-b059-d6cc84575f67-kube-api-access-9k6xx\") pod \"cluster-samples-operator-86499c89c4-qtf5g\" (UID: \"fdb719e6-5372-4e41-b059-d6cc84575f67\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-qtf5g"
Sep 11 23:05:42.229078 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.229039    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6b4gs\" (UniqueName: \"kubernetes.io/projected/3604600b-6950-442f-a4cc-1120274abc96-kube-api-access-6b4gs\") pod \"network-check-source-6f8b768455-5qpm8\" (UID: \"3604600b-6950-442f-a4cc-1120274abc96\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-5qpm8"
Sep 11 23:05:42.297749 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.297719    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-5tht9"
Sep 11 23:05:42.303917 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.303895    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-8psvk"
Sep 11 23:05:42.392877 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.392841    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-5qpm8"
Sep 11 23:05:42.413075 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.413043    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-xr6z6"
Sep 11 23:05:42.576438 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.576369    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-5tht9"]
Sep 11 23:05:42.579095 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.579076    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-8psvk"]
Sep 11 23:05:42.579614 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:05:42.579541    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b564021_1e62_4eac_a86f_7a3a92980aa3.slice/crio-07a60acf97b9a2cda24d4829fcc56a00105e7a7780dc19f79c5ec848aa791c9a WatchSource:0}: Error finding container 07a60acf97b9a2cda24d4829fcc56a00105e7a7780dc19f79c5ec848aa791c9a: Status 404 returned error can't find the container with id 07a60acf97b9a2cda24d4829fcc56a00105e7a7780dc19f79c5ec848aa791c9a
Sep 11 23:05:42.581379 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:05:42.581358    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23f6c775_d2ea_4c5f_8bd4_da89b88950c0.slice/crio-3ae2c42d036f3edbc7660aa5a7f3fd27eab8808f1fe44995f808a79e83fd7630 WatchSource:0}: Error finding container 3ae2c42d036f3edbc7660aa5a7f3fd27eab8808f1fe44995f808a79e83fd7630: Status 404 returned error can't find the container with id 3ae2c42d036f3edbc7660aa5a7f3fd27eab8808f1fe44995f808a79e83fd7630
Sep 11 23:05:42.662399 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.662265    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-5qpm8"]
Sep 11 23:05:42.664655 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:05:42.664619    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3604600b_6950_442f_a4cc_1120274abc96.slice/crio-2a688ef1db74d3474d61d0ce742ecdf63311c1c1fec1c9ae3dfd2c122dd67b83 WatchSource:0}: Error finding container 2a688ef1db74d3474d61d0ce742ecdf63311c1c1fec1c9ae3dfd2c122dd67b83: Status 404 returned error can't find the container with id 2a688ef1db74d3474d61d0ce742ecdf63311c1c1fec1c9ae3dfd2c122dd67b83
Sep 11 23:05:42.687378 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.687352    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-xr6z6"]
Sep 11 23:05:42.695114 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:05:42.695087    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68e4123a_25ea_4c58_a934_d44495740042.slice/crio-7e0310c72751fa8f0d05b1001546e881be94c7c23ebda849ec7091844b57830e WatchSource:0}: Error finding container 7e0310c72751fa8f0d05b1001546e881be94c7c23ebda849ec7091844b57830e: Status 404 returned error can't find the container with id 7e0310c72751fa8f0d05b1001546e881be94c7c23ebda849ec7091844b57830e
Sep 11 23:05:42.721103 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.721079    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fdb719e6-5372-4e41-b059-d6cc84575f67-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-qtf5g\" (UID: \"fdb719e6-5372-4e41-b059-d6cc84575f67\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-qtf5g"
Sep 11 23:05:42.721226 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.721116    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42864299-d484-4b6b-a2c7-dea808adb9f6-metrics-certs\") pod \"router-default-c8fff6cd4-tr742\" (UID: \"42864299-d484-4b6b-a2c7-dea808adb9f6\") " pod="openshift-ingress/router-default-c8fff6cd4-tr742"
Sep 11 23:05:42.721226 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.721177    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42864299-d484-4b6b-a2c7-dea808adb9f6-service-ca-bundle\") pod \"router-default-c8fff6cd4-tr742\" (UID: \"42864299-d484-4b6b-a2c7-dea808adb9f6\") " pod="openshift-ingress/router-default-c8fff6cd4-tr742"
Sep 11 23:05:42.721329 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:42.721260    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a6dd2771-bd9e-4e48-976e-d2f4c14eba11-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-h6w6f\" (UID: \"a6dd2771-bd9e-4e48-976e-d2f4c14eba11\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-h6w6f"
Sep 11 23:05:42.721329 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:05:42.721271    2546 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 11 23:05:42.721439 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:05:42.721342    2546 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 11 23:05:42.721439 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:05:42.721274    2546 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 11 23:05:42.721439 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:05:42.721346    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42864299-d484-4b6b-a2c7-dea808adb9f6-metrics-certs podName:42864299-d484-4b6b-a2c7-dea808adb9f6 nodeName:}" failed. No retries permitted until 2026-09-11 23:05:43.721326415 +0000 UTC m=+267.754455626 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/42864299-d484-4b6b-a2c7-dea808adb9f6-metrics-certs") pod "router-default-c8fff6cd4-tr742" (UID: "42864299-d484-4b6b-a2c7-dea808adb9f6") : secret "router-metrics-certs-default" not found
Sep 11 23:05:42.721439 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:05:42.721396    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdb719e6-5372-4e41-b059-d6cc84575f67-samples-operator-tls podName:fdb719e6-5372-4e41-b059-d6cc84575f67 nodeName:}" failed. No retries permitted until 2026-09-11 23:05:43.721379433 +0000 UTC m=+267.754508629 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/fdb719e6-5372-4e41-b059-d6cc84575f67-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-qtf5g" (UID: "fdb719e6-5372-4e41-b059-d6cc84575f67") : secret "samples-operator-tls" not found
Sep 11 23:05:42.721439 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:05:42.721421    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/42864299-d484-4b6b-a2c7-dea808adb9f6-service-ca-bundle podName:42864299-d484-4b6b-a2c7-dea808adb9f6 nodeName:}" failed. No retries permitted until 2026-09-11 23:05:43.721413891 +0000 UTC m=+267.754543085 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/42864299-d484-4b6b-a2c7-dea808adb9f6-service-ca-bundle") pod "router-default-c8fff6cd4-tr742" (UID: "42864299-d484-4b6b-a2c7-dea808adb9f6") : configmap references non-existent config key: service-ca.crt
Sep 11 23:05:42.721439 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:05:42.721432    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6dd2771-bd9e-4e48-976e-d2f4c14eba11-cluster-monitoring-operator-tls podName:a6dd2771-bd9e-4e48-976e-d2f4c14eba11 nodeName:}" failed. No retries permitted until 2026-09-11 23:05:43.721426839 +0000 UTC m=+267.754556033 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a6dd2771-bd9e-4e48-976e-d2f4c14eba11-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-h6w6f" (UID: "a6dd2771-bd9e-4e48-976e-d2f4c14eba11") : secret "cluster-monitoring-operator-tls" not found
Sep 11 23:05:43.278180 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:43.278085    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-5tht9" event={"ID":"8b564021-1e62-4eac-a86f-7a3a92980aa3","Type":"ContainerStarted","Data":"07a60acf97b9a2cda24d4829fcc56a00105e7a7780dc19f79c5ec848aa791c9a"}
Sep 11 23:05:43.280025 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:43.279992    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-xr6z6" event={"ID":"68e4123a-25ea-4c58-a934-d44495740042","Type":"ContainerStarted","Data":"7e0310c72751fa8f0d05b1001546e881be94c7c23ebda849ec7091844b57830e"}
Sep 11 23:05:43.283811 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:43.283780    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-5qpm8" event={"ID":"3604600b-6950-442f-a4cc-1120274abc96","Type":"ContainerStarted","Data":"f41dbcdb9011c40ec9cb648c9ada643daa3706371d1cc82e2e7c9cb7fd4e0f9a"}
Sep 11 23:05:43.283943 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:43.283832    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-5qpm8" event={"ID":"3604600b-6950-442f-a4cc-1120274abc96","Type":"ContainerStarted","Data":"2a688ef1db74d3474d61d0ce742ecdf63311c1c1fec1c9ae3dfd2c122dd67b83"}
Sep 11 23:05:43.285964 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:43.285937    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-8psvk" event={"ID":"23f6c775-d2ea-4c5f-8bd4-da89b88950c0","Type":"ContainerStarted","Data":"3ae2c42d036f3edbc7660aa5a7f3fd27eab8808f1fe44995f808a79e83fd7630"}
Sep 11 23:05:43.302949 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:43.302893    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-5qpm8" podStartSLOduration=1.3028738020000001 podStartE2EDuration="1.302873802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 23:05:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:05:43.301082101 +0000 UTC m=+267.334211317" watchObservedRunningTime="2026-09-11 23:05:43.302873802 +0000 UTC m=+267.336003018"
Sep 11 23:05:43.729620 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:43.729582    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fdb719e6-5372-4e41-b059-d6cc84575f67-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-qtf5g\" (UID: \"fdb719e6-5372-4e41-b059-d6cc84575f67\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-qtf5g"
Sep 11 23:05:43.729620 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:43.729627    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42864299-d484-4b6b-a2c7-dea808adb9f6-metrics-certs\") pod \"router-default-c8fff6cd4-tr742\" (UID: \"42864299-d484-4b6b-a2c7-dea808adb9f6\") " pod="openshift-ingress/router-default-c8fff6cd4-tr742"
Sep 11 23:05:43.729877 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:43.729663    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42864299-d484-4b6b-a2c7-dea808adb9f6-service-ca-bundle\") pod \"router-default-c8fff6cd4-tr742\" (UID: \"42864299-d484-4b6b-a2c7-dea808adb9f6\") " pod="openshift-ingress/router-default-c8fff6cd4-tr742"
Sep 11 23:05:43.729877 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:05:43.729759    2546 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 11 23:05:43.729877 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:05:43.729760    2546 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 11 23:05:43.729877 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:05:43.729833    2546 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 11 23:05:43.729877 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:05:43.729843    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdb719e6-5372-4e41-b059-d6cc84575f67-samples-operator-tls podName:fdb719e6-5372-4e41-b059-d6cc84575f67 nodeName:}" failed. No retries permitted until 2026-09-11 23:05:45.729820691 +0000 UTC m=+269.762949886 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/fdb719e6-5372-4e41-b059-d6cc84575f67-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-qtf5g" (UID: "fdb719e6-5372-4e41-b059-d6cc84575f67") : secret "samples-operator-tls" not found
Sep 11 23:05:43.729877 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:43.729772    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a6dd2771-bd9e-4e48-976e-d2f4c14eba11-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-h6w6f\" (UID: \"a6dd2771-bd9e-4e48-976e-d2f4c14eba11\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-h6w6f"
Sep 11 23:05:43.729877 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:05:43.729863    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42864299-d484-4b6b-a2c7-dea808adb9f6-metrics-certs podName:42864299-d484-4b6b-a2c7-dea808adb9f6 nodeName:}" failed. No retries permitted until 2026-09-11 23:05:45.729853517 +0000 UTC m=+269.762982718 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/42864299-d484-4b6b-a2c7-dea808adb9f6-metrics-certs") pod "router-default-c8fff6cd4-tr742" (UID: "42864299-d484-4b6b-a2c7-dea808adb9f6") : secret "router-metrics-certs-default" not found
Sep 11 23:05:43.730194 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:05:43.729883    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6dd2771-bd9e-4e48-976e-d2f4c14eba11-cluster-monitoring-operator-tls podName:a6dd2771-bd9e-4e48-976e-d2f4c14eba11 nodeName:}" failed. No retries permitted until 2026-09-11 23:05:45.729875161 +0000 UTC m=+269.763004357 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a6dd2771-bd9e-4e48-976e-d2f4c14eba11-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-h6w6f" (UID: "a6dd2771-bd9e-4e48-976e-d2f4c14eba11") : secret "cluster-monitoring-operator-tls" not found
Sep 11 23:05:43.730194 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:05:43.729905    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/42864299-d484-4b6b-a2c7-dea808adb9f6-service-ca-bundle podName:42864299-d484-4b6b-a2c7-dea808adb9f6 nodeName:}" failed. No retries permitted until 2026-09-11 23:05:45.72989539 +0000 UTC m=+269.763024590 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/42864299-d484-4b6b-a2c7-dea808adb9f6-service-ca-bundle") pod "router-default-c8fff6cd4-tr742" (UID: "42864299-d484-4b6b-a2c7-dea808adb9f6") : configmap references non-existent config key: service-ca.crt
Sep 11 23:05:44.293717 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:44.293567    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-5tht9" event={"ID":"8b564021-1e62-4eac-a86f-7a3a92980aa3","Type":"ContainerStarted","Data":"8ca14b91c0239f61dbb8af1135f02cbda8e31f91bb8575e8ac198abca201ea06"}
Sep 11 23:05:44.313736 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:44.313617    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-5tht9" podStartSLOduration=2.109009638 podStartE2EDuration="3.313594353s" totalImagesPullingTime="1.204584715s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 23:05:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:05:44.312702511 +0000 UTC m=+268.345831741" watchObservedRunningTime="2026-09-11 23:05:44.313594353 +0000 UTC m=+268.346723575"
Sep 11 23:05:45.299468 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:45.299403    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-xr6z6" event={"ID":"68e4123a-25ea-4c58-a934-d44495740042","Type":"ContainerStarted","Data":"ec72a8e3f9470e5ad68816fc7bebf42c841018de3178c945f0492b503ead0b5d"}
Sep 11 23:05:45.301270 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:45.301245    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-8psvk" event={"ID":"23f6c775-d2ea-4c5f-8bd4-da89b88950c0","Type":"ContainerStarted","Data":"5aa67f351362f69b32e007d39408f9e1de1a63fa57784d7930e3e678fa227fbc"}
Sep 11 23:05:45.318595 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:45.318547    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-xr6z6" podStartSLOduration=0.908826832 podStartE2EDuration="3.318531266s" totalImagesPullingTime="2.409704434s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 23:05:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:05:45.316621798 +0000 UTC m=+269.349751014" watchObservedRunningTime="2026-09-11 23:05:45.318531266 +0000 UTC m=+269.351660482"
Sep 11 23:05:45.333387 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:45.333341    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-8psvk" podStartSLOduration=1.8109885399999999 podStartE2EDuration="4.333322026s" totalImagesPullingTime="2.522333486s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 23:05:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:05:45.332291965 +0000 UTC m=+269.365421181" watchObservedRunningTime="2026-09-11 23:05:45.333322026 +0000 UTC m=+269.366451244"
Sep 11 23:05:45.750340 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:45.750301    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a6dd2771-bd9e-4e48-976e-d2f4c14eba11-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-h6w6f\" (UID: \"a6dd2771-bd9e-4e48-976e-d2f4c14eba11\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-h6w6f"
Sep 11 23:05:45.750517 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:45.750356    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fdb719e6-5372-4e41-b059-d6cc84575f67-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-qtf5g\" (UID: \"fdb719e6-5372-4e41-b059-d6cc84575f67\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-qtf5g"
Sep 11 23:05:45.750517 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:05:45.750443    2546 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 11 23:05:45.750517 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:05:45.750466    2546 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 11 23:05:45.750517 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:45.750486    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42864299-d484-4b6b-a2c7-dea808adb9f6-metrics-certs\") pod \"router-default-c8fff6cd4-tr742\" (UID: \"42864299-d484-4b6b-a2c7-dea808adb9f6\") " pod="openshift-ingress/router-default-c8fff6cd4-tr742"
Sep 11 23:05:45.750517 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:05:45.750511    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdb719e6-5372-4e41-b059-d6cc84575f67-samples-operator-tls podName:fdb719e6-5372-4e41-b059-d6cc84575f67 nodeName:}" failed. No retries permitted until 2026-09-11 23:05:49.750496732 +0000 UTC m=+273.783625951 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/fdb719e6-5372-4e41-b059-d6cc84575f67-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-qtf5g" (UID: "fdb719e6-5372-4e41-b059-d6cc84575f67") : secret "samples-operator-tls" not found
Sep 11 23:05:45.750762 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:05:45.750538    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6dd2771-bd9e-4e48-976e-d2f4c14eba11-cluster-monitoring-operator-tls podName:a6dd2771-bd9e-4e48-976e-d2f4c14eba11 nodeName:}" failed. No retries permitted until 2026-09-11 23:05:49.750519331 +0000 UTC m=+273.783648533 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a6dd2771-bd9e-4e48-976e-d2f4c14eba11-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-h6w6f" (UID: "a6dd2771-bd9e-4e48-976e-d2f4c14eba11") : secret "cluster-monitoring-operator-tls" not found
Sep 11 23:05:45.750762 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:05:45.750572    2546 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 11 23:05:45.750762 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:45.750604    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42864299-d484-4b6b-a2c7-dea808adb9f6-service-ca-bundle\") pod \"router-default-c8fff6cd4-tr742\" (UID: \"42864299-d484-4b6b-a2c7-dea808adb9f6\") " pod="openshift-ingress/router-default-c8fff6cd4-tr742"
Sep 11 23:05:45.750762 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:05:45.750616    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42864299-d484-4b6b-a2c7-dea808adb9f6-metrics-certs podName:42864299-d484-4b6b-a2c7-dea808adb9f6 nodeName:}" failed. No retries permitted until 2026-09-11 23:05:49.750601292 +0000 UTC m=+273.783730489 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/42864299-d484-4b6b-a2c7-dea808adb9f6-metrics-certs") pod "router-default-c8fff6cd4-tr742" (UID: "42864299-d484-4b6b-a2c7-dea808adb9f6") : secret "router-metrics-certs-default" not found
Sep 11 23:05:45.750762 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:05:45.750675    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/42864299-d484-4b6b-a2c7-dea808adb9f6-service-ca-bundle podName:42864299-d484-4b6b-a2c7-dea808adb9f6 nodeName:}" failed. No retries permitted until 2026-09-11 23:05:49.750664219 +0000 UTC m=+273.783793451 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/42864299-d484-4b6b-a2c7-dea808adb9f6-service-ca-bundle") pod "router-default-c8fff6cd4-tr742" (UID: "42864299-d484-4b6b-a2c7-dea808adb9f6") : configmap references non-existent config key: service-ca.crt
Sep 11 23:05:46.476951 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:46.476911    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-t2g8v"]
Sep 11 23:05:46.480552 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:46.480527    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-t2g8v"
Sep 11 23:05:46.491174 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:46.491131    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 11 23:05:46.491609 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:05:46.491570    2546 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"insights-runtime-extractor-tls\" is forbidden: User \"system:node:ip-10-0-128-161.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-128-161.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" type="*v1.Secret"
Sep 11 23:05:46.492729 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:46.492712    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-thjf7\""
Sep 11 23:05:46.529318 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:46.529283    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-t2g8v"]
Sep 11 23:05:46.559653 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:46.559624    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b24f52c2-9cad-4c69-9b65-f6752ecf1afb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t2g8v\" (UID: \"b24f52c2-9cad-4c69-9b65-f6752ecf1afb\") " pod="openshift-insights/insights-runtime-extractor-t2g8v"
Sep 11 23:05:46.559810 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:46.559670    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggqjl\" (UniqueName: \"kubernetes.io/projected/b24f52c2-9cad-4c69-9b65-f6752ecf1afb-kube-api-access-ggqjl\") pod \"insights-runtime-extractor-t2g8v\" (UID: \"b24f52c2-9cad-4c69-9b65-f6752ecf1afb\") " pod="openshift-insights/insights-runtime-extractor-t2g8v"
Sep 11 23:05:46.559810 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:46.559694    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b24f52c2-9cad-4c69-9b65-f6752ecf1afb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t2g8v\" (UID: \"b24f52c2-9cad-4c69-9b65-f6752ecf1afb\") " pod="openshift-insights/insights-runtime-extractor-t2g8v"
Sep 11 23:05:46.559810 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:46.559806    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b24f52c2-9cad-4c69-9b65-f6752ecf1afb-data-volume\") pod \"insights-runtime-extractor-t2g8v\" (UID: \"b24f52c2-9cad-4c69-9b65-f6752ecf1afb\") " pod="openshift-insights/insights-runtime-extractor-t2g8v"
Sep 11 23:05:46.559931 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:46.559829    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b24f52c2-9cad-4c69-9b65-f6752ecf1afb-crio-socket\") pod \"insights-runtime-extractor-t2g8v\" (UID: \"b24f52c2-9cad-4c69-9b65-f6752ecf1afb\") " pod="openshift-insights/insights-runtime-extractor-t2g8v"
Sep 11 23:05:46.660989 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:46.660949    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b24f52c2-9cad-4c69-9b65-f6752ecf1afb-data-volume\") pod \"insights-runtime-extractor-t2g8v\" (UID: \"b24f52c2-9cad-4c69-9b65-f6752ecf1afb\") " pod="openshift-insights/insights-runtime-extractor-t2g8v"
Sep 11 23:05:46.661211 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:46.660997    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b24f52c2-9cad-4c69-9b65-f6752ecf1afb-crio-socket\") pod \"insights-runtime-extractor-t2g8v\" (UID: \"b24f52c2-9cad-4c69-9b65-f6752ecf1afb\") " pod="openshift-insights/insights-runtime-extractor-t2g8v"
Sep 11 23:05:46.661211 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:46.661085    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b24f52c2-9cad-4c69-9b65-f6752ecf1afb-crio-socket\") pod \"insights-runtime-extractor-t2g8v\" (UID: \"b24f52c2-9cad-4c69-9b65-f6752ecf1afb\") " pod="openshift-insights/insights-runtime-extractor-t2g8v"
Sep 11 23:05:46.661211 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:46.661141    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b24f52c2-9cad-4c69-9b65-f6752ecf1afb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t2g8v\" (UID: \"b24f52c2-9cad-4c69-9b65-f6752ecf1afb\") " pod="openshift-insights/insights-runtime-extractor-t2g8v"
Sep 11 23:05:46.661390 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:46.661218    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggqjl\" (UniqueName: \"kubernetes.io/projected/b24f52c2-9cad-4c69-9b65-f6752ecf1afb-kube-api-access-ggqjl\") pod \"insights-runtime-extractor-t2g8v\" (UID: \"b24f52c2-9cad-4c69-9b65-f6752ecf1afb\") " pod="openshift-insights/insights-runtime-extractor-t2g8v"
Sep 11 23:05:46.661390 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:46.661242    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b24f52c2-9cad-4c69-9b65-f6752ecf1afb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t2g8v\" (UID: \"b24f52c2-9cad-4c69-9b65-f6752ecf1afb\") " pod="openshift-insights/insights-runtime-extractor-t2g8v"
Sep 11 23:05:46.661390 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:05:46.661359    2546 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 23:05:46.661512 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:46.661389    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b24f52c2-9cad-4c69-9b65-f6752ecf1afb-data-volume\") pod \"insights-runtime-extractor-t2g8v\" (UID: \"b24f52c2-9cad-4c69-9b65-f6752ecf1afb\") " pod="openshift-insights/insights-runtime-extractor-t2g8v"
Sep 11 23:05:46.661512 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:05:46.661435    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b24f52c2-9cad-4c69-9b65-f6752ecf1afb-kube-rbac-proxy-cm podName:b24f52c2-9cad-4c69-9b65-f6752ecf1afb nodeName:}" failed. No retries permitted until 2026-09-11 23:05:47.161414544 +0000 UTC m=+271.194543754 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b24f52c2-9cad-4c69-9b65-f6752ecf1afb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-t2g8v" (UID: "b24f52c2-9cad-4c69-9b65-f6752ecf1afb") : configmap "kube-rbac-proxy" not found
Sep 11 23:05:46.678881 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:46.678851    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggqjl\" (UniqueName: \"kubernetes.io/projected/b24f52c2-9cad-4c69-9b65-f6752ecf1afb-kube-api-access-ggqjl\") pod \"insights-runtime-extractor-t2g8v\" (UID: \"b24f52c2-9cad-4c69-9b65-f6752ecf1afb\") " pod="openshift-insights/insights-runtime-extractor-t2g8v"
Sep 11 23:05:47.166710 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:47.166641    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b24f52c2-9cad-4c69-9b65-f6752ecf1afb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t2g8v\" (UID: \"b24f52c2-9cad-4c69-9b65-f6752ecf1afb\") " pod="openshift-insights/insights-runtime-extractor-t2g8v"
Sep 11 23:05:47.166893 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:05:47.166775    2546 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 23:05:47.166893 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:05:47.166840    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b24f52c2-9cad-4c69-9b65-f6752ecf1afb-kube-rbac-proxy-cm podName:b24f52c2-9cad-4c69-9b65-f6752ecf1afb nodeName:}" failed. No retries permitted until 2026-09-11 23:05:48.166823472 +0000 UTC m=+272.199952668 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b24f52c2-9cad-4c69-9b65-f6752ecf1afb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-t2g8v" (UID: "b24f52c2-9cad-4c69-9b65-f6752ecf1afb") : configmap "kube-rbac-proxy" not found
Sep 11 23:05:47.661764 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:05:47.661727    2546 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: failed to sync secret cache: timed out waiting for the condition
Sep 11 23:05:47.662235 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:05:47.661870    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b24f52c2-9cad-4c69-9b65-f6752ecf1afb-insights-runtime-extractor-tls podName:b24f52c2-9cad-4c69-9b65-f6752ecf1afb nodeName:}" failed. No retries permitted until 2026-09-11 23:05:48.161847651 +0000 UTC m=+272.194976852 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b24f52c2-9cad-4c69-9b65-f6752ecf1afb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-t2g8v" (UID: "b24f52c2-9cad-4c69-9b65-f6752ecf1afb") : failed to sync secret cache: timed out waiting for the condition
Sep 11 23:05:47.797671 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:47.797636    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 11 23:05:47.863071 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:47.863034    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-4tdps"]
Sep 11 23:05:47.866362 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:47.866345    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-4tdps"
Sep 11 23:05:47.869282 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:47.869241    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 11 23:05:47.869412 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:47.869340    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 11 23:05:47.870488 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:47.870468    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-s4ddv\""
Sep 11 23:05:47.875853 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:47.875833    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-4tdps"]
Sep 11 23:05:47.973730 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:47.973641    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frljj\" (UniqueName: \"kubernetes.io/projected/51cae370-f275-4cc9-8c5e-5f5e6998b6a1-kube-api-access-frljj\") pod \"migrator-8b4d697c5-4tdps\" (UID: \"51cae370-f275-4cc9-8c5e-5f5e6998b6a1\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-4tdps"
Sep 11 23:05:48.074294 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:48.074236    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frljj\" (UniqueName: \"kubernetes.io/projected/51cae370-f275-4cc9-8c5e-5f5e6998b6a1-kube-api-access-frljj\") pod \"migrator-8b4d697c5-4tdps\" (UID: \"51cae370-f275-4cc9-8c5e-5f5e6998b6a1\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-4tdps"
Sep 11 23:05:48.083492 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:48.083458    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-frljj\" (UniqueName: \"kubernetes.io/projected/51cae370-f275-4cc9-8c5e-5f5e6998b6a1-kube-api-access-frljj\") pod \"migrator-8b4d697c5-4tdps\" (UID: \"51cae370-f275-4cc9-8c5e-5f5e6998b6a1\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-4tdps"
Sep 11 23:05:48.174996 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:48.174960    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-4tdps"
Sep 11 23:05:48.175254 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:48.175228    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b24f52c2-9cad-4c69-9b65-f6752ecf1afb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t2g8v\" (UID: \"b24f52c2-9cad-4c69-9b65-f6752ecf1afb\") " pod="openshift-insights/insights-runtime-extractor-t2g8v"
Sep 11 23:05:48.175302 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:48.175278    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b24f52c2-9cad-4c69-9b65-f6752ecf1afb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t2g8v\" (UID: \"b24f52c2-9cad-4c69-9b65-f6752ecf1afb\") " pod="openshift-insights/insights-runtime-extractor-t2g8v"
Sep 11 23:05:48.175361 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:05:48.175349    2546 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 23:05:48.175394 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:05:48.175373    2546 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 23:05:48.175426 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:05:48.175402    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b24f52c2-9cad-4c69-9b65-f6752ecf1afb-kube-rbac-proxy-cm podName:b24f52c2-9cad-4c69-9b65-f6752ecf1afb nodeName:}" failed. No retries permitted until 2026-09-11 23:05:50.175388931 +0000 UTC m=+274.208518139 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b24f52c2-9cad-4c69-9b65-f6752ecf1afb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-t2g8v" (UID: "b24f52c2-9cad-4c69-9b65-f6752ecf1afb") : configmap "kube-rbac-proxy" not found
Sep 11 23:05:48.175467 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:05:48.175423    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b24f52c2-9cad-4c69-9b65-f6752ecf1afb-insights-runtime-extractor-tls podName:b24f52c2-9cad-4c69-9b65-f6752ecf1afb nodeName:}" failed. No retries permitted until 2026-09-11 23:05:49.175410101 +0000 UTC m=+273.208539296 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b24f52c2-9cad-4c69-9b65-f6752ecf1afb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-t2g8v" (UID: "b24f52c2-9cad-4c69-9b65-f6752ecf1afb") : secret "insights-runtime-extractor-tls" not found
Sep 11 23:05:48.445941 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:48.445910    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-4tdps"]
Sep 11 23:05:48.448825 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:05:48.448800    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51cae370_f275_4cc9_8c5e_5f5e6998b6a1.slice/crio-084927fa2ddbebdb6e0fc31b482add16048edcf701c55cdd8717f513a85b1aff WatchSource:0}: Error finding container 084927fa2ddbebdb6e0fc31b482add16048edcf701c55cdd8717f513a85b1aff: Status 404 returned error can't find the container with id 084927fa2ddbebdb6e0fc31b482add16048edcf701c55cdd8717f513a85b1aff
Sep 11 23:05:48.603764 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:48.603684    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vqs5t_9dcaac8a-b529-4ba2-a3f3-a617817741c2/dns-node-resolver/0.log"
Sep 11 23:05:49.184733 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:49.184684    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b24f52c2-9cad-4c69-9b65-f6752ecf1afb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t2g8v\" (UID: \"b24f52c2-9cad-4c69-9b65-f6752ecf1afb\") " pod="openshift-insights/insights-runtime-extractor-t2g8v"
Sep 11 23:05:49.185242 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:05:49.184859    2546 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 23:05:49.185242 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:05:49.184951    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b24f52c2-9cad-4c69-9b65-f6752ecf1afb-insights-runtime-extractor-tls podName:b24f52c2-9cad-4c69-9b65-f6752ecf1afb nodeName:}" failed. No retries permitted until 2026-09-11 23:05:51.184928216 +0000 UTC m=+275.218057433 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b24f52c2-9cad-4c69-9b65-f6752ecf1afb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-t2g8v" (UID: "b24f52c2-9cad-4c69-9b65-f6752ecf1afb") : secret "insights-runtime-extractor-tls" not found
Sep 11 23:05:49.315765 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:49.315728    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-4tdps" event={"ID":"51cae370-f275-4cc9-8c5e-5f5e6998b6a1","Type":"ContainerStarted","Data":"084927fa2ddbebdb6e0fc31b482add16048edcf701c55cdd8717f513a85b1aff"}
Sep 11 23:05:49.610727 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:49.610644    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zgtd5_aa21ad55-513d-46a1-89ce-78e88c3e0efa/node-ca/0.log"
Sep 11 23:05:49.791839 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:49.791808    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a6dd2771-bd9e-4e48-976e-d2f4c14eba11-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-h6w6f\" (UID: \"a6dd2771-bd9e-4e48-976e-d2f4c14eba11\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-h6w6f"
Sep 11 23:05:49.791935 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:49.791894    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fdb719e6-5372-4e41-b059-d6cc84575f67-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-qtf5g\" (UID: \"fdb719e6-5372-4e41-b059-d6cc84575f67\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-qtf5g"
Sep 11 23:05:49.791975 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:49.791933    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42864299-d484-4b6b-a2c7-dea808adb9f6-metrics-certs\") pod \"router-default-c8fff6cd4-tr742\" (UID: \"42864299-d484-4b6b-a2c7-dea808adb9f6\") " pod="openshift-ingress/router-default-c8fff6cd4-tr742"
Sep 11 23:05:49.792018 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:05:49.791937    2546 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 11 23:05:49.792069 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:05:49.792050    2546 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 11 23:05:49.792103 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:05:49.791986    2546 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 11 23:05:49.792103 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:05:49.792073    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6dd2771-bd9e-4e48-976e-d2f4c14eba11-cluster-monitoring-operator-tls podName:a6dd2771-bd9e-4e48-976e-d2f4c14eba11 nodeName:}" failed. No retries permitted until 2026-09-11 23:05:57.792052584 +0000 UTC m=+281.825181781 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a6dd2771-bd9e-4e48-976e-d2f4c14eba11-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-h6w6f" (UID: "a6dd2771-bd9e-4e48-976e-d2f4c14eba11") : secret "cluster-monitoring-operator-tls" not found
Sep 11 23:05:49.792103 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:49.792001    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42864299-d484-4b6b-a2c7-dea808adb9f6-service-ca-bundle\") pod \"router-default-c8fff6cd4-tr742\" (UID: \"42864299-d484-4b6b-a2c7-dea808adb9f6\") " pod="openshift-ingress/router-default-c8fff6cd4-tr742"
Sep 11 23:05:49.792341 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:05:49.792109    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/42864299-d484-4b6b-a2c7-dea808adb9f6-service-ca-bundle podName:42864299-d484-4b6b-a2c7-dea808adb9f6 nodeName:}" failed. No retries permitted until 2026-09-11 23:05:57.792093463 +0000 UTC m=+281.825222661 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/42864299-d484-4b6b-a2c7-dea808adb9f6-service-ca-bundle") pod "router-default-c8fff6cd4-tr742" (UID: "42864299-d484-4b6b-a2c7-dea808adb9f6") : configmap references non-existent config key: service-ca.crt
Sep 11 23:05:49.792341 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:05:49.792132    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42864299-d484-4b6b-a2c7-dea808adb9f6-metrics-certs podName:42864299-d484-4b6b-a2c7-dea808adb9f6 nodeName:}" failed. No retries permitted until 2026-09-11 23:05:57.792123137 +0000 UTC m=+281.825252334 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/42864299-d484-4b6b-a2c7-dea808adb9f6-metrics-certs") pod "router-default-c8fff6cd4-tr742" (UID: "42864299-d484-4b6b-a2c7-dea808adb9f6") : secret "router-metrics-certs-default" not found
Sep 11 23:05:49.792341 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:05:49.792202    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdb719e6-5372-4e41-b059-d6cc84575f67-samples-operator-tls podName:fdb719e6-5372-4e41-b059-d6cc84575f67 nodeName:}" failed. No retries permitted until 2026-09-11 23:05:57.792175815 +0000 UTC m=+281.825305014 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/fdb719e6-5372-4e41-b059-d6cc84575f67-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-qtf5g" (UID: "fdb719e6-5372-4e41-b059-d6cc84575f67") : secret "samples-operator-tls" not found
Sep 11 23:05:50.004691 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:50.004666    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-glt22_efcb43e4-c096-422f-bc0b-2c42795b0adc/serve-healthcheck-canary/0.log"
Sep 11 23:05:50.195676 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:50.195638    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b24f52c2-9cad-4c69-9b65-f6752ecf1afb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t2g8v\" (UID: \"b24f52c2-9cad-4c69-9b65-f6752ecf1afb\") " pod="openshift-insights/insights-runtime-extractor-t2g8v"
Sep 11 23:05:50.196188 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:05:50.195781    2546 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 23:05:50.196188 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:05:50.195867    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b24f52c2-9cad-4c69-9b65-f6752ecf1afb-kube-rbac-proxy-cm podName:b24f52c2-9cad-4c69-9b65-f6752ecf1afb nodeName:}" failed. No retries permitted until 2026-09-11 23:05:54.195845623 +0000 UTC m=+278.228974821 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b24f52c2-9cad-4c69-9b65-f6752ecf1afb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-t2g8v" (UID: "b24f52c2-9cad-4c69-9b65-f6752ecf1afb") : configmap "kube-rbac-proxy" not found
Sep 11 23:05:50.320977 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:50.320886    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-4tdps" event={"ID":"51cae370-f275-4cc9-8c5e-5f5e6998b6a1","Type":"ContainerStarted","Data":"d80762eb38fccc1a5ccfb1559c6479603ce66d831a760e02faccc79b32027a5e"}
Sep 11 23:05:50.320977 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:50.320943    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-4tdps" event={"ID":"51cae370-f275-4cc9-8c5e-5f5e6998b6a1","Type":"ContainerStarted","Data":"c2f72244c10f3b0f49a5248a462a5b21c0be88df808303fa136d24dbf21dd985"}
Sep 11 23:05:50.339146 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:50.339088    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-4tdps" podStartSLOduration=2.094197202 podStartE2EDuration="3.339073226s" totalImagesPullingTime="1.244876024s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 23:05:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:05:50.337354455 +0000 UTC m=+274.370483671" watchObservedRunningTime="2026-09-11 23:05:50.339073226 +0000 UTC m=+274.372202443"
Sep 11 23:05:51.204466 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:51.204427    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b24f52c2-9cad-4c69-9b65-f6752ecf1afb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t2g8v\" (UID: \"b24f52c2-9cad-4c69-9b65-f6752ecf1afb\") " pod="openshift-insights/insights-runtime-extractor-t2g8v"
Sep 11 23:05:51.204847 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:05:51.204587    2546 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 23:05:51.204847 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:05:51.204653    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b24f52c2-9cad-4c69-9b65-f6752ecf1afb-insights-runtime-extractor-tls podName:b24f52c2-9cad-4c69-9b65-f6752ecf1afb nodeName:}" failed. No retries permitted until 2026-09-11 23:05:55.204637385 +0000 UTC m=+279.237766601 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b24f52c2-9cad-4c69-9b65-f6752ecf1afb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-t2g8v" (UID: "b24f52c2-9cad-4c69-9b65-f6752ecf1afb") : secret "insights-runtime-extractor-tls" not found
Sep 11 23:05:54.231842 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:54.231802    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b24f52c2-9cad-4c69-9b65-f6752ecf1afb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t2g8v\" (UID: \"b24f52c2-9cad-4c69-9b65-f6752ecf1afb\") " pod="openshift-insights/insights-runtime-extractor-t2g8v"
Sep 11 23:05:54.232252 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:05:54.231925    2546 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 23:05:54.232252 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:05:54.231993    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b24f52c2-9cad-4c69-9b65-f6752ecf1afb-kube-rbac-proxy-cm podName:b24f52c2-9cad-4c69-9b65-f6752ecf1afb nodeName:}" failed. No retries permitted until 2026-09-11 23:06:02.231977467 +0000 UTC m=+286.265106681 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b24f52c2-9cad-4c69-9b65-f6752ecf1afb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-t2g8v" (UID: "b24f52c2-9cad-4c69-9b65-f6752ecf1afb") : configmap "kube-rbac-proxy" not found
Sep 11 23:05:55.240360 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:55.240311    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b24f52c2-9cad-4c69-9b65-f6752ecf1afb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t2g8v\" (UID: \"b24f52c2-9cad-4c69-9b65-f6752ecf1afb\") " pod="openshift-insights/insights-runtime-extractor-t2g8v"
Sep 11 23:05:55.240846 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:05:55.240498    2546 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 23:05:55.240846 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:05:55.240582    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b24f52c2-9cad-4c69-9b65-f6752ecf1afb-insights-runtime-extractor-tls podName:b24f52c2-9cad-4c69-9b65-f6752ecf1afb nodeName:}" failed. No retries permitted until 2026-09-11 23:06:03.240560414 +0000 UTC m=+287.273689612 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b24f52c2-9cad-4c69-9b65-f6752ecf1afb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-t2g8v" (UID: "b24f52c2-9cad-4c69-9b65-f6752ecf1afb") : secret "insights-runtime-extractor-tls" not found
Sep 11 23:05:57.864072 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:57.864038    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42864299-d484-4b6b-a2c7-dea808adb9f6-service-ca-bundle\") pod \"router-default-c8fff6cd4-tr742\" (UID: \"42864299-d484-4b6b-a2c7-dea808adb9f6\") " pod="openshift-ingress/router-default-c8fff6cd4-tr742"
Sep 11 23:05:57.864534 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:57.864142    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a6dd2771-bd9e-4e48-976e-d2f4c14eba11-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-h6w6f\" (UID: \"a6dd2771-bd9e-4e48-976e-d2f4c14eba11\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-h6w6f"
Sep 11 23:05:57.864534 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:57.864202    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fdb719e6-5372-4e41-b059-d6cc84575f67-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-qtf5g\" (UID: \"fdb719e6-5372-4e41-b059-d6cc84575f67\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-qtf5g"
Sep 11 23:05:57.864534 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:05:57.864225    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/42864299-d484-4b6b-a2c7-dea808adb9f6-service-ca-bundle podName:42864299-d484-4b6b-a2c7-dea808adb9f6 nodeName:}" failed. No retries permitted until 2026-09-11 23:06:13.864208389 +0000 UTC m=+297.897337584 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/42864299-d484-4b6b-a2c7-dea808adb9f6-service-ca-bundle") pod "router-default-c8fff6cd4-tr742" (UID: "42864299-d484-4b6b-a2c7-dea808adb9f6") : configmap references non-existent config key: service-ca.crt
Sep 11 23:05:57.864534 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:57.864267    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42864299-d484-4b6b-a2c7-dea808adb9f6-metrics-certs\") pod \"router-default-c8fff6cd4-tr742\" (UID: \"42864299-d484-4b6b-a2c7-dea808adb9f6\") " pod="openshift-ingress/router-default-c8fff6cd4-tr742"
Sep 11 23:05:57.864534 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:05:57.864302    2546 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 11 23:05:57.864534 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:05:57.864363    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6dd2771-bd9e-4e48-976e-d2f4c14eba11-cluster-monitoring-operator-tls podName:a6dd2771-bd9e-4e48-976e-d2f4c14eba11 nodeName:}" failed. No retries permitted until 2026-09-11 23:06:13.864349182 +0000 UTC m=+297.897478405 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a6dd2771-bd9e-4e48-976e-d2f4c14eba11-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-h6w6f" (UID: "a6dd2771-bd9e-4e48-976e-d2f4c14eba11") : secret "cluster-monitoring-operator-tls" not found
Sep 11 23:05:57.866725 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:57.866687    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42864299-d484-4b6b-a2c7-dea808adb9f6-metrics-certs\") pod \"router-default-c8fff6cd4-tr742\" (UID: \"42864299-d484-4b6b-a2c7-dea808adb9f6\") " pod="openshift-ingress/router-default-c8fff6cd4-tr742"
Sep 11 23:05:57.866844 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:57.866827    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fdb719e6-5372-4e41-b059-d6cc84575f67-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-qtf5g\" (UID: \"fdb719e6-5372-4e41-b059-d6cc84575f67\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-qtf5g"
Sep 11 23:05:57.999636 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:57.999601    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-qtf5g"
Sep 11 23:05:58.013679 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:05:58.013649    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-5c8f46447c-cdpct" podUID="8ea4ca4e-b321-4324-b380-dab72513b02a"
Sep 11 23:05:58.013779 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:05:58.013671    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-7q8hx" podUID="9783f5ec-7c57-4a7a-a688-3afad6cda55d"
Sep 11 23:05:58.013779 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:05:58.013688    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-fzfcr" podUID="98962ebe-ac82-4d92-a979-022ad4873bc9"
Sep 11 23:05:58.268364 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:58.268326    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-qtf5g"]
Sep 11 23:05:58.348124 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:58.348099    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fzfcr"
Sep 11 23:05:58.348326 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:58.348124    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-qtf5g" event={"ID":"fdb719e6-5372-4e41-b059-d6cc84575f67","Type":"ContainerStarted","Data":"a1a83306a2a059b25e8b24da272fa9c4fb47ed1923e63ef7c2b720f73c64690a"}
Sep 11 23:05:58.348326 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:58.348172    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c8f46447c-cdpct"
Sep 11 23:05:58.348326 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:05:58.348190    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-7q8hx"
Sep 11 23:06:01.195561 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:01.195521    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9783f5ec-7c57-4a7a-a688-3afad6cda55d-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-7q8hx\" (UID: \"9783f5ec-7c57-4a7a-a688-3afad6cda55d\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-7q8hx"
Sep 11 23:06:01.195967 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:01.195574    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8ea4ca4e-b321-4324-b380-dab72513b02a-registry-tls\") pod \"image-registry-5c8f46447c-cdpct\" (UID: \"8ea4ca4e-b321-4324-b380-dab72513b02a\") " pod="openshift-image-registry/image-registry-5c8f46447c-cdpct"
Sep 11 23:06:01.198145 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:01.198116    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9783f5ec-7c57-4a7a-a688-3afad6cda55d-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-7q8hx\" (UID: \"9783f5ec-7c57-4a7a-a688-3afad6cda55d\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-7q8hx"
Sep 11 23:06:01.198298 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:01.198277    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8ea4ca4e-b321-4324-b380-dab72513b02a-registry-tls\") pod \"image-registry-5c8f46447c-cdpct\" (UID: \"8ea4ca4e-b321-4324-b380-dab72513b02a\") " pod="openshift-image-registry/image-registry-5c8f46447c-cdpct"
Sep 11 23:06:01.296164 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:01.296113    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/98962ebe-ac82-4d92-a979-022ad4873bc9-metrics-tls\") pod \"dns-default-fzfcr\" (UID: \"98962ebe-ac82-4d92-a979-022ad4873bc9\") " pod="openshift-dns/dns-default-fzfcr"
Sep 11 23:06:01.298522 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:01.298502    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/98962ebe-ac82-4d92-a979-022ad4873bc9-metrics-tls\") pod \"dns-default-fzfcr\" (UID: \"98962ebe-ac82-4d92-a979-022ad4873bc9\") " pod="openshift-dns/dns-default-fzfcr"
Sep 11 23:06:01.355071 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:01.355039    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-jqw7w\""
Sep 11 23:06:01.355071 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:01.355060    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-x84tr\""
Sep 11 23:06:01.355313 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:01.355060    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-t54dv\""
Sep 11 23:06:01.358857 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:01.358839    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-7q8hx"
Sep 11 23:06:01.358988 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:01.358859    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fzfcr"
Sep 11 23:06:01.358988 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:01.358927    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c8f46447c-cdpct"
Sep 11 23:06:01.359335 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:01.359311    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-qtf5g" event={"ID":"fdb719e6-5372-4e41-b059-d6cc84575f67","Type":"ContainerStarted","Data":"e38ed380852b80696d414d61b7c61e522fe64d3ce5338d7e57617007fb06d0a4"}
Sep 11 23:06:01.359439 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:01.359358    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-qtf5g" event={"ID":"fdb719e6-5372-4e41-b059-d6cc84575f67","Type":"ContainerStarted","Data":"cbba1bdb52382cc3be06e6711cfdaac613fcf023be4743b44d75a5c4fc5b6930"}
Sep 11 23:06:01.379708 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:01.379663    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-qtf5g" podStartSLOduration=17.321245022 podStartE2EDuration="19.379647266s" totalImagesPullingTime="2.058402244s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 23:05:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:06:01.378183565 +0000 UTC m=+285.411312784" watchObservedRunningTime="2026-09-11 23:06:01.379647266 +0000 UTC m=+285.412776483"
Sep 11 23:06:01.645334 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:01.645302    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fzfcr"]
Sep 11 23:06:01.648301 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:06:01.648266    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98962ebe_ac82_4d92_a979_022ad4873bc9.slice/crio-ba50df3ba704fa41efa9bd055cc28fae39474578de406320fd25256c2be662cb WatchSource:0}: Error finding container ba50df3ba704fa41efa9bd055cc28fae39474578de406320fd25256c2be662cb: Status 404 returned error can't find the container with id ba50df3ba704fa41efa9bd055cc28fae39474578de406320fd25256c2be662cb
Sep 11 23:06:01.649572 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:01.649484    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-7q8hx"]
Sep 11 23:06:01.651564 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:06:01.651544    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9783f5ec_7c57_4a7a_a688_3afad6cda55d.slice/crio-3e0504a0c12182ccbdb5bbc69efbf55adedd102ad2b671ffe026c4cea21609b9 WatchSource:0}: Error finding container 3e0504a0c12182ccbdb5bbc69efbf55adedd102ad2b671ffe026c4cea21609b9: Status 404 returned error can't find the container with id 3e0504a0c12182ccbdb5bbc69efbf55adedd102ad2b671ffe026c4cea21609b9
Sep 11 23:06:01.657266 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:01.657241    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5c8f46447c-cdpct"]
Sep 11 23:06:01.661203 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:06:01.661181    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ea4ca4e_b321_4324_b380_dab72513b02a.slice/crio-6bdbafe025fa1d53e573219b37dff8b9ad6a6c594f84d23aa8bb6610ef8a9188 WatchSource:0}: Error finding container 6bdbafe025fa1d53e573219b37dff8b9ad6a6c594f84d23aa8bb6610ef8a9188: Status 404 returned error can't find the container with id 6bdbafe025fa1d53e573219b37dff8b9ad6a6c594f84d23aa8bb6610ef8a9188
Sep 11 23:06:02.302928 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:02.302892    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b24f52c2-9cad-4c69-9b65-f6752ecf1afb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t2g8v\" (UID: \"b24f52c2-9cad-4c69-9b65-f6752ecf1afb\") " pod="openshift-insights/insights-runtime-extractor-t2g8v"
Sep 11 23:06:02.303571 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:02.303545    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b24f52c2-9cad-4c69-9b65-f6752ecf1afb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t2g8v\" (UID: \"b24f52c2-9cad-4c69-9b65-f6752ecf1afb\") " pod="openshift-insights/insights-runtime-extractor-t2g8v"
Sep 11 23:06:02.364296 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:02.364261    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5c8f46447c-cdpct" event={"ID":"8ea4ca4e-b321-4324-b380-dab72513b02a","Type":"ContainerStarted","Data":"9cda1c095880367248025a50d1024e81d0178ea993df2dbaa2ba237c64a36890"}
Sep 11 23:06:02.364465 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:02.364312    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5c8f46447c-cdpct" event={"ID":"8ea4ca4e-b321-4324-b380-dab72513b02a","Type":"ContainerStarted","Data":"6bdbafe025fa1d53e573219b37dff8b9ad6a6c594f84d23aa8bb6610ef8a9188"}
Sep 11 23:06:02.364465 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:02.364390    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5c8f46447c-cdpct"
Sep 11 23:06:02.365486 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:02.365464    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fzfcr" event={"ID":"98962ebe-ac82-4d92-a979-022ad4873bc9","Type":"ContainerStarted","Data":"ba50df3ba704fa41efa9bd055cc28fae39474578de406320fd25256c2be662cb"}
Sep 11 23:06:02.366550 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:02.366523    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-7q8hx" event={"ID":"9783f5ec-7c57-4a7a-a688-3afad6cda55d","Type":"ContainerStarted","Data":"3e0504a0c12182ccbdb5bbc69efbf55adedd102ad2b671ffe026c4cea21609b9"}
Sep 11 23:06:02.388688 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:02.388646    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5c8f46447c-cdpct" podStartSLOduration=263.38863358 podStartE2EDuration="4m23.38863358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 23:01:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:06:02.387246706 +0000 UTC m=+286.420375925" watchObservedRunningTime="2026-09-11 23:06:02.38863358 +0000 UTC m=+286.421762796"
Sep 11 23:06:03.311938 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:03.311891    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b24f52c2-9cad-4c69-9b65-f6752ecf1afb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t2g8v\" (UID: \"b24f52c2-9cad-4c69-9b65-f6752ecf1afb\") " pod="openshift-insights/insights-runtime-extractor-t2g8v"
Sep 11 23:06:03.314658 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:03.314619    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b24f52c2-9cad-4c69-9b65-f6752ecf1afb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t2g8v\" (UID: \"b24f52c2-9cad-4c69-9b65-f6752ecf1afb\") " pod="openshift-insights/insights-runtime-extractor-t2g8v"
Sep 11 23:06:03.371642 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:03.371551    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-7q8hx" event={"ID":"9783f5ec-7c57-4a7a-a688-3afad6cda55d","Type":"ContainerStarted","Data":"4dcc2641fe541445394fdf0081b019ec3ad67ad0bc69d77242072f8dfe4b2105"}
Sep 11 23:06:03.388245 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:03.388194    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-7q8hx" podStartSLOduration=284.965524344 podStartE2EDuration="4m46.388179365s" totalImagesPullingTime="1.422655021s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 23:01:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:06:03.387346343 +0000 UTC m=+287.420475561" watchObservedRunningTime="2026-09-11 23:06:03.388179365 +0000 UTC m=+287.421308582"
Sep 11 23:06:03.588770 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:03.588733    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-t2g8v"
Sep 11 23:06:03.885399 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:03.885376    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-t2g8v"]
Sep 11 23:06:03.887954 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:06:03.887929    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb24f52c2_9cad_4c69_9b65_f6752ecf1afb.slice/crio-46bcb55894a4ed298d51aa9cd000ee2fa8a8ce1a92e78ba37ad563ffad7809d0 WatchSource:0}: Error finding container 46bcb55894a4ed298d51aa9cd000ee2fa8a8ce1a92e78ba37ad563ffad7809d0: Status 404 returned error can't find the container with id 46bcb55894a4ed298d51aa9cd000ee2fa8a8ce1a92e78ba37ad563ffad7809d0
Sep 11 23:06:04.377148 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:04.377059    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fzfcr" event={"ID":"98962ebe-ac82-4d92-a979-022ad4873bc9","Type":"ContainerStarted","Data":"184e8f351b3150069f1b1847cb61e10a1cd605e74ca70437a6dada0bd8128929"}
Sep 11 23:06:04.377148 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:04.377113    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fzfcr" event={"ID":"98962ebe-ac82-4d92-a979-022ad4873bc9","Type":"ContainerStarted","Data":"131e87634e102835bb3b61509acc646d47555ebbd7ba7c0c6007e431dc0acda0"}
Sep 11 23:06:04.377148 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:04.377143    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-fzfcr"
Sep 11 23:06:04.378531 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:04.378509    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-t2g8v" event={"ID":"b24f52c2-9cad-4c69-9b65-f6752ecf1afb","Type":"ContainerStarted","Data":"df4289bb3c23081c5dcd076dcfb4a48793159ea7e413d3cf6e40959bd4e13f1e"}
Sep 11 23:06:04.378596 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:04.378556    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-t2g8v" event={"ID":"b24f52c2-9cad-4c69-9b65-f6752ecf1afb","Type":"ContainerStarted","Data":"46bcb55894a4ed298d51aa9cd000ee2fa8a8ce1a92e78ba37ad563ffad7809d0"}
Sep 11 23:06:04.395707 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:04.395663    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-fzfcr" podStartSLOduration=251.38008293 podStartE2EDuration="4m13.39564987s" totalImagesPullingTime="2.01556694s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 23:01:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:06:04.394643712 +0000 UTC m=+288.427772928" watchObservedRunningTime="2026-09-11 23:06:04.39564987 +0000 UTC m=+288.428779086"
Sep 11 23:06:05.384534 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:05.384497    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-t2g8v" event={"ID":"b24f52c2-9cad-4c69-9b65-f6752ecf1afb","Type":"ContainerStarted","Data":"11a5637aeef73ac898680d995369086fe20fafe83c64ca79bb2ea76efd57b43f"}
Sep 11 23:06:06.390295 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:06.390206    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-t2g8v" event={"ID":"b24f52c2-9cad-4c69-9b65-f6752ecf1afb","Type":"ContainerStarted","Data":"16b3cea666a9a2358d5c9abd9716162ded4c11e04aa8a18b0c90d610cdc5574d"}
Sep 11 23:06:06.410306 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:06.410232    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-t2g8v" podStartSLOduration=18.340919998 podStartE2EDuration="20.410212013s" totalImagesPullingTime="2.069292015s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 23:05:46 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:06:06.407788285 +0000 UTC m=+290.440917503" watchObservedRunningTime="2026-09-11 23:06:06.410212013 +0000 UTC m=+290.443341231"
Sep 11 23:06:10.489123 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:06:10.489078    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-vbgsz" podUID="50607ccd-47f9-463a-bdc7-181484a00caa"
Sep 11 23:06:13.898082 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:13.898025    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a6dd2771-bd9e-4e48-976e-d2f4c14eba11-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-h6w6f\" (UID: \"a6dd2771-bd9e-4e48-976e-d2f4c14eba11\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-h6w6f"
Sep 11 23:06:13.898483 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:13.898100    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42864299-d484-4b6b-a2c7-dea808adb9f6-service-ca-bundle\") pod \"router-default-c8fff6cd4-tr742\" (UID: \"42864299-d484-4b6b-a2c7-dea808adb9f6\") " pod="openshift-ingress/router-default-c8fff6cd4-tr742"
Sep 11 23:06:13.898693 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:13.898675    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42864299-d484-4b6b-a2c7-dea808adb9f6-service-ca-bundle\") pod \"router-default-c8fff6cd4-tr742\" (UID: \"42864299-d484-4b6b-a2c7-dea808adb9f6\") " pod="openshift-ingress/router-default-c8fff6cd4-tr742"
Sep 11 23:06:13.900665 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:13.900640    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a6dd2771-bd9e-4e48-976e-d2f4c14eba11-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-h6w6f\" (UID: \"a6dd2771-bd9e-4e48-976e-d2f4c14eba11\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-h6w6f"
Sep 11 23:06:13.907416 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:13.907395    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-h6w6f"
Sep 11 23:06:13.924453 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:13.924434    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-c8fff6cd4-tr742"
Sep 11 23:06:14.181930 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:14.181909    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-h6w6f"]
Sep 11 23:06:14.184236 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:06:14.184207    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6dd2771_bd9e_4e48_976e_d2f4c14eba11.slice/crio-df40e56770afd652d1562e6121fb1452873de62690d26d9f005916673145ddd6 WatchSource:0}: Error finding container df40e56770afd652d1562e6121fb1452873de62690d26d9f005916673145ddd6: Status 404 returned error can't find the container with id df40e56770afd652d1562e6121fb1452873de62690d26d9f005916673145ddd6
Sep 11 23:06:14.207315 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:14.207291    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-c8fff6cd4-tr742"]
Sep 11 23:06:14.209409 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:06:14.209385    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42864299_d484_4b6b_a2c7_dea808adb9f6.slice/crio-3aa848320541fedef0ce6c7fbc995b9331bed6533c10d004bbca25301d2fcef0 WatchSource:0}: Error finding container 3aa848320541fedef0ce6c7fbc995b9331bed6533c10d004bbca25301d2fcef0: Status 404 returned error can't find the container with id 3aa848320541fedef0ce6c7fbc995b9331bed6533c10d004bbca25301d2fcef0
Sep 11 23:06:14.386770 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:14.386743    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-fzfcr"
Sep 11 23:06:14.419971 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:14.419930    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-c8fff6cd4-tr742" event={"ID":"42864299-d484-4b6b-a2c7-dea808adb9f6","Type":"ContainerStarted","Data":"50a8b3b2767332d03941fa3c3d84088ec34a5ac3dae135e5c88faaca4cbfc57c"}
Sep 11 23:06:14.420182 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:14.419991    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-c8fff6cd4-tr742" event={"ID":"42864299-d484-4b6b-a2c7-dea808adb9f6","Type":"ContainerStarted","Data":"3aa848320541fedef0ce6c7fbc995b9331bed6533c10d004bbca25301d2fcef0"}
Sep 11 23:06:14.421764 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:14.421737    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-h6w6f" event={"ID":"a6dd2771-bd9e-4e48-976e-d2f4c14eba11","Type":"ContainerStarted","Data":"df40e56770afd652d1562e6121fb1452873de62690d26d9f005916673145ddd6"}
Sep 11 23:06:14.440351 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:14.440251    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-c8fff6cd4-tr742" podStartSLOduration=32.440233748 podStartE2EDuration="32.440233748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 23:05:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:06:14.439147781 +0000 UTC m=+298.472277000" watchObservedRunningTime="2026-09-11 23:06:14.440233748 +0000 UTC m=+298.473362965"
Sep 11 23:06:14.925234 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:14.925202    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-c8fff6cd4-tr742"
Sep 11 23:06:14.928063 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:14.928039    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-c8fff6cd4-tr742"
Sep 11 23:06:15.426121 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:15.426090    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-c8fff6cd4-tr742"
Sep 11 23:06:15.427398 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:15.427376    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-c8fff6cd4-tr742"
Sep 11 23:06:16.404832 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:16.404753    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7244_6b0a8af3-932c-4091-b7a0-180fe1751a3a/ovn-acl-logging/0.log"
Sep 11 23:06:16.405303 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:16.404982    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7244_6b0a8af3-932c-4091-b7a0-180fe1751a3a/ovn-acl-logging/0.log"
Sep 11 23:06:16.411807 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:16.411786    2546 kubelet.go:1648] "Image garbage collection succeeded"
Sep 11 23:06:16.431381 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:16.431336    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-h6w6f" event={"ID":"a6dd2771-bd9e-4e48-976e-d2f4c14eba11","Type":"ContainerStarted","Data":"d6027ccc5b604967f65c4ba3b82d79999eed9315cdafe837328c47ced53e2f58"}
Sep 11 23:06:16.451882 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:16.451712    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-h6w6f" podStartSLOduration=32.626680189 podStartE2EDuration="34.451692892s" totalImagesPullingTime="1.825012703s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 23:05:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:06:16.451245629 +0000 UTC m=+300.484374845" watchObservedRunningTime="2026-09-11 23:06:16.451692892 +0000 UTC m=+300.484822109"
Sep 11 23:06:21.364103 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:21.364054    2546 patch_prober.go:28] interesting pod/image-registry-5c8f46447c-cdpct container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 11 23:06:21.364587 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:21.364123    2546 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-5c8f46447c-cdpct" podUID="8ea4ca4e-b321-4324-b380-dab72513b02a" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 23:06:23.375892 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:23.375860    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5c8f46447c-cdpct"
Sep 11 23:06:23.487986 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:23.487956    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vbgsz"
Sep 11 23:06:29.614463 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:29.614429    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5c8f46447c-cdpct"]
Sep 11 23:06:34.029775 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:34.029740    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-p4wm5"]
Sep 11 23:06:34.035593 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:34.035569    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-p4wm5"
Sep 11 23:06:34.036927 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:34.036896    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45cs5\" (UniqueName: \"kubernetes.io/projected/c04c75b9-4bcf-4e64-9e21-f017aa288b6f-kube-api-access-45cs5\") pod \"kube-state-metrics-59fbdd77d-p4wm5\" (UID: \"c04c75b9-4bcf-4e64-9e21-f017aa288b6f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-p4wm5"
Sep 11 23:06:34.037024 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:34.036944    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c04c75b9-4bcf-4e64-9e21-f017aa288b6f-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-p4wm5\" (UID: \"c04c75b9-4bcf-4e64-9e21-f017aa288b6f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-p4wm5"
Sep 11 23:06:34.037024 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:34.036979    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c04c75b9-4bcf-4e64-9e21-f017aa288b6f-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-p4wm5\" (UID: \"c04c75b9-4bcf-4e64-9e21-f017aa288b6f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-p4wm5"
Sep 11 23:06:34.037093 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:34.037048    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c04c75b9-4bcf-4e64-9e21-f017aa288b6f-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-p4wm5\" (UID: \"c04c75b9-4bcf-4e64-9e21-f017aa288b6f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-p4wm5"
Sep 11 23:06:34.037093 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:34.037069    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c04c75b9-4bcf-4e64-9e21-f017aa288b6f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-p4wm5\" (UID: \"c04c75b9-4bcf-4e64-9e21-f017aa288b6f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-p4wm5"
Sep 11 23:06:34.037176 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:34.037097    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c04c75b9-4bcf-4e64-9e21-f017aa288b6f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-p4wm5\" (UID: \"c04c75b9-4bcf-4e64-9e21-f017aa288b6f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-p4wm5"
Sep 11 23:06:34.038629 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:34.038612    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 11 23:06:34.039035 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:34.039020    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 11 23:06:34.040175 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:34.040126    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 11 23:06:34.040296 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:34.040133    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-bxm9v\""
Sep 11 23:06:34.040646 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:34.040629    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 11 23:06:34.050847 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:34.050821    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-p4wm5"]
Sep 11 23:06:34.053268 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:34.053248    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-kwrgh"]
Sep 11 23:06:34.056812 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:34.056793    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kwrgh"
Sep 11 23:06:34.062538 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:34.060399    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 11 23:06:34.062538 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:34.060909    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-hcq8z\""
Sep 11 23:06:34.062538 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:34.061215    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 11 23:06:34.062538 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:34.061431    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 11 23:06:34.137723 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:34.137683    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c04c75b9-4bcf-4e64-9e21-f017aa288b6f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-p4wm5\" (UID: \"c04c75b9-4bcf-4e64-9e21-f017aa288b6f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-p4wm5"
Sep 11 23:06:34.137723 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:34.137728    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2668c2c0-570b-4b37-b41a-6baddf95128b-metrics-client-ca\") pod \"node-exporter-kwrgh\" (UID: \"2668c2c0-570b-4b37-b41a-6baddf95128b\") " pod="openshift-monitoring/node-exporter-kwrgh"
Sep 11 23:06:34.138006 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:34.137755    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2668c2c0-570b-4b37-b41a-6baddf95128b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kwrgh\" (UID: \"2668c2c0-570b-4b37-b41a-6baddf95128b\") " pod="openshift-monitoring/node-exporter-kwrgh"
Sep 11 23:06:34.138006 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:34.137801    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45cs5\" (UniqueName: \"kubernetes.io/projected/c04c75b9-4bcf-4e64-9e21-f017aa288b6f-kube-api-access-45cs5\") pod \"kube-state-metrics-59fbdd77d-p4wm5\" (UID: \"c04c75b9-4bcf-4e64-9e21-f017aa288b6f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-p4wm5"
Sep 11 23:06:34.138123 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:34.138047    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c04c75b9-4bcf-4e64-9e21-f017aa288b6f-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-p4wm5\" (UID: \"c04c75b9-4bcf-4e64-9e21-f017aa288b6f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-p4wm5"
Sep 11 23:06:34.138123 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:34.138090    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2668c2c0-570b-4b37-b41a-6baddf95128b-node-exporter-accelerators-collector-config\") pod \"node-exporter-kwrgh\" (UID: \"2668c2c0-570b-4b37-b41a-6baddf95128b\") " pod="openshift-monitoring/node-exporter-kwrgh"
Sep 11 23:06:34.138264 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:34.138127    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c04c75b9-4bcf-4e64-9e21-f017aa288b6f-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-p4wm5\" (UID: \"c04c75b9-4bcf-4e64-9e21-f017aa288b6f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-p4wm5"
Sep 11 23:06:34.138264 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:34.138190    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2668c2c0-570b-4b37-b41a-6baddf95128b-root\") pod \"node-exporter-kwrgh\" (UID: \"2668c2c0-570b-4b37-b41a-6baddf95128b\") " pod="openshift-monitoring/node-exporter-kwrgh"
Sep 11 23:06:34.138355 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:06:34.138268    2546 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Sep 11 23:06:34.138355 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:34.138322    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2668c2c0-570b-4b37-b41a-6baddf95128b-node-exporter-wtmp\") pod \"node-exporter-kwrgh\" (UID: \"2668c2c0-570b-4b37-b41a-6baddf95128b\") " pod="openshift-monitoring/node-exporter-kwrgh"
Sep 11 23:06:34.138452 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:06:34.138375    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c04c75b9-4bcf-4e64-9e21-f017aa288b6f-kube-state-metrics-tls podName:c04c75b9-4bcf-4e64-9e21-f017aa288b6f nodeName:}" failed. No retries permitted until 2026-09-11 23:06:34.638349988 +0000 UTC m=+318.671479183 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/c04c75b9-4bcf-4e64-9e21-f017aa288b6f-kube-state-metrics-tls") pod "kube-state-metrics-59fbdd77d-p4wm5" (UID: "c04c75b9-4bcf-4e64-9e21-f017aa288b6f") : secret "kube-state-metrics-tls" not found
Sep 11 23:06:34.138452 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:34.138440    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c04c75b9-4bcf-4e64-9e21-f017aa288b6f-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-p4wm5\" (UID: \"c04c75b9-4bcf-4e64-9e21-f017aa288b6f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-p4wm5"
Sep 11 23:06:34.138561 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:34.138462    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c04c75b9-4bcf-4e64-9e21-f017aa288b6f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-p4wm5\" (UID: \"c04c75b9-4bcf-4e64-9e21-f017aa288b6f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-p4wm5"
Sep 11 23:06:34.138561 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:34.138482    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c04c75b9-4bcf-4e64-9e21-f017aa288b6f-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-p4wm5\" (UID: \"c04c75b9-4bcf-4e64-9e21-f017aa288b6f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-p4wm5"
Sep 11 23:06:34.138561 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:34.138493    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2668c2c0-570b-4b37-b41a-6baddf95128b-sys\") pod \"node-exporter-kwrgh\" (UID: \"2668c2c0-570b-4b37-b41a-6baddf95128b\") " pod="openshift-monitoring/node-exporter-kwrgh"
Sep 11 23:06:34.138561 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:34.138534    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2668c2c0-570b-4b37-b41a-6baddf95128b-node-exporter-textfile\") pod \"node-exporter-kwrgh\" (UID: \"2668c2c0-570b-4b37-b41a-6baddf95128b\") " pod="openshift-monitoring/node-exporter-kwrgh"
Sep 11 23:06:34.138728 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:34.138584    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28rtq\" (UniqueName: \"kubernetes.io/projected/2668c2c0-570b-4b37-b41a-6baddf95128b-kube-api-access-28rtq\") pod \"node-exporter-kwrgh\" (UID: \"2668c2c0-570b-4b37-b41a-6baddf95128b\") " pod="openshift-monitoring/node-exporter-kwrgh"
Sep 11 23:06:34.138728 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:34.138619    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2668c2c0-570b-4b37-b41a-6baddf95128b-node-exporter-tls\") pod \"node-exporter-kwrgh\" (UID: \"2668c2c0-570b-4b37-b41a-6baddf95128b\") " pod="openshift-monitoring/node-exporter-kwrgh"
Sep 11 23:06:34.139030 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:34.139014    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c04c75b9-4bcf-4e64-9e21-f017aa288b6f-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-p4wm5\" (UID: \"c04c75b9-4bcf-4e64-9e21-f017aa288b6f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-p4wm5"
Sep 11 23:06:34.139067 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:34.139045    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c04c75b9-4bcf-4e64-9e21-f017aa288b6f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-p4wm5\" (UID: \"c04c75b9-4bcf-4e64-9e21-f017aa288b6f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-p4wm5"
Sep 11 23:06:34.140777 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:34.140753    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c04c75b9-4bcf-4e64-9e21-f017aa288b6f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-p4wm5\" (UID: \"c04c75b9-4bcf-4e64-9e21-f017aa288b6f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-p4wm5"
Sep 11 23:06:34.148184 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:34.148147    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45cs5\" (UniqueName: \"kubernetes.io/projected/c04c75b9-4bcf-4e64-9e21-f017aa288b6f-kube-api-access-45cs5\") pod \"kube-state-metrics-59fbdd77d-p4wm5\" (UID: \"c04c75b9-4bcf-4e64-9e21-f017aa288b6f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-p4wm5"
Sep 11 23:06:34.239808 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:34.239763    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2668c2c0-570b-4b37-b41a-6baddf95128b-sys\") pod \"node-exporter-kwrgh\" (UID: \"2668c2c0-570b-4b37-b41a-6baddf95128b\") " pod="openshift-monitoring/node-exporter-kwrgh"
Sep 11 23:06:34.239808 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:34.239806    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2668c2c0-570b-4b37-b41a-6baddf95128b-node-exporter-textfile\") pod \"node-exporter-kwrgh\" (UID: \"2668c2c0-570b-4b37-b41a-6baddf95128b\") " pod="openshift-monitoring/node-exporter-kwrgh"
Sep 11 23:06:34.240066 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:34.239832    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28rtq\" (UniqueName: \"kubernetes.io/projected/2668c2c0-570b-4b37-b41a-6baddf95128b-kube-api-access-28rtq\") pod \"node-exporter-kwrgh\" (UID: \"2668c2c0-570b-4b37-b41a-6baddf95128b\") " pod="openshift-monitoring/node-exporter-kwrgh"
Sep 11 23:06:34.240066 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:34.239859    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2668c2c0-570b-4b37-b41a-6baddf95128b-node-exporter-tls\") pod \"node-exporter-kwrgh\" (UID: \"2668c2c0-570b-4b37-b41a-6baddf95128b\") " pod="openshift-monitoring/node-exporter-kwrgh"
Sep 11 23:06:34.240066 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:34.239880    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2668c2c0-570b-4b37-b41a-6baddf95128b-sys\") pod \"node-exporter-kwrgh\" (UID: \"2668c2c0-570b-4b37-b41a-6baddf95128b\") " pod="openshift-monitoring/node-exporter-kwrgh"
Sep 11 23:06:34.240066 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:34.239894    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2668c2c0-570b-4b37-b41a-6baddf95128b-metrics-client-ca\") pod \"node-exporter-kwrgh\" (UID: \"2668c2c0-570b-4b37-b41a-6baddf95128b\") " pod="openshift-monitoring/node-exporter-kwrgh"
Sep 11 23:06:34.240066 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:34.239916    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2668c2c0-570b-4b37-b41a-6baddf95128b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kwrgh\" (UID: \"2668c2c0-570b-4b37-b41a-6baddf95128b\") " pod="openshift-monitoring/node-exporter-kwrgh"
Sep 11 23:06:34.240066 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:34.239972    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2668c2c0-570b-4b37-b41a-6baddf95128b-node-exporter-accelerators-collector-config\") pod \"node-exporter-kwrgh\" (UID: \"2668c2c0-570b-4b37-b41a-6baddf95128b\") " pod="openshift-monitoring/node-exporter-kwrgh"
Sep 11 23:06:34.240066 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:06:34.239998    2546 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 11 23:06:34.240066 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:34.240025    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2668c2c0-570b-4b37-b41a-6baddf95128b-root\") pod \"node-exporter-kwrgh\" (UID: \"2668c2c0-570b-4b37-b41a-6baddf95128b\") " pod="openshift-monitoring/node-exporter-kwrgh"
Sep 11 23:06:34.240066 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:06:34.240060    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2668c2c0-570b-4b37-b41a-6baddf95128b-node-exporter-tls podName:2668c2c0-570b-4b37-b41a-6baddf95128b nodeName:}" failed. No retries permitted until 2026-09-11 23:06:34.740039567 +0000 UTC m=+318.773168763 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/2668c2c0-570b-4b37-b41a-6baddf95128b-node-exporter-tls") pod "node-exporter-kwrgh" (UID: "2668c2c0-570b-4b37-b41a-6baddf95128b") : secret "node-exporter-tls" not found
Sep 11 23:06:34.240505 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:34.240080    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2668c2c0-570b-4b37-b41a-6baddf95128b-node-exporter-wtmp\") pod \"node-exporter-kwrgh\" (UID: \"2668c2c0-570b-4b37-b41a-6baddf95128b\") " pod="openshift-monitoring/node-exporter-kwrgh"
Sep 11 23:06:34.240505 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:34.240086    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2668c2c0-570b-4b37-b41a-6baddf95128b-root\") pod \"node-exporter-kwrgh\" (UID: \"2668c2c0-570b-4b37-b41a-6baddf95128b\") " pod="openshift-monitoring/node-exporter-kwrgh"
Sep 11 23:06:34.240505 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:34.240240    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2668c2c0-570b-4b37-b41a-6baddf95128b-node-exporter-textfile\") pod \"node-exporter-kwrgh\" (UID: \"2668c2c0-570b-4b37-b41a-6baddf95128b\") " pod="openshift-monitoring/node-exporter-kwrgh"
Sep 11 23:06:34.240505 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:34.240402    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2668c2c0-570b-4b37-b41a-6baddf95128b-node-exporter-wtmp\") pod \"node-exporter-kwrgh\" (UID: \"2668c2c0-570b-4b37-b41a-6baddf95128b\") " pod="openshift-monitoring/node-exporter-kwrgh"
Sep 11 23:06:34.240630 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:34.240526    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2668c2c0-570b-4b37-b41a-6baddf95128b-metrics-client-ca\") pod \"node-exporter-kwrgh\" (UID: \"2668c2c0-570b-4b37-b41a-6baddf95128b\") " pod="openshift-monitoring/node-exporter-kwrgh"
Sep 11 23:06:34.240630 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:34.240550    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2668c2c0-570b-4b37-b41a-6baddf95128b-node-exporter-accelerators-collector-config\") pod \"node-exporter-kwrgh\" (UID: \"2668c2c0-570b-4b37-b41a-6baddf95128b\") " pod="openshift-monitoring/node-exporter-kwrgh"
Sep 11 23:06:34.242506 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:34.242483    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2668c2c0-570b-4b37-b41a-6baddf95128b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kwrgh\" (UID: \"2668c2c0-570b-4b37-b41a-6baddf95128b\") " pod="openshift-monitoring/node-exporter-kwrgh"
Sep 11 23:06:34.252192 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:34.252173    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28rtq\" (UniqueName: \"kubernetes.io/projected/2668c2c0-570b-4b37-b41a-6baddf95128b-kube-api-access-28rtq\") pod \"node-exporter-kwrgh\" (UID: \"2668c2c0-570b-4b37-b41a-6baddf95128b\") " pod="openshift-monitoring/node-exporter-kwrgh"
Sep 11 23:06:34.643958 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:34.643918    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c04c75b9-4bcf-4e64-9e21-f017aa288b6f-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-p4wm5\" (UID: \"c04c75b9-4bcf-4e64-9e21-f017aa288b6f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-p4wm5"
Sep 11 23:06:34.646665 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:34.646639    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c04c75b9-4bcf-4e64-9e21-f017aa288b6f-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-p4wm5\" (UID: \"c04c75b9-4bcf-4e64-9e21-f017aa288b6f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-p4wm5"
Sep 11 23:06:34.744959 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:34.744918    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2668c2c0-570b-4b37-b41a-6baddf95128b-node-exporter-tls\") pod \"node-exporter-kwrgh\" (UID: \"2668c2c0-570b-4b37-b41a-6baddf95128b\") " pod="openshift-monitoring/node-exporter-kwrgh"
Sep 11 23:06:34.747540 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:34.747514    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2668c2c0-570b-4b37-b41a-6baddf95128b-node-exporter-tls\") pod \"node-exporter-kwrgh\" (UID: \"2668c2c0-570b-4b37-b41a-6baddf95128b\") " pod="openshift-monitoring/node-exporter-kwrgh"
Sep 11 23:06:34.944958 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:34.944915    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-p4wm5"
Sep 11 23:06:34.970950 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:34.970916    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kwrgh"
Sep 11 23:06:34.984459 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:06:34.984432    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2668c2c0_570b_4b37_b41a_6baddf95128b.slice/crio-93dd8fbf745bfe341b9ec9e0af6c0df79a0b48b4faf8df6b059d79206f558f1e WatchSource:0}: Error finding container 93dd8fbf745bfe341b9ec9e0af6c0df79a0b48b4faf8df6b059d79206f558f1e: Status 404 returned error can't find the container with id 93dd8fbf745bfe341b9ec9e0af6c0df79a0b48b4faf8df6b059d79206f558f1e
Sep 11 23:06:34.986884 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:34.986750    2546 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 23:06:35.222933 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:35.222908    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-p4wm5"]
Sep 11 23:06:35.225595 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:06:35.225563    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc04c75b9_4bcf_4e64_9e21_f017aa288b6f.slice/crio-117d43df53b3863a0f25faf35620dd5561feffad40ed85cad3d0a14229726896 WatchSource:0}: Error finding container 117d43df53b3863a0f25faf35620dd5561feffad40ed85cad3d0a14229726896: Status 404 returned error can't find the container with id 117d43df53b3863a0f25faf35620dd5561feffad40ed85cad3d0a14229726896
Sep 11 23:06:35.497227 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:35.497125    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-p4wm5" event={"ID":"c04c75b9-4bcf-4e64-9e21-f017aa288b6f","Type":"ContainerStarted","Data":"117d43df53b3863a0f25faf35620dd5561feffad40ed85cad3d0a14229726896"}
Sep 11 23:06:35.498612 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:35.498579    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kwrgh" event={"ID":"2668c2c0-570b-4b37-b41a-6baddf95128b","Type":"ContainerStarted","Data":"93dd8fbf745bfe341b9ec9e0af6c0df79a0b48b4faf8df6b059d79206f558f1e"}
Sep 11 23:06:35.929612 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:35.929583    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-8559bbc876-crxkz"]
Sep 11 23:06:35.934318 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:35.934296    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-8559bbc876-crxkz"
Sep 11 23:06:35.937482 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:35.937463    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 11 23:06:35.938016 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:35.937828    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 11 23:06:35.938016 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:35.937840    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 11 23:06:35.938016 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:35.937878    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 11 23:06:35.938016 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:35.937913    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-36ogps7tq3ddf\""
Sep 11 23:06:35.938016 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:35.937878    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 11 23:06:35.938016 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:35.937878    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-g29gv\""
Sep 11 23:06:35.946033 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:35.946008    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-8559bbc876-crxkz"]
Sep 11 23:06:35.955310 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:35.955284    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f0617d4e-a6e6-47a9-9ba8-8902f4adda81-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-8559bbc876-crxkz\" (UID: \"f0617d4e-a6e6-47a9-9ba8-8902f4adda81\") " pod="openshift-monitoring/thanos-querier-8559bbc876-crxkz"
Sep 11 23:06:35.955384 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:35.955339    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f0617d4e-a6e6-47a9-9ba8-8902f4adda81-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-8559bbc876-crxkz\" (UID: \"f0617d4e-a6e6-47a9-9ba8-8902f4adda81\") " pod="openshift-monitoring/thanos-querier-8559bbc876-crxkz"
Sep 11 23:06:35.955436 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:35.955414    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f0617d4e-a6e6-47a9-9ba8-8902f4adda81-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-8559bbc876-crxkz\" (UID: \"f0617d4e-a6e6-47a9-9ba8-8902f4adda81\") " pod="openshift-monitoring/thanos-querier-8559bbc876-crxkz"
Sep 11 23:06:35.955544 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:35.955512    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f0617d4e-a6e6-47a9-9ba8-8902f4adda81-metrics-client-ca\") pod \"thanos-querier-8559bbc876-crxkz\" (UID: \"f0617d4e-a6e6-47a9-9ba8-8902f4adda81\") " pod="openshift-monitoring/thanos-querier-8559bbc876-crxkz"
Sep 11 23:06:35.955591 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:35.955555    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f0617d4e-a6e6-47a9-9ba8-8902f4adda81-secret-grpc-tls\") pod \"thanos-querier-8559bbc876-crxkz\" (UID: \"f0617d4e-a6e6-47a9-9ba8-8902f4adda81\") " pod="openshift-monitoring/thanos-querier-8559bbc876-crxkz"
Sep 11 23:06:35.955642 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:35.955597    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f0617d4e-a6e6-47a9-9ba8-8902f4adda81-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-8559bbc876-crxkz\" (UID: \"f0617d4e-a6e6-47a9-9ba8-8902f4adda81\") " pod="openshift-monitoring/thanos-querier-8559bbc876-crxkz"
Sep 11 23:06:35.955642 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:35.955628    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f0617d4e-a6e6-47a9-9ba8-8902f4adda81-secret-thanos-querier-tls\") pod \"thanos-querier-8559bbc876-crxkz\" (UID: \"f0617d4e-a6e6-47a9-9ba8-8902f4adda81\") " pod="openshift-monitoring/thanos-querier-8559bbc876-crxkz"
Sep 11 23:06:35.955738 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:35.955723    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l58w5\" (UniqueName: \"kubernetes.io/projected/f0617d4e-a6e6-47a9-9ba8-8902f4adda81-kube-api-access-l58w5\") pod \"thanos-querier-8559bbc876-crxkz\" (UID: \"f0617d4e-a6e6-47a9-9ba8-8902f4adda81\") " pod="openshift-monitoring/thanos-querier-8559bbc876-crxkz"
Sep 11 23:06:36.056628 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:36.056595    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f0617d4e-a6e6-47a9-9ba8-8902f4adda81-secret-grpc-tls\") pod \"thanos-querier-8559bbc876-crxkz\" (UID: \"f0617d4e-a6e6-47a9-9ba8-8902f4adda81\") " pod="openshift-monitoring/thanos-querier-8559bbc876-crxkz"
Sep 11 23:06:36.056791 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:36.056638    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f0617d4e-a6e6-47a9-9ba8-8902f4adda81-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-8559bbc876-crxkz\" (UID: \"f0617d4e-a6e6-47a9-9ba8-8902f4adda81\") " pod="openshift-monitoring/thanos-querier-8559bbc876-crxkz"
Sep 11 23:06:36.056791 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:36.056668    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f0617d4e-a6e6-47a9-9ba8-8902f4adda81-secret-thanos-querier-tls\") pod \"thanos-querier-8559bbc876-crxkz\" (UID: \"f0617d4e-a6e6-47a9-9ba8-8902f4adda81\") " pod="openshift-monitoring/thanos-querier-8559bbc876-crxkz"
Sep 11 23:06:36.056791 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:36.056739    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l58w5\" (UniqueName: \"kubernetes.io/projected/f0617d4e-a6e6-47a9-9ba8-8902f4adda81-kube-api-access-l58w5\") pod \"thanos-querier-8559bbc876-crxkz\" (UID: \"f0617d4e-a6e6-47a9-9ba8-8902f4adda81\") " pod="openshift-monitoring/thanos-querier-8559bbc876-crxkz"
Sep 11 23:06:36.056904 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:36.056767    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f0617d4e-a6e6-47a9-9ba8-8902f4adda81-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-8559bbc876-crxkz\" (UID: \"f0617d4e-a6e6-47a9-9ba8-8902f4adda81\") " pod="openshift-monitoring/thanos-querier-8559bbc876-crxkz"
Sep 11 23:06:36.056904 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:36.056830    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f0617d4e-a6e6-47a9-9ba8-8902f4adda81-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-8559bbc876-crxkz\" (UID: \"f0617d4e-a6e6-47a9-9ba8-8902f4adda81\") " pod="openshift-monitoring/thanos-querier-8559bbc876-crxkz"
Sep 11 23:06:36.056904 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:36.056862    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f0617d4e-a6e6-47a9-9ba8-8902f4adda81-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-8559bbc876-crxkz\" (UID: \"f0617d4e-a6e6-47a9-9ba8-8902f4adda81\") " pod="openshift-monitoring/thanos-querier-8559bbc876-crxkz"
Sep 11 23:06:36.057057 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:36.056906    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f0617d4e-a6e6-47a9-9ba8-8902f4adda81-metrics-client-ca\") pod \"thanos-querier-8559bbc876-crxkz\" (UID: \"f0617d4e-a6e6-47a9-9ba8-8902f4adda81\") " pod="openshift-monitoring/thanos-querier-8559bbc876-crxkz"
Sep 11 23:06:36.057922 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:36.057878    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f0617d4e-a6e6-47a9-9ba8-8902f4adda81-metrics-client-ca\") pod \"thanos-querier-8559bbc876-crxkz\" (UID: \"f0617d4e-a6e6-47a9-9ba8-8902f4adda81\") " pod="openshift-monitoring/thanos-querier-8559bbc876-crxkz"
Sep 11 23:06:36.059797 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:36.059774    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f0617d4e-a6e6-47a9-9ba8-8902f4adda81-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-8559bbc876-crxkz\" (UID: \"f0617d4e-a6e6-47a9-9ba8-8902f4adda81\") " pod="openshift-monitoring/thanos-querier-8559bbc876-crxkz"
Sep 11 23:06:36.059908 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:36.059842    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f0617d4e-a6e6-47a9-9ba8-8902f4adda81-secret-grpc-tls\") pod \"thanos-querier-8559bbc876-crxkz\" (UID: \"f0617d4e-a6e6-47a9-9ba8-8902f4adda81\") " pod="openshift-monitoring/thanos-querier-8559bbc876-crxkz"
Sep 11 23:06:36.060253 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:36.060216    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f0617d4e-a6e6-47a9-9ba8-8902f4adda81-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-8559bbc876-crxkz\" (UID: \"f0617d4e-a6e6-47a9-9ba8-8902f4adda81\") " pod="openshift-monitoring/thanos-querier-8559bbc876-crxkz"
Sep 11 23:06:36.060253 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:36.060229    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f0617d4e-a6e6-47a9-9ba8-8902f4adda81-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-8559bbc876-crxkz\" (UID: \"f0617d4e-a6e6-47a9-9ba8-8902f4adda81\") " pod="openshift-monitoring/thanos-querier-8559bbc876-crxkz"
Sep 11 23:06:36.060388 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:36.060356    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f0617d4e-a6e6-47a9-9ba8-8902f4adda81-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-8559bbc876-crxkz\" (UID: \"f0617d4e-a6e6-47a9-9ba8-8902f4adda81\") " pod="openshift-monitoring/thanos-querier-8559bbc876-crxkz"
Sep 11 23:06:36.060469 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:36.060449    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f0617d4e-a6e6-47a9-9ba8-8902f4adda81-secret-thanos-querier-tls\") pod \"thanos-querier-8559bbc876-crxkz\" (UID: \"f0617d4e-a6e6-47a9-9ba8-8902f4adda81\") " pod="openshift-monitoring/thanos-querier-8559bbc876-crxkz"
Sep 11 23:06:36.066233 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:36.066211    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l58w5\" (UniqueName: \"kubernetes.io/projected/f0617d4e-a6e6-47a9-9ba8-8902f4adda81-kube-api-access-l58w5\") pod \"thanos-querier-8559bbc876-crxkz\" (UID: \"f0617d4e-a6e6-47a9-9ba8-8902f4adda81\") " pod="openshift-monitoring/thanos-querier-8559bbc876-crxkz"
Sep 11 23:06:36.247380 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:36.247326    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-8559bbc876-crxkz"
Sep 11 23:06:36.503173 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:36.503070    2546 generic.go:356] "Generic (PLEG): container finished" podID="2668c2c0-570b-4b37-b41a-6baddf95128b" containerID="15e5c0d77c1135bb7b09d77509d5ae518b9a8831b09e4d9b2ab1c2d91ce78924" exitCode=0
Sep 11 23:06:36.503342 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:36.503168    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kwrgh" event={"ID":"2668c2c0-570b-4b37-b41a-6baddf95128b","Type":"ContainerDied","Data":"15e5c0d77c1135bb7b09d77509d5ae518b9a8831b09e4d9b2ab1c2d91ce78924"}
Sep 11 23:06:36.547379 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:36.547338    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-8559bbc876-crxkz"]
Sep 11 23:06:36.553254 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:06:36.553224    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0617d4e_a6e6_47a9_9ba8_8902f4adda81.slice/crio-fb3b2128e074a987b475013ca3be2a428799f383ac456d6cf3c08b0de804bbf2 WatchSource:0}: Error finding container fb3b2128e074a987b475013ca3be2a428799f383ac456d6cf3c08b0de804bbf2: Status 404 returned error can't find the container with id fb3b2128e074a987b475013ca3be2a428799f383ac456d6cf3c08b0de804bbf2
Sep 11 23:06:37.509293 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:37.509195    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8559bbc876-crxkz" event={"ID":"f0617d4e-a6e6-47a9-9ba8-8902f4adda81","Type":"ContainerStarted","Data":"fb3b2128e074a987b475013ca3be2a428799f383ac456d6cf3c08b0de804bbf2"}
Sep 11 23:06:37.512437 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:37.512401    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kwrgh" event={"ID":"2668c2c0-570b-4b37-b41a-6baddf95128b","Type":"ContainerStarted","Data":"ceda0b9f1d9f173a9c781aa6d89473956d05c90c04655decd456b241a2d351c0"}
Sep 11 23:06:37.512586 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:37.512463    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kwrgh" event={"ID":"2668c2c0-570b-4b37-b41a-6baddf95128b","Type":"ContainerStarted","Data":"85a00e2b7f1b8e7502ebc6911f8bd584660a6ecdb060eea56043e10283474ee8"}
Sep 11 23:06:37.515008 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:37.514979    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-p4wm5" event={"ID":"c04c75b9-4bcf-4e64-9e21-f017aa288b6f","Type":"ContainerStarted","Data":"70558e1457cc5badc475a92b9f015ca89b4fe0ffaa1708f9303a52c74dd1a5a0"}
Sep 11 23:06:37.515171 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:37.515040    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-p4wm5" event={"ID":"c04c75b9-4bcf-4e64-9e21-f017aa288b6f","Type":"ContainerStarted","Data":"cf9f17bacfb59f7b0ad6bfb106b4a4d4cacef7765f216dd105523468cf0cffba"}
Sep 11 23:06:37.515171 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:37.515072    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-p4wm5" event={"ID":"c04c75b9-4bcf-4e64-9e21-f017aa288b6f","Type":"ContainerStarted","Data":"b019ef6178f9547a42e75a66cae9d027f6d046a2170eb7e4da4753b4cf7168b9"}
Sep 11 23:06:37.540034 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:37.539976    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-kwrgh" podStartSLOduration=2.562974386 podStartE2EDuration="3.539956914s" totalImagesPullingTime="902.349387ms" totalInitContainerRuntime="74.633141ms" isStatefulPod=true podCreationTimestamp="2026-09-11 23:06:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:06:37.539201405 +0000 UTC m=+321.572330622" watchObservedRunningTime="2026-09-11 23:06:37.539956914 +0000 UTC m=+321.573086130"
Sep 11 23:06:37.560818 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:37.560528    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-p4wm5" podStartSLOduration=1.7473778370000002 podStartE2EDuration="3.560507949s" totalImagesPullingTime="1.813130112s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 23:06:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:06:37.560335415 +0000 UTC m=+321.593464655" watchObservedRunningTime="2026-09-11 23:06:37.560507949 +0000 UTC m=+321.593637167"
Sep 11 23:06:38.718237 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:38.718205    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-89z9j"]
Sep 11 23:06:38.722554 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:38.722533    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-89z9j"
Sep 11 23:06:38.725372 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:38.725344    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 11 23:06:38.725372 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:38.725364    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-4rpbs\""
Sep 11 23:06:38.732437 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:38.732280    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-89z9j"]
Sep 11 23:06:38.785343 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:38.785303    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/df384b1d-9bce-49ed-814d-4a83c3c4c15c-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-89z9j\" (UID: \"df384b1d-9bce-49ed-814d-4a83c3c4c15c\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-89z9j"
Sep 11 23:06:38.785523 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:38.785443    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsjrv\" (UniqueName: \"kubernetes.io/projected/df384b1d-9bce-49ed-814d-4a83c3c4c15c-kube-api-access-zsjrv\") pod \"monitoring-plugin-5d85778b9d-89z9j\" (UID: \"df384b1d-9bce-49ed-814d-4a83c3c4c15c\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-89z9j"
Sep 11 23:06:38.886608 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:38.886572    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/df384b1d-9bce-49ed-814d-4a83c3c4c15c-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-89z9j\" (UID: \"df384b1d-9bce-49ed-814d-4a83c3c4c15c\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-89z9j"
Sep 11 23:06:38.886785 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:38.886685    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsjrv\" (UniqueName: \"kubernetes.io/projected/df384b1d-9bce-49ed-814d-4a83c3c4c15c-kube-api-access-zsjrv\") pod \"monitoring-plugin-5d85778b9d-89z9j\" (UID: \"df384b1d-9bce-49ed-814d-4a83c3c4c15c\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-89z9j"
Sep 11 23:06:38.886785 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:06:38.886719    2546 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Sep 11 23:06:38.886902 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:06:38.886794    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df384b1d-9bce-49ed-814d-4a83c3c4c15c-monitoring-plugin-cert podName:df384b1d-9bce-49ed-814d-4a83c3c4c15c nodeName:}" failed. No retries permitted until 2026-09-11 23:06:39.386779163 +0000 UTC m=+323.419908358 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/df384b1d-9bce-49ed-814d-4a83c3c4c15c-monitoring-plugin-cert") pod "monitoring-plugin-5d85778b9d-89z9j" (UID: "df384b1d-9bce-49ed-814d-4a83c3c4c15c") : secret "monitoring-plugin-cert" not found
Sep 11 23:06:38.900926 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:38.900892    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsjrv\" (UniqueName: \"kubernetes.io/projected/df384b1d-9bce-49ed-814d-4a83c3c4c15c-kube-api-access-zsjrv\") pod \"monitoring-plugin-5d85778b9d-89z9j\" (UID: \"df384b1d-9bce-49ed-814d-4a83c3c4c15c\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-89z9j"
Sep 11 23:06:39.392011 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:39.391921    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/df384b1d-9bce-49ed-814d-4a83c3c4c15c-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-89z9j\" (UID: \"df384b1d-9bce-49ed-814d-4a83c3c4c15c\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-89z9j"
Sep 11 23:06:39.394518 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:39.394492    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/df384b1d-9bce-49ed-814d-4a83c3c4c15c-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-89z9j\" (UID: \"df384b1d-9bce-49ed-814d-4a83c3c4c15c\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-89z9j"
Sep 11 23:06:39.524703 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:39.524665    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8559bbc876-crxkz" event={"ID":"f0617d4e-a6e6-47a9-9ba8-8902f4adda81","Type":"ContainerStarted","Data":"fd2f8c04d02d616f4a3be023f494fa35987e575864aee5b5f314b1d0e35388ce"}
Sep 11 23:06:39.524865 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:39.524746    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8559bbc876-crxkz" event={"ID":"f0617d4e-a6e6-47a9-9ba8-8902f4adda81","Type":"ContainerStarted","Data":"22cc4db0e03571c6846f1fa1394f9f4bda8fb6a82690d2f613fa35af94a36a06"}
Sep 11 23:06:39.524865 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:39.524782    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8559bbc876-crxkz" event={"ID":"f0617d4e-a6e6-47a9-9ba8-8902f4adda81","Type":"ContainerStarted","Data":"e890bf4896f82f514f82d7540cafd9da4dd2d1eb32e89392d6d50dca18b2ba91"}
Sep 11 23:06:39.635084 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:39.635046    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-89z9j"
Sep 11 23:06:39.906384 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:39.906357    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-89z9j"]
Sep 11 23:06:39.909189 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:06:39.909137    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf384b1d_9bce_49ed_814d_4a83c3c4c15c.slice/crio-ffb9db103c438cc3c7b6f078140a288f95f5d3539b8e25fcd7de0334b9ee5531 WatchSource:0}: Error finding container ffb9db103c438cc3c7b6f078140a288f95f5d3539b8e25fcd7de0334b9ee5531: Status 404 returned error can't find the container with id ffb9db103c438cc3c7b6f078140a288f95f5d3539b8e25fcd7de0334b9ee5531
Sep 11 23:06:40.531244 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:40.531200    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-89z9j" event={"ID":"df384b1d-9bce-49ed-814d-4a83c3c4c15c","Type":"ContainerStarted","Data":"ffb9db103c438cc3c7b6f078140a288f95f5d3539b8e25fcd7de0334b9ee5531"}
Sep 11 23:06:41.536242 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:41.536196    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-89z9j" event={"ID":"df384b1d-9bce-49ed-814d-4a83c3c4c15c","Type":"ContainerStarted","Data":"fe34bb330c130bc8c7a539ea4d5dfb1180be31ecf04cb050a0915a31387c0d66"}
Sep 11 23:06:41.536680 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:41.536264    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-89z9j"
Sep 11 23:06:41.539651 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:41.539619    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8559bbc876-crxkz" event={"ID":"f0617d4e-a6e6-47a9-9ba8-8902f4adda81","Type":"ContainerStarted","Data":"6493cb0e10fb5134867a0f3b932d9692df217f18503042503a8e6d48a6f72bc7"}
Sep 11 23:06:41.539785 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:41.539693    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8559bbc876-crxkz" event={"ID":"f0617d4e-a6e6-47a9-9ba8-8902f4adda81","Type":"ContainerStarted","Data":"f8c5a2a65177288778190def038d7b967f89f3685c184c628c95635fecaa0cbe"}
Sep 11 23:06:41.539785 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:41.539734    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8559bbc876-crxkz" event={"ID":"f0617d4e-a6e6-47a9-9ba8-8902f4adda81","Type":"ContainerStarted","Data":"693451560bef9d6bfa3b95e1208487f1e4902fa5dbf3ba2e8d59d7ce6a7818dd"}
Sep 11 23:06:41.539785 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:41.539778    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-8559bbc876-crxkz"
Sep 11 23:06:41.541572 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:41.541554    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-89z9j"
Sep 11 23:06:41.553808 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:41.553757    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-89z9j" podStartSLOduration=2.07922694 podStartE2EDuration="3.553742381s" totalImagesPullingTime="1.474515441s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 23:06:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:06:41.552364994 +0000 UTC m=+325.585494211" watchObservedRunningTime="2026-09-11 23:06:41.553742381 +0000 UTC m=+325.586871598"
Sep 11 23:06:41.577411 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:41.577361    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-8559bbc876-crxkz" podStartSLOduration=2.705185718 podStartE2EDuration="6.577346576s" totalImagesPullingTime="3.872160858s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 23:06:35 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:06:41.576381356 +0000 UTC m=+325.609510575" watchObservedRunningTime="2026-09-11 23:06:41.577346576 +0000 UTC m=+325.610475793"
Sep 11 23:06:47.550499 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:47.550469    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-8559bbc876-crxkz"
Sep 11 23:06:54.633741 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:54.633703    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5c8f46447c-cdpct" podUID="8ea4ca4e-b321-4324-b380-dab72513b02a" containerName="registry" containerID="cri-o://9cda1c095880367248025a50d1024e81d0178ea993df2dbaa2ba237c64a36890" gracePeriod=30
Sep 11 23:06:54.878716 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:54.878692    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c8f46447c-cdpct"
Sep 11 23:06:54.925903 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:54.925872    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8ea4ca4e-b321-4324-b380-dab72513b02a-ca-trust-extracted\") pod \"8ea4ca4e-b321-4324-b380-dab72513b02a\" (UID: \"8ea4ca4e-b321-4324-b380-dab72513b02a\") "
Sep 11 23:06:54.926104 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:54.925910    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8ea4ca4e-b321-4324-b380-dab72513b02a-installation-pull-secrets\") pod \"8ea4ca4e-b321-4324-b380-dab72513b02a\" (UID: \"8ea4ca4e-b321-4324-b380-dab72513b02a\") "
Sep 11 23:06:54.926104 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:54.925935    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-987rm\" (UniqueName: \"kubernetes.io/projected/8ea4ca4e-b321-4324-b380-dab72513b02a-kube-api-access-987rm\") pod \"8ea4ca4e-b321-4324-b380-dab72513b02a\" (UID: \"8ea4ca4e-b321-4324-b380-dab72513b02a\") "
Sep 11 23:06:54.926104 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:54.925960    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8ea4ca4e-b321-4324-b380-dab72513b02a-bound-sa-token\") pod \"8ea4ca4e-b321-4324-b380-dab72513b02a\" (UID: \"8ea4ca4e-b321-4324-b380-dab72513b02a\") "
Sep 11 23:06:54.926104 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:54.925990    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8ea4ca4e-b321-4324-b380-dab72513b02a-trusted-ca\") pod \"8ea4ca4e-b321-4324-b380-dab72513b02a\" (UID: \"8ea4ca4e-b321-4324-b380-dab72513b02a\") "
Sep 11 23:06:54.926104 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:54.926025    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8ea4ca4e-b321-4324-b380-dab72513b02a-image-registry-private-configuration\") pod \"8ea4ca4e-b321-4324-b380-dab72513b02a\" (UID: \"8ea4ca4e-b321-4324-b380-dab72513b02a\") "
Sep 11 23:06:54.926104 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:54.926060    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8ea4ca4e-b321-4324-b380-dab72513b02a-registry-certificates\") pod \"8ea4ca4e-b321-4324-b380-dab72513b02a\" (UID: \"8ea4ca4e-b321-4324-b380-dab72513b02a\") "
Sep 11 23:06:54.926446 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:54.926112    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8ea4ca4e-b321-4324-b380-dab72513b02a-registry-tls\") pod \"8ea4ca4e-b321-4324-b380-dab72513b02a\" (UID: \"8ea4ca4e-b321-4324-b380-dab72513b02a\") "
Sep 11 23:06:54.926941 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:54.926582    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ea4ca4e-b321-4324-b380-dab72513b02a-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "8ea4ca4e-b321-4324-b380-dab72513b02a" (UID: "8ea4ca4e-b321-4324-b380-dab72513b02a"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 23:06:54.926941 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:54.926893    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ea4ca4e-b321-4324-b380-dab72513b02a-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "8ea4ca4e-b321-4324-b380-dab72513b02a" (UID: "8ea4ca4e-b321-4324-b380-dab72513b02a"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 23:06:54.928607 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:54.928578    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ea4ca4e-b321-4324-b380-dab72513b02a-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "8ea4ca4e-b321-4324-b380-dab72513b02a" (UID: "8ea4ca4e-b321-4324-b380-dab72513b02a"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 23:06:54.929034 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:54.929004    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ea4ca4e-b321-4324-b380-dab72513b02a-kube-api-access-987rm" (OuterVolumeSpecName: "kube-api-access-987rm") pod "8ea4ca4e-b321-4324-b380-dab72513b02a" (UID: "8ea4ca4e-b321-4324-b380-dab72513b02a"). InnerVolumeSpecName "kube-api-access-987rm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 23:06:54.929125 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:54.929042    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ea4ca4e-b321-4324-b380-dab72513b02a-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "8ea4ca4e-b321-4324-b380-dab72513b02a" (UID: "8ea4ca4e-b321-4324-b380-dab72513b02a"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 23:06:54.929125 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:54.929081    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ea4ca4e-b321-4324-b380-dab72513b02a-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "8ea4ca4e-b321-4324-b380-dab72513b02a" (UID: "8ea4ca4e-b321-4324-b380-dab72513b02a"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 23:06:54.929125 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:54.929086    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ea4ca4e-b321-4324-b380-dab72513b02a-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "8ea4ca4e-b321-4324-b380-dab72513b02a" (UID: "8ea4ca4e-b321-4324-b380-dab72513b02a"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 23:06:54.934048 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:54.934024    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ea4ca4e-b321-4324-b380-dab72513b02a-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "8ea4ca4e-b321-4324-b380-dab72513b02a" (UID: "8ea4ca4e-b321-4324-b380-dab72513b02a"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:06:55.027001 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:55.026955    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-987rm\" (UniqueName: \"kubernetes.io/projected/8ea4ca4e-b321-4324-b380-dab72513b02a-kube-api-access-987rm\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:06:55.027001 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:55.026987    2546 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8ea4ca4e-b321-4324-b380-dab72513b02a-bound-sa-token\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:06:55.027001 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:55.026996    2546 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8ea4ca4e-b321-4324-b380-dab72513b02a-trusted-ca\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:06:55.027001 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:55.027005    2546 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8ea4ca4e-b321-4324-b380-dab72513b02a-image-registry-private-configuration\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:06:55.027001 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:55.027015    2546 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8ea4ca4e-b321-4324-b380-dab72513b02a-registry-certificates\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:06:55.027374 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:55.027024    2546 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8ea4ca4e-b321-4324-b380-dab72513b02a-registry-tls\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:06:55.027374 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:55.027033    2546 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8ea4ca4e-b321-4324-b380-dab72513b02a-ca-trust-extracted\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:06:55.027374 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:55.027042    2546 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8ea4ca4e-b321-4324-b380-dab72513b02a-installation-pull-secrets\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:06:55.594485 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:55.594450    2546 generic.go:356] "Generic (PLEG): container finished" podID="8ea4ca4e-b321-4324-b380-dab72513b02a" containerID="9cda1c095880367248025a50d1024e81d0178ea993df2dbaa2ba237c64a36890" exitCode=0
Sep 11 23:06:55.594641 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:55.594514    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c8f46447c-cdpct"
Sep 11 23:06:55.594641 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:55.594519    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5c8f46447c-cdpct" event={"ID":"8ea4ca4e-b321-4324-b380-dab72513b02a","Type":"ContainerDied","Data":"9cda1c095880367248025a50d1024e81d0178ea993df2dbaa2ba237c64a36890"}
Sep 11 23:06:55.594641 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:55.594631    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5c8f46447c-cdpct" event={"ID":"8ea4ca4e-b321-4324-b380-dab72513b02a","Type":"ContainerDied","Data":"6bdbafe025fa1d53e573219b37dff8b9ad6a6c594f84d23aa8bb6610ef8a9188"}
Sep 11 23:06:55.594801 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:55.594650    2546 scope.go:117] "RemoveContainer" containerID="9cda1c095880367248025a50d1024e81d0178ea993df2dbaa2ba237c64a36890"
Sep 11 23:06:55.603721 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:55.603700    2546 scope.go:117] "RemoveContainer" containerID="9cda1c095880367248025a50d1024e81d0178ea993df2dbaa2ba237c64a36890"
Sep 11 23:06:55.604019 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:06:55.604000    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cda1c095880367248025a50d1024e81d0178ea993df2dbaa2ba237c64a36890\": container with ID starting with 9cda1c095880367248025a50d1024e81d0178ea993df2dbaa2ba237c64a36890 not found: ID does not exist" containerID="9cda1c095880367248025a50d1024e81d0178ea993df2dbaa2ba237c64a36890"
Sep 11 23:06:55.604070 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:55.604029    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cda1c095880367248025a50d1024e81d0178ea993df2dbaa2ba237c64a36890"} err="failed to get container status \"9cda1c095880367248025a50d1024e81d0178ea993df2dbaa2ba237c64a36890\": rpc error: code = NotFound desc = could not find container \"9cda1c095880367248025a50d1024e81d0178ea993df2dbaa2ba237c64a36890\": container with ID starting with 9cda1c095880367248025a50d1024e81d0178ea993df2dbaa2ba237c64a36890 not found: ID does not exist"
Sep 11 23:06:55.616365 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:55.616337    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5c8f46447c-cdpct"]
Sep 11 23:06:55.619864 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:55.619841    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5c8f46447c-cdpct"]
Sep 11 23:06:56.493111 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:56.493070    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ea4ca4e-b321-4324-b380-dab72513b02a" path="/var/lib/kubelet/pods/8ea4ca4e-b321-4324-b380-dab72513b02a/volumes"
Sep 11 23:06:56.600585 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:56.600548    2546 generic.go:356] "Generic (PLEG): container finished" podID="23f6c775-d2ea-4c5f-8bd4-da89b88950c0" containerID="5aa67f351362f69b32e007d39408f9e1de1a63fa57784d7930e3e678fa227fbc" exitCode=0
Sep 11 23:06:56.600744 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:56.600619    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-8psvk" event={"ID":"23f6c775-d2ea-4c5f-8bd4-da89b88950c0","Type":"ContainerDied","Data":"5aa67f351362f69b32e007d39408f9e1de1a63fa57784d7930e3e678fa227fbc"}
Sep 11 23:06:56.600954 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:56.600940    2546 scope.go:117] "RemoveContainer" containerID="5aa67f351362f69b32e007d39408f9e1de1a63fa57784d7930e3e678fa227fbc"
Sep 11 23:06:57.606639 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:06:57.606603    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-8psvk" event={"ID":"23f6c775-d2ea-4c5f-8bd4-da89b88950c0","Type":"ContainerStarted","Data":"17af706a6720e8456a5f112625d53cae357200259a0719697194eda0e8a72de6"}
Sep 11 23:07:11.660050 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:07:11.660016    2546 generic.go:356] "Generic (PLEG): container finished" podID="68e4123a-25ea-4c58-a934-d44495740042" containerID="ec72a8e3f9470e5ad68816fc7bebf42c841018de3178c945f0492b503ead0b5d" exitCode=0
Sep 11 23:07:11.660492 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:07:11.660093    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-xr6z6" event={"ID":"68e4123a-25ea-4c58-a934-d44495740042","Type":"ContainerDied","Data":"ec72a8e3f9470e5ad68816fc7bebf42c841018de3178c945f0492b503ead0b5d"}
Sep 11 23:07:11.660492 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:07:11.660428    2546 scope.go:117] "RemoveContainer" containerID="ec72a8e3f9470e5ad68816fc7bebf42c841018de3178c945f0492b503ead0b5d"
Sep 11 23:07:12.666257 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:07:12.666223    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-xr6z6" event={"ID":"68e4123a-25ea-4c58-a934-d44495740042","Type":"ContainerStarted","Data":"9140b4bff7858a90953d532ae442b1072d3f701bd7de8761384a76f4068457b7"}
Sep 11 23:07:29.406176 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:07:29.406113    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50607ccd-47f9-463a-bdc7-181484a00caa-metrics-certs\") pod \"network-metrics-daemon-vbgsz\" (UID: \"50607ccd-47f9-463a-bdc7-181484a00caa\") " pod="openshift-multus/network-metrics-daemon-vbgsz"
Sep 11 23:07:29.408672 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:07:29.408646    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50607ccd-47f9-463a-bdc7-181484a00caa-metrics-certs\") pod \"network-metrics-daemon-vbgsz\" (UID: \"50607ccd-47f9-463a-bdc7-181484a00caa\") " pod="openshift-multus/network-metrics-daemon-vbgsz"
Sep 11 23:07:29.491269 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:07:29.491232    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-rtlm9\""
Sep 11 23:07:29.498513 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:07:29.498491    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vbgsz"
Sep 11 23:07:29.781233 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:07:29.781208    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vbgsz"]
Sep 11 23:07:29.783821 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:07:29.783781    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50607ccd_47f9_463a_bdc7_181484a00caa.slice/crio-b9b4affc9accb74b45df50782996aab806c6cef998c557e542089c6e8518e702 WatchSource:0}: Error finding container b9b4affc9accb74b45df50782996aab806c6cef998c557e542089c6e8518e702: Status 404 returned error can't find the container with id b9b4affc9accb74b45df50782996aab806c6cef998c557e542089c6e8518e702
Sep 11 23:07:30.744005 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:07:30.743959    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vbgsz" event={"ID":"50607ccd-47f9-463a-bdc7-181484a00caa","Type":"ContainerStarted","Data":"b9b4affc9accb74b45df50782996aab806c6cef998c557e542089c6e8518e702"}
Sep 11 23:07:31.750087 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:07:31.750047    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vbgsz" event={"ID":"50607ccd-47f9-463a-bdc7-181484a00caa","Type":"ContainerStarted","Data":"4fc6545396e2b08c1f8c2903f96ba39697bae8545b763717ed595dc9029e9aa4"}
Sep 11 23:07:31.750610 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:07:31.750119    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vbgsz" event={"ID":"50607ccd-47f9-463a-bdc7-181484a00caa","Type":"ContainerStarted","Data":"37b70618977329a010ce854c61d5bb876784a3a76edc57a28a567f43fb6db242"}
Sep 11 23:07:31.770248 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:07:31.770192    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-vbgsz" podStartSLOduration=374.710945457 podStartE2EDuration="6m15.770144474s" totalImagesPullingTime="1.059199017s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 23:01:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:07:31.768916757 +0000 UTC m=+375.802045977" watchObservedRunningTime="2026-09-11 23:07:31.770144474 +0000 UTC m=+375.803273694"
Sep 11 23:07:58.355684 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:07:58.355640    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-c5b4b88f5-b89zk"]
Sep 11 23:07:58.356277 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:07:58.356093    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ea4ca4e-b321-4324-b380-dab72513b02a" containerName="registry"
Sep 11 23:07:58.356277 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:07:58.356112    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ea4ca4e-b321-4324-b380-dab72513b02a" containerName="registry"
Sep 11 23:07:58.356277 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:07:58.356227    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ea4ca4e-b321-4324-b380-dab72513b02a" containerName="registry"
Sep 11 23:07:58.361135 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:07:58.361113    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-c5b4b88f5-b89zk"
Sep 11 23:07:58.364404 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:07:58.364378    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 11 23:07:58.364539 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:07:58.364412    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-d6vdh\""
Sep 11 23:07:58.364539 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:07:58.364433    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 11 23:07:58.364539 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:07:58.364441    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 11 23:07:58.364539 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:07:58.364412    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 11 23:07:58.364539 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:07:58.364514    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 11 23:07:58.370975 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:07:58.370954    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 11 23:07:58.371834 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:07:58.371816    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-c5b4b88f5-b89zk"]
Sep 11 23:07:58.422808 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:07:58.422774    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/9e92a8c5-2c4c-4853-9934-d1034caac9a2-secret-telemeter-client\") pod \"telemeter-client-c5b4b88f5-b89zk\" (UID: \"9e92a8c5-2c4c-4853-9934-d1034caac9a2\") " pod="openshift-monitoring/telemeter-client-c5b4b88f5-b89zk"
Sep 11 23:07:58.422967 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:07:58.422825    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e92a8c5-2c4c-4853-9934-d1034caac9a2-telemeter-trusted-ca-bundle\") pod \"telemeter-client-c5b4b88f5-b89zk\" (UID: \"9e92a8c5-2c4c-4853-9934-d1034caac9a2\") " pod="openshift-monitoring/telemeter-client-c5b4b88f5-b89zk"
Sep 11 23:07:58.422967 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:07:58.422903    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e92a8c5-2c4c-4853-9934-d1034caac9a2-serving-certs-ca-bundle\") pod \"telemeter-client-c5b4b88f5-b89zk\" (UID: \"9e92a8c5-2c4c-4853-9934-d1034caac9a2\") " pod="openshift-monitoring/telemeter-client-c5b4b88f5-b89zk"
Sep 11 23:07:58.422967 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:07:58.422920    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9e92a8c5-2c4c-4853-9934-d1034caac9a2-metrics-client-ca\") pod \"telemeter-client-c5b4b88f5-b89zk\" (UID: \"9e92a8c5-2c4c-4853-9934-d1034caac9a2\") " pod="openshift-monitoring/telemeter-client-c5b4b88f5-b89zk"
Sep 11 23:07:58.422967 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:07:58.422945    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/9e92a8c5-2c4c-4853-9934-d1034caac9a2-federate-client-tls\") pod \"telemeter-client-c5b4b88f5-b89zk\" (UID: \"9e92a8c5-2c4c-4853-9934-d1034caac9a2\") " pod="openshift-monitoring/telemeter-client-c5b4b88f5-b89zk"
Sep 11 23:07:58.423097 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:07:58.422967    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnz8k\" (UniqueName: \"kubernetes.io/projected/9e92a8c5-2c4c-4853-9934-d1034caac9a2-kube-api-access-hnz8k\") pod \"telemeter-client-c5b4b88f5-b89zk\" (UID: \"9e92a8c5-2c4c-4853-9934-d1034caac9a2\") " pod="openshift-monitoring/telemeter-client-c5b4b88f5-b89zk"
Sep 11 23:07:58.423097 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:07:58.422992    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9e92a8c5-2c4c-4853-9934-d1034caac9a2-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-c5b4b88f5-b89zk\" (UID: \"9e92a8c5-2c4c-4853-9934-d1034caac9a2\") " pod="openshift-monitoring/telemeter-client-c5b4b88f5-b89zk"
Sep 11 23:07:58.423097 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:07:58.423017    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/9e92a8c5-2c4c-4853-9934-d1034caac9a2-telemeter-client-tls\") pod \"telemeter-client-c5b4b88f5-b89zk\" (UID: \"9e92a8c5-2c4c-4853-9934-d1034caac9a2\") " pod="openshift-monitoring/telemeter-client-c5b4b88f5-b89zk"
Sep 11 23:07:58.523454 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:07:58.523422    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e92a8c5-2c4c-4853-9934-d1034caac9a2-serving-certs-ca-bundle\") pod \"telemeter-client-c5b4b88f5-b89zk\" (UID: \"9e92a8c5-2c4c-4853-9934-d1034caac9a2\") " pod="openshift-monitoring/telemeter-client-c5b4b88f5-b89zk"
Sep 11 23:07:58.523454 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:07:58.523455    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9e92a8c5-2c4c-4853-9934-d1034caac9a2-metrics-client-ca\") pod \"telemeter-client-c5b4b88f5-b89zk\" (UID: \"9e92a8c5-2c4c-4853-9934-d1034caac9a2\") " pod="openshift-monitoring/telemeter-client-c5b4b88f5-b89zk"
Sep 11 23:07:58.523671 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:07:58.523491    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/9e92a8c5-2c4c-4853-9934-d1034caac9a2-federate-client-tls\") pod \"telemeter-client-c5b4b88f5-b89zk\" (UID: \"9e92a8c5-2c4c-4853-9934-d1034caac9a2\") " pod="openshift-monitoring/telemeter-client-c5b4b88f5-b89zk"
Sep 11 23:07:58.523671 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:07:58.523631    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnz8k\" (UniqueName: \"kubernetes.io/projected/9e92a8c5-2c4c-4853-9934-d1034caac9a2-kube-api-access-hnz8k\") pod \"telemeter-client-c5b4b88f5-b89zk\" (UID: \"9e92a8c5-2c4c-4853-9934-d1034caac9a2\") " pod="openshift-monitoring/telemeter-client-c5b4b88f5-b89zk"
Sep 11 23:07:58.523774 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:07:58.523687    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9e92a8c5-2c4c-4853-9934-d1034caac9a2-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-c5b4b88f5-b89zk\" (UID: \"9e92a8c5-2c4c-4853-9934-d1034caac9a2\") " pod="openshift-monitoring/telemeter-client-c5b4b88f5-b89zk"
Sep 11 23:07:58.523774 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:07:58.523732    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/9e92a8c5-2c4c-4853-9934-d1034caac9a2-telemeter-client-tls\") pod \"telemeter-client-c5b4b88f5-b89zk\" (UID: \"9e92a8c5-2c4c-4853-9934-d1034caac9a2\") " pod="openshift-monitoring/telemeter-client-c5b4b88f5-b89zk"
Sep 11 23:07:58.523879 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:07:58.523781    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/9e92a8c5-2c4c-4853-9934-d1034caac9a2-secret-telemeter-client\") pod \"telemeter-client-c5b4b88f5-b89zk\" (UID: \"9e92a8c5-2c4c-4853-9934-d1034caac9a2\") " pod="openshift-monitoring/telemeter-client-c5b4b88f5-b89zk"
Sep 11 23:07:58.523879 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:07:58.523845    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e92a8c5-2c4c-4853-9934-d1034caac9a2-telemeter-trusted-ca-bundle\") pod \"telemeter-client-c5b4b88f5-b89zk\" (UID: \"9e92a8c5-2c4c-4853-9934-d1034caac9a2\") " pod="openshift-monitoring/telemeter-client-c5b4b88f5-b89zk"
Sep 11 23:07:58.524339 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:07:58.524285    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e92a8c5-2c4c-4853-9934-d1034caac9a2-serving-certs-ca-bundle\") pod \"telemeter-client-c5b4b88f5-b89zk\" (UID: \"9e92a8c5-2c4c-4853-9934-d1034caac9a2\") " pod="openshift-monitoring/telemeter-client-c5b4b88f5-b89zk"
Sep 11 23:07:58.524673 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:07:58.524347    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9e92a8c5-2c4c-4853-9934-d1034caac9a2-metrics-client-ca\") pod \"telemeter-client-c5b4b88f5-b89zk\" (UID: \"9e92a8c5-2c4c-4853-9934-d1034caac9a2\") " pod="openshift-monitoring/telemeter-client-c5b4b88f5-b89zk"
Sep 11 23:07:58.524799 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:07:58.524723    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e92a8c5-2c4c-4853-9934-d1034caac9a2-telemeter-trusted-ca-bundle\") pod \"telemeter-client-c5b4b88f5-b89zk\" (UID: \"9e92a8c5-2c4c-4853-9934-d1034caac9a2\") " pod="openshift-monitoring/telemeter-client-c5b4b88f5-b89zk"
Sep 11 23:07:58.526225 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:07:58.526206    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/9e92a8c5-2c4c-4853-9934-d1034caac9a2-federate-client-tls\") pod \"telemeter-client-c5b4b88f5-b89zk\" (UID: \"9e92a8c5-2c4c-4853-9934-d1034caac9a2\") " pod="openshift-monitoring/telemeter-client-c5b4b88f5-b89zk"
Sep 11 23:07:58.526433 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:07:58.526412    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/9e92a8c5-2c4c-4853-9934-d1034caac9a2-telemeter-client-tls\") pod \"telemeter-client-c5b4b88f5-b89zk\" (UID: \"9e92a8c5-2c4c-4853-9934-d1034caac9a2\") " pod="openshift-monitoring/telemeter-client-c5b4b88f5-b89zk"
Sep 11 23:07:58.526765 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:07:58.526747    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9e92a8c5-2c4c-4853-9934-d1034caac9a2-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-c5b4b88f5-b89zk\" (UID: \"9e92a8c5-2c4c-4853-9934-d1034caac9a2\") " pod="openshift-monitoring/telemeter-client-c5b4b88f5-b89zk"
Sep 11 23:07:58.526832 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:07:58.526794    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/9e92a8c5-2c4c-4853-9934-d1034caac9a2-secret-telemeter-client\") pod \"telemeter-client-c5b4b88f5-b89zk\" (UID: \"9e92a8c5-2c4c-4853-9934-d1034caac9a2\") " pod="openshift-monitoring/telemeter-client-c5b4b88f5-b89zk"
Sep 11 23:07:58.532185 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:07:58.532165    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnz8k\" (UniqueName: \"kubernetes.io/projected/9e92a8c5-2c4c-4853-9934-d1034caac9a2-kube-api-access-hnz8k\") pod \"telemeter-client-c5b4b88f5-b89zk\" (UID: \"9e92a8c5-2c4c-4853-9934-d1034caac9a2\") " pod="openshift-monitoring/telemeter-client-c5b4b88f5-b89zk"
Sep 11 23:07:58.671701 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:07:58.671673    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-c5b4b88f5-b89zk"
Sep 11 23:07:58.955401 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:07:58.955371    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-c5b4b88f5-b89zk"]
Sep 11 23:07:58.956378 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:07:58.956351    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e92a8c5_2c4c_4853_9934_d1034caac9a2.slice/crio-a9f7ec7be3e16b8e6eabebabe56c713e4f3e4a340c08140c3ffc75e671c8277c WatchSource:0}: Error finding container a9f7ec7be3e16b8e6eabebabe56c713e4f3e4a340c08140c3ffc75e671c8277c: Status 404 returned error can't find the container with id a9f7ec7be3e16b8e6eabebabe56c713e4f3e4a340c08140c3ffc75e671c8277c
Sep 11 23:07:59.862294 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:07:59.862255    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-c5b4b88f5-b89zk" event={"ID":"9e92a8c5-2c4c-4853-9934-d1034caac9a2","Type":"ContainerStarted","Data":"a9f7ec7be3e16b8e6eabebabe56c713e4f3e4a340c08140c3ffc75e671c8277c"}
Sep 11 23:08:01.872496 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:08:01.872453    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-c5b4b88f5-b89zk" event={"ID":"9e92a8c5-2c4c-4853-9934-d1034caac9a2","Type":"ContainerStarted","Data":"c095a84c4b7a1d7c052852c48d0d46780b02a6e8b6ef060dacd0fa07317cc597"}
Sep 11 23:08:02.879128 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:08:02.879093    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-c5b4b88f5-b89zk" event={"ID":"9e92a8c5-2c4c-4853-9934-d1034caac9a2","Type":"ContainerStarted","Data":"6a5ba1f7d187c64d5da31403cd636ba90548ba2be47a12212bccf6940ef92089"}
Sep 11 23:08:02.879526 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:08:02.879174    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-c5b4b88f5-b89zk" event={"ID":"9e92a8c5-2c4c-4853-9934-d1034caac9a2","Type":"ContainerStarted","Data":"7b211f855de57d916222a6f2f93ad8056fe6fa8f0eaa2ce299d5508406b2c857"}
Sep 11 23:08:02.905253 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:08:02.905199    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-c5b4b88f5-b89zk" podStartSLOduration=1.7158246080000001 podStartE2EDuration="4.905181838s" totalImagesPullingTime="3.18935723s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 23:07:58 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:08:02.902616335 +0000 UTC m=+406.935745553" watchObservedRunningTime="2026-09-11 23:08:02.905181838 +0000 UTC m=+406.938311055"
Sep 11 23:10:22.686642 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:22.686564    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lffpr"]
Sep 11 23:10:22.689898 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:22.689881    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lffpr"
Sep 11 23:10:22.692971 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:22.692948    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 11 23:10:22.693118 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:22.692948    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 11 23:10:22.694120 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:22.694106    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-qbxpl\""
Sep 11 23:10:22.699999 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:22.699976    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lffpr"]
Sep 11 23:10:22.779638 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:22.779598    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm6zh\" (UniqueName: \"kubernetes.io/projected/8fc1f556-e454-462f-9c5f-ecc3cbd5c739-kube-api-access-fm6zh\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lffpr\" (UID: \"8fc1f556-e454-462f-9c5f-ecc3cbd5c739\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lffpr"
Sep 11 23:10:22.779819 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:22.779648    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8fc1f556-e454-462f-9c5f-ecc3cbd5c739-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lffpr\" (UID: \"8fc1f556-e454-462f-9c5f-ecc3cbd5c739\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lffpr"
Sep 11 23:10:22.779819 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:22.779719    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8fc1f556-e454-462f-9c5f-ecc3cbd5c739-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lffpr\" (UID: \"8fc1f556-e454-462f-9c5f-ecc3cbd5c739\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lffpr"
Sep 11 23:10:22.881142 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:22.881101    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm6zh\" (UniqueName: \"kubernetes.io/projected/8fc1f556-e454-462f-9c5f-ecc3cbd5c739-kube-api-access-fm6zh\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lffpr\" (UID: \"8fc1f556-e454-462f-9c5f-ecc3cbd5c739\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lffpr"
Sep 11 23:10:22.881330 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:22.881178    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8fc1f556-e454-462f-9c5f-ecc3cbd5c739-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lffpr\" (UID: \"8fc1f556-e454-462f-9c5f-ecc3cbd5c739\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lffpr"
Sep 11 23:10:22.881330 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:22.881215    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8fc1f556-e454-462f-9c5f-ecc3cbd5c739-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lffpr\" (UID: \"8fc1f556-e454-462f-9c5f-ecc3cbd5c739\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lffpr"
Sep 11 23:10:22.881700 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:22.881677    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8fc1f556-e454-462f-9c5f-ecc3cbd5c739-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lffpr\" (UID: \"8fc1f556-e454-462f-9c5f-ecc3cbd5c739\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lffpr"
Sep 11 23:10:22.881736 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:22.881692    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8fc1f556-e454-462f-9c5f-ecc3cbd5c739-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lffpr\" (UID: \"8fc1f556-e454-462f-9c5f-ecc3cbd5c739\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lffpr"
Sep 11 23:10:22.890889 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:22.890857    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm6zh\" (UniqueName: \"kubernetes.io/projected/8fc1f556-e454-462f-9c5f-ecc3cbd5c739-kube-api-access-fm6zh\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lffpr\" (UID: \"8fc1f556-e454-462f-9c5f-ecc3cbd5c739\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lffpr"
Sep 11 23:10:22.998987 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:22.998900    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lffpr"
Sep 11 23:10:23.275722 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:23.275577    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lffpr"]
Sep 11 23:10:23.278027 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:10:23.277997    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fc1f556_e454_462f_9c5f_ecc3cbd5c739.slice/crio-ba27668d60c5e2a9726fb97fa29a4541979681a13dd6c6bbb044278ac99811a8 WatchSource:0}: Error finding container ba27668d60c5e2a9726fb97fa29a4541979681a13dd6c6bbb044278ac99811a8: Status 404 returned error can't find the container with id ba27668d60c5e2a9726fb97fa29a4541979681a13dd6c6bbb044278ac99811a8
Sep 11 23:10:23.449053 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:23.449019    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lffpr" event={"ID":"8fc1f556-e454-462f-9c5f-ecc3cbd5c739","Type":"ContainerStarted","Data":"ba27668d60c5e2a9726fb97fa29a4541979681a13dd6c6bbb044278ac99811a8"}
Sep 11 23:10:30.483320 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:30.483285    2546 generic.go:356] "Generic (PLEG): container finished" podID="8fc1f556-e454-462f-9c5f-ecc3cbd5c739" containerID="ce8a321c4fb5657bdf65afc2427459fe3336df0d3483cba6147b9779c9bf125f" exitCode=0
Sep 11 23:10:30.483708 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:30.483350    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lffpr" event={"ID":"8fc1f556-e454-462f-9c5f-ecc3cbd5c739","Type":"ContainerDied","Data":"ce8a321c4fb5657bdf65afc2427459fe3336df0d3483cba6147b9779c9bf125f"}
Sep 11 23:10:33.499505 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:33.499469    2546 generic.go:356] "Generic (PLEG): container finished" podID="8fc1f556-e454-462f-9c5f-ecc3cbd5c739" containerID="d2217b88cfb54205c1c3f4a770c9d8e4a2073dc92a8ad352c247795e8ccc4c3d" exitCode=0
Sep 11 23:10:33.499898 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:33.499516    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lffpr" event={"ID":"8fc1f556-e454-462f-9c5f-ecc3cbd5c739","Type":"ContainerDied","Data":"d2217b88cfb54205c1c3f4a770c9d8e4a2073dc92a8ad352c247795e8ccc4c3d"}
Sep 11 23:10:40.534616 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:40.534581    2546 generic.go:356] "Generic (PLEG): container finished" podID="8fc1f556-e454-462f-9c5f-ecc3cbd5c739" containerID="03003c46fa1cbf82f1d0c18e8eca2d264bb663a103731f25675f3caa73684f26" exitCode=0
Sep 11 23:10:40.534955 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:40.534656    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lffpr" event={"ID":"8fc1f556-e454-462f-9c5f-ecc3cbd5c739","Type":"ContainerDied","Data":"03003c46fa1cbf82f1d0c18e8eca2d264bb663a103731f25675f3caa73684f26"}
Sep 11 23:10:41.666747 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:41.666718    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lffpr"
Sep 11 23:10:41.748408 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:41.748367    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8fc1f556-e454-462f-9c5f-ecc3cbd5c739-bundle\") pod \"8fc1f556-e454-462f-9c5f-ecc3cbd5c739\" (UID: \"8fc1f556-e454-462f-9c5f-ecc3cbd5c739\") "
Sep 11 23:10:41.748582 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:41.748440    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fm6zh\" (UniqueName: \"kubernetes.io/projected/8fc1f556-e454-462f-9c5f-ecc3cbd5c739-kube-api-access-fm6zh\") pod \"8fc1f556-e454-462f-9c5f-ecc3cbd5c739\" (UID: \"8fc1f556-e454-462f-9c5f-ecc3cbd5c739\") "
Sep 11 23:10:41.748582 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:41.748458    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8fc1f556-e454-462f-9c5f-ecc3cbd5c739-util\") pod \"8fc1f556-e454-462f-9c5f-ecc3cbd5c739\" (UID: \"8fc1f556-e454-462f-9c5f-ecc3cbd5c739\") "
Sep 11 23:10:41.749057 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:41.749031    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fc1f556-e454-462f-9c5f-ecc3cbd5c739-bundle" (OuterVolumeSpecName: "bundle") pod "8fc1f556-e454-462f-9c5f-ecc3cbd5c739" (UID: "8fc1f556-e454-462f-9c5f-ecc3cbd5c739"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:10:41.750817 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:41.750793    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fc1f556-e454-462f-9c5f-ecc3cbd5c739-kube-api-access-fm6zh" (OuterVolumeSpecName: "kube-api-access-fm6zh") pod "8fc1f556-e454-462f-9c5f-ecc3cbd5c739" (UID: "8fc1f556-e454-462f-9c5f-ecc3cbd5c739"). InnerVolumeSpecName "kube-api-access-fm6zh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 23:10:41.753980 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:41.753949    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fc1f556-e454-462f-9c5f-ecc3cbd5c739-util" (OuterVolumeSpecName: "util") pod "8fc1f556-e454-462f-9c5f-ecc3cbd5c739" (UID: "8fc1f556-e454-462f-9c5f-ecc3cbd5c739"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:10:41.849738 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:41.849640    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fm6zh\" (UniqueName: \"kubernetes.io/projected/8fc1f556-e454-462f-9c5f-ecc3cbd5c739-kube-api-access-fm6zh\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:10:41.849738 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:41.849676    2546 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8fc1f556-e454-462f-9c5f-ecc3cbd5c739-util\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:10:41.849738 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:41.849686    2546 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8fc1f556-e454-462f-9c5f-ecc3cbd5c739-bundle\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:10:42.545938 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:42.545909    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lffpr"
Sep 11 23:10:42.546101 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:42.545905    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lffpr" event={"ID":"8fc1f556-e454-462f-9c5f-ecc3cbd5c739","Type":"ContainerDied","Data":"ba27668d60c5e2a9726fb97fa29a4541979681a13dd6c6bbb044278ac99811a8"}
Sep 11 23:10:42.546101 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:42.546038    2546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba27668d60c5e2a9726fb97fa29a4541979681a13dd6c6bbb044278ac99811a8"
Sep 11 23:10:45.854014 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:45.853977    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-rr9rg"]
Sep 11 23:10:45.854499 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:45.854330    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fc1f556-e454-462f-9c5f-ecc3cbd5c739" containerName="pull"
Sep 11 23:10:45.854499 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:45.854342    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fc1f556-e454-462f-9c5f-ecc3cbd5c739" containerName="pull"
Sep 11 23:10:45.854499 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:45.854355    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fc1f556-e454-462f-9c5f-ecc3cbd5c739" containerName="util"
Sep 11 23:10:45.854499 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:45.854360    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fc1f556-e454-462f-9c5f-ecc3cbd5c739" containerName="util"
Sep 11 23:10:45.854499 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:45.854368    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fc1f556-e454-462f-9c5f-ecc3cbd5c739" containerName="extract"
Sep 11 23:10:45.854499 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:45.854373    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fc1f556-e454-462f-9c5f-ecc3cbd5c739" containerName="extract"
Sep 11 23:10:45.854499 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:45.854434    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fc1f556-e454-462f-9c5f-ecc3cbd5c739" containerName="extract"
Sep 11 23:10:45.887266 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:45.887220    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-rr9rg"]
Sep 11 23:10:45.887451 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:45.887358    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-rr9rg"
Sep 11 23:10:45.891296 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:45.891275    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 11 23:10:45.891612 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:45.891598    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 11 23:10:45.891750 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:45.891732    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-588xm\""
Sep 11 23:10:45.983510 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:45.983469    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2td2\" (UniqueName: \"kubernetes.io/projected/5afcaea2-dfd5-4008-b031-00b1710f88e6-kube-api-access-b2td2\") pod \"cert-manager-operator-controller-manager-55c444bf86-rr9rg\" (UID: \"5afcaea2-dfd5-4008-b031-00b1710f88e6\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-rr9rg"
Sep 11 23:10:45.983693 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:45.983554    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5afcaea2-dfd5-4008-b031-00b1710f88e6-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-rr9rg\" (UID: \"5afcaea2-dfd5-4008-b031-00b1710f88e6\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-rr9rg"
Sep 11 23:10:46.084067 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:46.084037    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2td2\" (UniqueName: \"kubernetes.io/projected/5afcaea2-dfd5-4008-b031-00b1710f88e6-kube-api-access-b2td2\") pod \"cert-manager-operator-controller-manager-55c444bf86-rr9rg\" (UID: \"5afcaea2-dfd5-4008-b031-00b1710f88e6\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-rr9rg"
Sep 11 23:10:46.084246 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:46.084098    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5afcaea2-dfd5-4008-b031-00b1710f88e6-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-rr9rg\" (UID: \"5afcaea2-dfd5-4008-b031-00b1710f88e6\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-rr9rg"
Sep 11 23:10:46.084538 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:46.084519    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5afcaea2-dfd5-4008-b031-00b1710f88e6-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-rr9rg\" (UID: \"5afcaea2-dfd5-4008-b031-00b1710f88e6\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-rr9rg"
Sep 11 23:10:46.094106 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:46.094070    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2td2\" (UniqueName: \"kubernetes.io/projected/5afcaea2-dfd5-4008-b031-00b1710f88e6-kube-api-access-b2td2\") pod \"cert-manager-operator-controller-manager-55c444bf86-rr9rg\" (UID: \"5afcaea2-dfd5-4008-b031-00b1710f88e6\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-rr9rg"
Sep 11 23:10:46.196778 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:46.196732    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-rr9rg"
Sep 11 23:10:46.506422 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:46.506394    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-rr9rg"]
Sep 11 23:10:46.507681 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:10:46.507655    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5afcaea2_dfd5_4008_b031_00b1710f88e6.slice/crio-e059b4831877e03b7f443bd1801b432206e9c128833f4ec0b475649b572b1868 WatchSource:0}: Error finding container e059b4831877e03b7f443bd1801b432206e9c128833f4ec0b475649b572b1868: Status 404 returned error can't find the container with id e059b4831877e03b7f443bd1801b432206e9c128833f4ec0b475649b572b1868
Sep 11 23:10:46.571082 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:46.571045    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-rr9rg" event={"ID":"5afcaea2-dfd5-4008-b031-00b1710f88e6","Type":"ContainerStarted","Data":"e059b4831877e03b7f443bd1801b432206e9c128833f4ec0b475649b572b1868"}
Sep 11 23:10:48.582246 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:48.582213    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-rr9rg" event={"ID":"5afcaea2-dfd5-4008-b031-00b1710f88e6","Type":"ContainerStarted","Data":"1f2032b7eb05f94311dca598e24f0b1619efa9e50816815c34452e029db550f0"}
Sep 11 23:10:48.604133 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:48.604020    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-rr9rg" podStartSLOduration=1.654442874 podStartE2EDuration="3.603997053s" totalImagesPullingTime="1.949554179s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 23:10:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:10:48.603086751 +0000 UTC m=+572.636215967" watchObservedRunningTime="2026-09-11 23:10:48.603997053 +0000 UTC m=+572.637126273"
Sep 11 23:10:49.939022 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:49.938980    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmbhgw"]
Sep 11 23:10:49.942751 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:49.942732    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmbhgw"
Sep 11 23:10:49.945900 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:49.945876    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 11 23:10:49.946038 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:49.945873    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 11 23:10:49.947299 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:49.947274    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-qbxpl\""
Sep 11 23:10:49.951826 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:49.951792    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmbhgw"]
Sep 11 23:10:50.117704 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:50.117667    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3de5e6e3-6114-42ef-9f12-bad0c68aaca2-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmbhgw\" (UID: \"3de5e6e3-6114-42ef-9f12-bad0c68aaca2\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmbhgw"
Sep 11 23:10:50.117704 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:50.117709    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d5mx\" (UniqueName: \"kubernetes.io/projected/3de5e6e3-6114-42ef-9f12-bad0c68aaca2-kube-api-access-8d5mx\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmbhgw\" (UID: \"3de5e6e3-6114-42ef-9f12-bad0c68aaca2\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmbhgw"
Sep 11 23:10:50.117933 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:50.117730    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3de5e6e3-6114-42ef-9f12-bad0c68aaca2-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmbhgw\" (UID: \"3de5e6e3-6114-42ef-9f12-bad0c68aaca2\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmbhgw"
Sep 11 23:10:50.218552 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:50.218471    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3de5e6e3-6114-42ef-9f12-bad0c68aaca2-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmbhgw\" (UID: \"3de5e6e3-6114-42ef-9f12-bad0c68aaca2\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmbhgw"
Sep 11 23:10:50.218552 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:50.218509    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d5mx\" (UniqueName: \"kubernetes.io/projected/3de5e6e3-6114-42ef-9f12-bad0c68aaca2-kube-api-access-8d5mx\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmbhgw\" (UID: \"3de5e6e3-6114-42ef-9f12-bad0c68aaca2\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmbhgw"
Sep 11 23:10:50.218552 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:50.218531    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3de5e6e3-6114-42ef-9f12-bad0c68aaca2-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmbhgw\" (UID: \"3de5e6e3-6114-42ef-9f12-bad0c68aaca2\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmbhgw"
Sep 11 23:10:50.218920 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:50.218898    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3de5e6e3-6114-42ef-9f12-bad0c68aaca2-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmbhgw\" (UID: \"3de5e6e3-6114-42ef-9f12-bad0c68aaca2\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmbhgw"
Sep 11 23:10:50.218985 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:50.218934    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3de5e6e3-6114-42ef-9f12-bad0c68aaca2-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmbhgw\" (UID: \"3de5e6e3-6114-42ef-9f12-bad0c68aaca2\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmbhgw"
Sep 11 23:10:50.227996 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:50.227969    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d5mx\" (UniqueName: \"kubernetes.io/projected/3de5e6e3-6114-42ef-9f12-bad0c68aaca2-kube-api-access-8d5mx\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmbhgw\" (UID: \"3de5e6e3-6114-42ef-9f12-bad0c68aaca2\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmbhgw"
Sep 11 23:10:50.252992 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:50.252952    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmbhgw"
Sep 11 23:10:50.453997 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:50.453967    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-l48qw"]
Sep 11 23:10:50.457506 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:50.457484    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-l48qw"
Sep 11 23:10:50.460529 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:50.460507    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 11 23:10:50.461996 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:50.461978    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-74kqr\""
Sep 11 23:10:50.461996 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:50.461995    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 11 23:10:50.466250 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:50.466229    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-l48qw"]
Sep 11 23:10:50.527824 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:50.527799    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmbhgw"]
Sep 11 23:10:50.530166 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:10:50.530119    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3de5e6e3_6114_42ef_9f12_bad0c68aaca2.slice/crio-ff41d02a85f2018eca16a22c5ca43a1e6e37a0d47e664c1506cb7438d942d252 WatchSource:0}: Error finding container ff41d02a85f2018eca16a22c5ca43a1e6e37a0d47e664c1506cb7438d942d252: Status 404 returned error can't find the container with id ff41d02a85f2018eca16a22c5ca43a1e6e37a0d47e664c1506cb7438d942d252
Sep 11 23:10:50.599478 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:50.599441    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmbhgw" event={"ID":"3de5e6e3-6114-42ef-9f12-bad0c68aaca2","Type":"ContainerStarted","Data":"59ed32f80fd0e7f80f14843ae1677ab2cdc04149742b345b3906309f1bc4ca67"}
Sep 11 23:10:50.599634 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:50.599508    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmbhgw" event={"ID":"3de5e6e3-6114-42ef-9f12-bad0c68aaca2","Type":"ContainerStarted","Data":"ff41d02a85f2018eca16a22c5ca43a1e6e37a0d47e664c1506cb7438d942d252"}
Sep 11 23:10:50.623204 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:50.623141    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9ffe714c-1d4b-41a5-bc45-5031152887d5-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-l48qw\" (UID: \"9ffe714c-1d4b-41a5-bc45-5031152887d5\") " pod="cert-manager/cert-manager-webhook-6486ffd997-l48qw"
Sep 11 23:10:50.623387 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:50.623270    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9f2s\" (UniqueName: \"kubernetes.io/projected/9ffe714c-1d4b-41a5-bc45-5031152887d5-kube-api-access-q9f2s\") pod \"cert-manager-webhook-6486ffd997-l48qw\" (UID: \"9ffe714c-1d4b-41a5-bc45-5031152887d5\") " pod="cert-manager/cert-manager-webhook-6486ffd997-l48qw"
Sep 11 23:10:50.724548 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:50.724511    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9f2s\" (UniqueName: \"kubernetes.io/projected/9ffe714c-1d4b-41a5-bc45-5031152887d5-kube-api-access-q9f2s\") pod \"cert-manager-webhook-6486ffd997-l48qw\" (UID: \"9ffe714c-1d4b-41a5-bc45-5031152887d5\") " pod="cert-manager/cert-manager-webhook-6486ffd997-l48qw"
Sep 11 23:10:50.724756 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:50.724737    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9ffe714c-1d4b-41a5-bc45-5031152887d5-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-l48qw\" (UID: \"9ffe714c-1d4b-41a5-bc45-5031152887d5\") " pod="cert-manager/cert-manager-webhook-6486ffd997-l48qw"
Sep 11 23:10:50.733604 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:50.733578    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9ffe714c-1d4b-41a5-bc45-5031152887d5-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-l48qw\" (UID: \"9ffe714c-1d4b-41a5-bc45-5031152887d5\") " pod="cert-manager/cert-manager-webhook-6486ffd997-l48qw"
Sep 11 23:10:50.733767 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:50.733747    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9f2s\" (UniqueName: \"kubernetes.io/projected/9ffe714c-1d4b-41a5-bc45-5031152887d5-kube-api-access-q9f2s\") pod \"cert-manager-webhook-6486ffd997-l48qw\" (UID: \"9ffe714c-1d4b-41a5-bc45-5031152887d5\") " pod="cert-manager/cert-manager-webhook-6486ffd997-l48qw"
Sep 11 23:10:50.776546 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:50.776507    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-l48qw"
Sep 11 23:10:51.051658 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:51.051629    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-l48qw"]
Sep 11 23:10:51.053214 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:10:51.053185    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9ffe714c_1d4b_41a5_bc45_5031152887d5.slice/crio-1c3c3f61ef6e47ac191a0d63eaa425772ba51512c2dad9b2aa2d2e3c159f2fa6 WatchSource:0}: Error finding container 1c3c3f61ef6e47ac191a0d63eaa425772ba51512c2dad9b2aa2d2e3c159f2fa6: Status 404 returned error can't find the container with id 1c3c3f61ef6e47ac191a0d63eaa425772ba51512c2dad9b2aa2d2e3c159f2fa6
Sep 11 23:10:51.606488 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:51.606458    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-l48qw" event={"ID":"9ffe714c-1d4b-41a5-bc45-5031152887d5","Type":"ContainerStarted","Data":"1c3c3f61ef6e47ac191a0d63eaa425772ba51512c2dad9b2aa2d2e3c159f2fa6"}
Sep 11 23:10:51.608099 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:51.608068    2546 generic.go:356] "Generic (PLEG): container finished" podID="3de5e6e3-6114-42ef-9f12-bad0c68aaca2" containerID="59ed32f80fd0e7f80f14843ae1677ab2cdc04149742b345b3906309f1bc4ca67" exitCode=0
Sep 11 23:10:51.608233 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:51.608130    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmbhgw" event={"ID":"3de5e6e3-6114-42ef-9f12-bad0c68aaca2","Type":"ContainerDied","Data":"59ed32f80fd0e7f80f14843ae1677ab2cdc04149742b345b3906309f1bc4ca67"}
Sep 11 23:10:53.622758 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:53.622718    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmbhgw" event={"ID":"3de5e6e3-6114-42ef-9f12-bad0c68aaca2","Type":"ContainerStarted","Data":"b9ffb66c6c9c3e7f245820ee561cee2ad62b5498b91a395892b65cb60912c3ec"}
Sep 11 23:10:54.630499 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:54.630447    2546 generic.go:356] "Generic (PLEG): container finished" podID="3de5e6e3-6114-42ef-9f12-bad0c68aaca2" containerID="b9ffb66c6c9c3e7f245820ee561cee2ad62b5498b91a395892b65cb60912c3ec" exitCode=0
Sep 11 23:10:54.630906 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:54.630545    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmbhgw" event={"ID":"3de5e6e3-6114-42ef-9f12-bad0c68aaca2","Type":"ContainerDied","Data":"b9ffb66c6c9c3e7f245820ee561cee2ad62b5498b91a395892b65cb60912c3ec"}
Sep 11 23:10:55.638461 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:55.638424    2546 generic.go:356] "Generic (PLEG): container finished" podID="3de5e6e3-6114-42ef-9f12-bad0c68aaca2" containerID="1ce7d4cd821cf1c7a80c5c724ae5567838f081d910249dfe15baa9910e90a0dc" exitCode=0
Sep 11 23:10:55.638851 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:55.638508    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmbhgw" event={"ID":"3de5e6e3-6114-42ef-9f12-bad0c68aaca2","Type":"ContainerDied","Data":"1ce7d4cd821cf1c7a80c5c724ae5567838f081d910249dfe15baa9910e90a0dc"}
Sep 11 23:10:55.640411 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:55.640384    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-l48qw" event={"ID":"9ffe714c-1d4b-41a5-bc45-5031152887d5","Type":"ContainerStarted","Data":"851d10c0d20285697479211a46a0e059da445d4cef33383564ed60f3090e4235"}
Sep 11 23:10:55.640551 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:55.640526    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-l48qw"
Sep 11 23:10:55.675071 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:55.675018    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-l48qw" podStartSLOduration=1.188411112 podStartE2EDuration="5.675000386s" totalImagesPullingTime="4.486589274s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 23:10:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:10:55.673539602 +0000 UTC m=+579.706668820" watchObservedRunningTime="2026-09-11 23:10:55.675000386 +0000 UTC m=+579.708129604"
Sep 11 23:10:55.868244 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:55.868140    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-wqswm"]
Sep 11 23:10:55.872360 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:55.872337    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-wqswm"
Sep 11 23:10:55.875276 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:55.875255    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-2l4gl\""
Sep 11 23:10:55.882579 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:55.882556    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-wqswm"]
Sep 11 23:10:55.971691 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:55.971659    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xt9jb\" (UniqueName: \"kubernetes.io/projected/e34c43f0-b96b-4f5d-ab93-85145c0e7d67-kube-api-access-xt9jb\") pod \"cert-manager-cainjector-559cdd8c87-wqswm\" (UID: \"e34c43f0-b96b-4f5d-ab93-85145c0e7d67\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-wqswm"
Sep 11 23:10:55.971844 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:55.971738    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e34c43f0-b96b-4f5d-ab93-85145c0e7d67-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-wqswm\" (UID: \"e34c43f0-b96b-4f5d-ab93-85145c0e7d67\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-wqswm"
Sep 11 23:10:56.072831 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:56.072794    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e34c43f0-b96b-4f5d-ab93-85145c0e7d67-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-wqswm\" (UID: \"e34c43f0-b96b-4f5d-ab93-85145c0e7d67\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-wqswm"
Sep 11 23:10:56.073021 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:56.072855    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xt9jb\" (UniqueName: \"kubernetes.io/projected/e34c43f0-b96b-4f5d-ab93-85145c0e7d67-kube-api-access-xt9jb\") pod \"cert-manager-cainjector-559cdd8c87-wqswm\" (UID: \"e34c43f0-b96b-4f5d-ab93-85145c0e7d67\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-wqswm"
Sep 11 23:10:56.082528 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:56.082495    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e34c43f0-b96b-4f5d-ab93-85145c0e7d67-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-wqswm\" (UID: \"e34c43f0-b96b-4f5d-ab93-85145c0e7d67\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-wqswm"
Sep 11 23:10:56.082652 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:56.082595    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xt9jb\" (UniqueName: \"kubernetes.io/projected/e34c43f0-b96b-4f5d-ab93-85145c0e7d67-kube-api-access-xt9jb\") pod \"cert-manager-cainjector-559cdd8c87-wqswm\" (UID: \"e34c43f0-b96b-4f5d-ab93-85145c0e7d67\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-wqswm"
Sep 11 23:10:56.181743 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:56.181705    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-wqswm"
Sep 11 23:10:56.453617 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:56.453590    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-wqswm"]
Sep 11 23:10:56.455191 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:10:56.455143    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode34c43f0_b96b_4f5d_ab93_85145c0e7d67.slice/crio-9c1f43284e89b5a86bb44dd01f37767b3fd0fe395bdeb1533acd4227934417d6 WatchSource:0}: Error finding container 9c1f43284e89b5a86bb44dd01f37767b3fd0fe395bdeb1533acd4227934417d6: Status 404 returned error can't find the container with id 9c1f43284e89b5a86bb44dd01f37767b3fd0fe395bdeb1533acd4227934417d6
Sep 11 23:10:56.646907 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:56.646871    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-wqswm" event={"ID":"e34c43f0-b96b-4f5d-ab93-85145c0e7d67","Type":"ContainerStarted","Data":"2b1fcd46131e5300d883fae41426b00514b976f10042d9434c7a613578672578"}
Sep 11 23:10:56.646907 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:56.646916    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-wqswm" event={"ID":"e34c43f0-b96b-4f5d-ab93-85145c0e7d67","Type":"ContainerStarted","Data":"9c1f43284e89b5a86bb44dd01f37767b3fd0fe395bdeb1533acd4227934417d6"}
Sep 11 23:10:56.677815 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:56.677746    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-wqswm" podStartSLOduration=1.677723458 podStartE2EDuration="1.677723458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 23:10:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:10:56.676071026 +0000 UTC m=+580.709200259" watchObservedRunningTime="2026-09-11 23:10:56.677723458 +0000 UTC m=+580.710852677"
Sep 11 23:10:56.785134 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:56.785108    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmbhgw"
Sep 11 23:10:56.980199 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:56.980085    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3de5e6e3-6114-42ef-9f12-bad0c68aaca2-bundle\") pod \"3de5e6e3-6114-42ef-9f12-bad0c68aaca2\" (UID: \"3de5e6e3-6114-42ef-9f12-bad0c68aaca2\") "
Sep 11 23:10:56.980199 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:56.980123    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3de5e6e3-6114-42ef-9f12-bad0c68aaca2-util\") pod \"3de5e6e3-6114-42ef-9f12-bad0c68aaca2\" (UID: \"3de5e6e3-6114-42ef-9f12-bad0c68aaca2\") "
Sep 11 23:10:56.980199 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:56.980144    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8d5mx\" (UniqueName: \"kubernetes.io/projected/3de5e6e3-6114-42ef-9f12-bad0c68aaca2-kube-api-access-8d5mx\") pod \"3de5e6e3-6114-42ef-9f12-bad0c68aaca2\" (UID: \"3de5e6e3-6114-42ef-9f12-bad0c68aaca2\") "
Sep 11 23:10:56.980558 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:56.980529    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3de5e6e3-6114-42ef-9f12-bad0c68aaca2-bundle" (OuterVolumeSpecName: "bundle") pod "3de5e6e3-6114-42ef-9f12-bad0c68aaca2" (UID: "3de5e6e3-6114-42ef-9f12-bad0c68aaca2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:10:56.982475 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:56.982453    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3de5e6e3-6114-42ef-9f12-bad0c68aaca2-kube-api-access-8d5mx" (OuterVolumeSpecName: "kube-api-access-8d5mx") pod "3de5e6e3-6114-42ef-9f12-bad0c68aaca2" (UID: "3de5e6e3-6114-42ef-9f12-bad0c68aaca2"). InnerVolumeSpecName "kube-api-access-8d5mx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 23:10:56.985779 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:56.985753    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3de5e6e3-6114-42ef-9f12-bad0c68aaca2-util" (OuterVolumeSpecName: "util") pod "3de5e6e3-6114-42ef-9f12-bad0c68aaca2" (UID: "3de5e6e3-6114-42ef-9f12-bad0c68aaca2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:10:57.081117 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:57.081078    2546 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3de5e6e3-6114-42ef-9f12-bad0c68aaca2-bundle\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:10:57.081117 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:57.081111    2546 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3de5e6e3-6114-42ef-9f12-bad0c68aaca2-util\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:10:57.081117 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:57.081120    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8d5mx\" (UniqueName: \"kubernetes.io/projected/3de5e6e3-6114-42ef-9f12-bad0c68aaca2-kube-api-access-8d5mx\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:10:57.653393 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:57.653358    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmbhgw"
Sep 11 23:10:57.653393 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:57.653391    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmbhgw" event={"ID":"3de5e6e3-6114-42ef-9f12-bad0c68aaca2","Type":"ContainerDied","Data":"ff41d02a85f2018eca16a22c5ca43a1e6e37a0d47e664c1506cb7438d942d252"}
Sep 11 23:10:57.653845 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:10:57.653434    2546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff41d02a85f2018eca16a22c5ca43a1e6e37a0d47e664c1506cb7438d942d252"
Sep 11 23:11:01.649838 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:01.649806    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-l48qw"
Sep 11 23:11:10.122177 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:10.122121    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afd8blz"]
Sep 11 23:11:10.122559 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:10.122449    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3de5e6e3-6114-42ef-9f12-bad0c68aaca2" containerName="pull"
Sep 11 23:11:10.122559 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:10.122460    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="3de5e6e3-6114-42ef-9f12-bad0c68aaca2" containerName="pull"
Sep 11 23:11:10.122559 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:10.122475    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3de5e6e3-6114-42ef-9f12-bad0c68aaca2" containerName="extract"
Sep 11 23:11:10.122559 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:10.122481    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="3de5e6e3-6114-42ef-9f12-bad0c68aaca2" containerName="extract"
Sep 11 23:11:10.122559 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:10.122491    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3de5e6e3-6114-42ef-9f12-bad0c68aaca2" containerName="util"
Sep 11 23:11:10.122559 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:10.122496    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="3de5e6e3-6114-42ef-9f12-bad0c68aaca2" containerName="util"
Sep 11 23:11:10.122559 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:10.122553    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="3de5e6e3-6114-42ef-9f12-bad0c68aaca2" containerName="extract"
Sep 11 23:11:10.125763 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:10.125744    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afd8blz"
Sep 11 23:11:10.128923 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:10.128904    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 11 23:11:10.130331 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:10.130310    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-qbxpl\""
Sep 11 23:11:10.130331 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:10.130324    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 11 23:11:10.135367 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:10.135345    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afd8blz"]
Sep 11 23:11:10.180856 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:10.180816    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lmts\" (UniqueName: \"kubernetes.io/projected/77e4391a-ab9c-49b1-8409-d4d21dbd04ee-kube-api-access-2lmts\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afd8blz\" (UID: \"77e4391a-ab9c-49b1-8409-d4d21dbd04ee\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afd8blz"
Sep 11 23:11:10.181030 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:10.180882    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/77e4391a-ab9c-49b1-8409-d4d21dbd04ee-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afd8blz\" (UID: \"77e4391a-ab9c-49b1-8409-d4d21dbd04ee\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afd8blz"
Sep 11 23:11:10.181030 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:10.180911    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/77e4391a-ab9c-49b1-8409-d4d21dbd04ee-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afd8blz\" (UID: \"77e4391a-ab9c-49b1-8409-d4d21dbd04ee\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afd8blz"
Sep 11 23:11:10.281899 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:10.281863    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lmts\" (UniqueName: \"kubernetes.io/projected/77e4391a-ab9c-49b1-8409-d4d21dbd04ee-kube-api-access-2lmts\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afd8blz\" (UID: \"77e4391a-ab9c-49b1-8409-d4d21dbd04ee\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afd8blz"
Sep 11 23:11:10.282071 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:10.281923    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/77e4391a-ab9c-49b1-8409-d4d21dbd04ee-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afd8blz\" (UID: \"77e4391a-ab9c-49b1-8409-d4d21dbd04ee\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afd8blz"
Sep 11 23:11:10.282071 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:10.281949    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/77e4391a-ab9c-49b1-8409-d4d21dbd04ee-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afd8blz\" (UID: \"77e4391a-ab9c-49b1-8409-d4d21dbd04ee\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afd8blz"
Sep 11 23:11:10.282315 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:10.282293    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/77e4391a-ab9c-49b1-8409-d4d21dbd04ee-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afd8blz\" (UID: \"77e4391a-ab9c-49b1-8409-d4d21dbd04ee\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afd8blz"
Sep 11 23:11:10.282360 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:10.282321    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/77e4391a-ab9c-49b1-8409-d4d21dbd04ee-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afd8blz\" (UID: \"77e4391a-ab9c-49b1-8409-d4d21dbd04ee\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afd8blz"
Sep 11 23:11:10.290899 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:10.290865    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lmts\" (UniqueName: \"kubernetes.io/projected/77e4391a-ab9c-49b1-8409-d4d21dbd04ee-kube-api-access-2lmts\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afd8blz\" (UID: \"77e4391a-ab9c-49b1-8409-d4d21dbd04ee\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afd8blz"
Sep 11 23:11:10.435551 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:10.435513    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afd8blz"
Sep 11 23:11:10.709923 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:10.709894    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afd8blz"]
Sep 11 23:11:10.712574 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:11:10.712535    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77e4391a_ab9c_49b1_8409_d4d21dbd04ee.slice/crio-a18377ebb6cff5054bf26781e6b1ac5c3db883356948f374a5f9f8000714b27d WatchSource:0}: Error finding container a18377ebb6cff5054bf26781e6b1ac5c3db883356948f374a5f9f8000714b27d: Status 404 returned error can't find the container with id a18377ebb6cff5054bf26781e6b1ac5c3db883356948f374a5f9f8000714b27d
Sep 11 23:11:11.726623 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:11.726584    2546 generic.go:356] "Generic (PLEG): container finished" podID="77e4391a-ab9c-49b1-8409-d4d21dbd04ee" containerID="0df4ca8d0d812a4d184109f543c7d57f6775243ace6881fead302d2b0fe2084b" exitCode=0
Sep 11 23:11:11.727103 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:11.726651    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afd8blz" event={"ID":"77e4391a-ab9c-49b1-8409-d4d21dbd04ee","Type":"ContainerDied","Data":"0df4ca8d0d812a4d184109f543c7d57f6775243ace6881fead302d2b0fe2084b"}
Sep 11 23:11:11.727103 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:11.726734    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afd8blz" event={"ID":"77e4391a-ab9c-49b1-8409-d4d21dbd04ee","Type":"ContainerStarted","Data":"a18377ebb6cff5054bf26781e6b1ac5c3db883356948f374a5f9f8000714b27d"}
Sep 11 23:11:12.733003 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:12.732946    2546 generic.go:356] "Generic (PLEG): container finished" podID="77e4391a-ab9c-49b1-8409-d4d21dbd04ee" containerID="419564b02066bb38bf06ae52648398e99999f6f32d871a7b83bd1eb6ed42bc2c" exitCode=0
Sep 11 23:11:12.733003 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:12.732998    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afd8blz" event={"ID":"77e4391a-ab9c-49b1-8409-d4d21dbd04ee","Type":"ContainerDied","Data":"419564b02066bb38bf06ae52648398e99999f6f32d871a7b83bd1eb6ed42bc2c"}
Sep 11 23:11:13.739993 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:13.739956    2546 generic.go:356] "Generic (PLEG): container finished" podID="77e4391a-ab9c-49b1-8409-d4d21dbd04ee" containerID="f658b19e52ebe1480468ece18151e86ff83bcc5645c6437e40b8e9fab2997ba8" exitCode=0
Sep 11 23:11:13.740421 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:13.740036    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afd8blz" event={"ID":"77e4391a-ab9c-49b1-8409-d4d21dbd04ee","Type":"ContainerDied","Data":"f658b19e52ebe1480468ece18151e86ff83bcc5645c6437e40b8e9fab2997ba8"}
Sep 11 23:11:14.892916 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:14.892886    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afd8blz"
Sep 11 23:11:14.920043 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:14.920014    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2lmts\" (UniqueName: \"kubernetes.io/projected/77e4391a-ab9c-49b1-8409-d4d21dbd04ee-kube-api-access-2lmts\") pod \"77e4391a-ab9c-49b1-8409-d4d21dbd04ee\" (UID: \"77e4391a-ab9c-49b1-8409-d4d21dbd04ee\") "
Sep 11 23:11:14.920229 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:14.920090    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/77e4391a-ab9c-49b1-8409-d4d21dbd04ee-util\") pod \"77e4391a-ab9c-49b1-8409-d4d21dbd04ee\" (UID: \"77e4391a-ab9c-49b1-8409-d4d21dbd04ee\") "
Sep 11 23:11:14.920229 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:14.920120    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/77e4391a-ab9c-49b1-8409-d4d21dbd04ee-bundle\") pod \"77e4391a-ab9c-49b1-8409-d4d21dbd04ee\" (UID: \"77e4391a-ab9c-49b1-8409-d4d21dbd04ee\") "
Sep 11 23:11:14.920944 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:14.920917    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77e4391a-ab9c-49b1-8409-d4d21dbd04ee-bundle" (OuterVolumeSpecName: "bundle") pod "77e4391a-ab9c-49b1-8409-d4d21dbd04ee" (UID: "77e4391a-ab9c-49b1-8409-d4d21dbd04ee"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:11:14.922495 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:14.922471    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77e4391a-ab9c-49b1-8409-d4d21dbd04ee-kube-api-access-2lmts" (OuterVolumeSpecName: "kube-api-access-2lmts") pod "77e4391a-ab9c-49b1-8409-d4d21dbd04ee" (UID: "77e4391a-ab9c-49b1-8409-d4d21dbd04ee"). InnerVolumeSpecName "kube-api-access-2lmts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 23:11:14.925714 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:14.925678    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77e4391a-ab9c-49b1-8409-d4d21dbd04ee-util" (OuterVolumeSpecName: "util") pod "77e4391a-ab9c-49b1-8409-d4d21dbd04ee" (UID: "77e4391a-ab9c-49b1-8409-d4d21dbd04ee"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:11:15.020938 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:15.020847    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lmts\" (UniqueName: \"kubernetes.io/projected/77e4391a-ab9c-49b1-8409-d4d21dbd04ee-kube-api-access-2lmts\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:11:15.020938 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:15.020878    2546 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/77e4391a-ab9c-49b1-8409-d4d21dbd04ee-util\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:11:15.020938 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:15.020894    2546 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/77e4391a-ab9c-49b1-8409-d4d21dbd04ee-bundle\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:11:15.751676 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:15.751638    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afd8blz" event={"ID":"77e4391a-ab9c-49b1-8409-d4d21dbd04ee","Type":"ContainerDied","Data":"a18377ebb6cff5054bf26781e6b1ac5c3db883356948f374a5f9f8000714b27d"}
Sep 11 23:11:15.751676 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:15.751676    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afd8blz"
Sep 11 23:11:15.751878 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:15.751694    2546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a18377ebb6cff5054bf26781e6b1ac5c3db883356948f374a5f9f8000714b27d"
Sep 11 23:11:16.439033 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:16.439001    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7244_6b0a8af3-932c-4091-b7a0-180fe1751a3a/ovn-acl-logging/0.log"
Sep 11 23:11:16.441108 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:16.441083    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7244_6b0a8af3-932c-4091-b7a0-180fe1751a3a/ovn-acl-logging/0.log"
Sep 11 23:11:27.140334 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:27.140303    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ljn58"]
Sep 11 23:11:27.145707 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:27.140609    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77e4391a-ab9c-49b1-8409-d4d21dbd04ee" containerName="extract"
Sep 11 23:11:27.145707 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:27.140620    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="77e4391a-ab9c-49b1-8409-d4d21dbd04ee" containerName="extract"
Sep 11 23:11:27.145707 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:27.140643    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77e4391a-ab9c-49b1-8409-d4d21dbd04ee" containerName="pull"
Sep 11 23:11:27.145707 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:27.140648    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="77e4391a-ab9c-49b1-8409-d4d21dbd04ee" containerName="pull"
Sep 11 23:11:27.145707 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:27.140659    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77e4391a-ab9c-49b1-8409-d4d21dbd04ee" containerName="util"
Sep 11 23:11:27.145707 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:27.140664    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="77e4391a-ab9c-49b1-8409-d4d21dbd04ee" containerName="util"
Sep 11 23:11:27.145707 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:27.140717    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="77e4391a-ab9c-49b1-8409-d4d21dbd04ee" containerName="extract"
Sep 11 23:11:27.146181 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:27.146144    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ljn58"
Sep 11 23:11:27.149055 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:27.149032    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 11 23:11:27.149166 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:27.149032    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 11 23:11:27.150393 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:27.150375    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-qbxpl\""
Sep 11 23:11:27.154423 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:27.154398    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ljn58"]
Sep 11 23:11:27.220373 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:27.220338    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c97b887f-8c90-4740-8bfd-8c5483e3f332-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ljn58\" (UID: \"c97b887f-8c90-4740-8bfd-8c5483e3f332\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ljn58"
Sep 11 23:11:27.220558 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:27.220395    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f79h2\" (UniqueName: \"kubernetes.io/projected/c97b887f-8c90-4740-8bfd-8c5483e3f332-kube-api-access-f79h2\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ljn58\" (UID: \"c97b887f-8c90-4740-8bfd-8c5483e3f332\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ljn58"
Sep 11 23:11:27.220558 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:27.220462    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c97b887f-8c90-4740-8bfd-8c5483e3f332-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ljn58\" (UID: \"c97b887f-8c90-4740-8bfd-8c5483e3f332\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ljn58"
Sep 11 23:11:27.321285 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:27.321252    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c97b887f-8c90-4740-8bfd-8c5483e3f332-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ljn58\" (UID: \"c97b887f-8c90-4740-8bfd-8c5483e3f332\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ljn58"
Sep 11 23:11:27.321463 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:27.321306    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c97b887f-8c90-4740-8bfd-8c5483e3f332-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ljn58\" (UID: \"c97b887f-8c90-4740-8bfd-8c5483e3f332\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ljn58"
Sep 11 23:11:27.321463 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:27.321348    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f79h2\" (UniqueName: \"kubernetes.io/projected/c97b887f-8c90-4740-8bfd-8c5483e3f332-kube-api-access-f79h2\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ljn58\" (UID: \"c97b887f-8c90-4740-8bfd-8c5483e3f332\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ljn58"
Sep 11 23:11:27.321649 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:27.321631    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c97b887f-8c90-4740-8bfd-8c5483e3f332-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ljn58\" (UID: \"c97b887f-8c90-4740-8bfd-8c5483e3f332\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ljn58"
Sep 11 23:11:27.321694 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:27.321678    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c97b887f-8c90-4740-8bfd-8c5483e3f332-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ljn58\" (UID: \"c97b887f-8c90-4740-8bfd-8c5483e3f332\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ljn58"
Sep 11 23:11:27.336300 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:27.336273    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f79h2\" (UniqueName: \"kubernetes.io/projected/c97b887f-8c90-4740-8bfd-8c5483e3f332-kube-api-access-f79h2\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ljn58\" (UID: \"c97b887f-8c90-4740-8bfd-8c5483e3f332\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ljn58"
Sep 11 23:11:27.352181 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:27.352127    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6857545767-t5kb6"]
Sep 11 23:11:27.390760 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:27.390688    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6857545767-t5kb6"]
Sep 11 23:11:27.390906 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:27.390809    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6857545767-t5kb6"
Sep 11 23:11:27.394391 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:27.394366    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 11 23:11:27.394526 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:27.394460    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 11 23:11:27.394607 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:27.394591    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-2lx4s\""
Sep 11 23:11:27.394659 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:27.394646    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 11 23:11:27.394891 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:27.394879    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 11 23:11:27.456530 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:27.456491    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ljn58"
Sep 11 23:11:27.523294 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:27.523251    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d6280fd5-65fd-4c95-a616-19794456d570-manifests\") pod \"opendatahub-operator-controller-manager-6857545767-t5kb6\" (UID: \"d6280fd5-65fd-4c95-a616-19794456d570\") " pod="opendatahub/opendatahub-operator-controller-manager-6857545767-t5kb6"
Sep 11 23:11:27.523294 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:27.523292    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d6280fd5-65fd-4c95-a616-19794456d570-tmp\") pod \"opendatahub-operator-controller-manager-6857545767-t5kb6\" (UID: \"d6280fd5-65fd-4c95-a616-19794456d570\") " pod="opendatahub/opendatahub-operator-controller-manager-6857545767-t5kb6"
Sep 11 23:11:27.523548 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:27.523323    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d6280fd5-65fd-4c95-a616-19794456d570-webhook-cert\") pod \"opendatahub-operator-controller-manager-6857545767-t5kb6\" (UID: \"d6280fd5-65fd-4c95-a616-19794456d570\") " pod="opendatahub/opendatahub-operator-controller-manager-6857545767-t5kb6"
Sep 11 23:11:27.523548 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:27.523404    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mrg2\" (UniqueName: \"kubernetes.io/projected/d6280fd5-65fd-4c95-a616-19794456d570-kube-api-access-5mrg2\") pod \"opendatahub-operator-controller-manager-6857545767-t5kb6\" (UID: \"d6280fd5-65fd-4c95-a616-19794456d570\") " pod="opendatahub/opendatahub-operator-controller-manager-6857545767-t5kb6"
Sep 11 23:11:27.523548 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:27.523439    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d6280fd5-65fd-4c95-a616-19794456d570-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6857545767-t5kb6\" (UID: \"d6280fd5-65fd-4c95-a616-19794456d570\") " pod="opendatahub/opendatahub-operator-controller-manager-6857545767-t5kb6"
Sep 11 23:11:27.623935 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:27.623899    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d6280fd5-65fd-4c95-a616-19794456d570-manifests\") pod \"opendatahub-operator-controller-manager-6857545767-t5kb6\" (UID: \"d6280fd5-65fd-4c95-a616-19794456d570\") " pod="opendatahub/opendatahub-operator-controller-manager-6857545767-t5kb6"
Sep 11 23:11:27.624140 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:27.623944    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d6280fd5-65fd-4c95-a616-19794456d570-tmp\") pod \"opendatahub-operator-controller-manager-6857545767-t5kb6\" (UID: \"d6280fd5-65fd-4c95-a616-19794456d570\") " pod="opendatahub/opendatahub-operator-controller-manager-6857545767-t5kb6"
Sep 11 23:11:27.624140 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:27.623989    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d6280fd5-65fd-4c95-a616-19794456d570-webhook-cert\") pod \"opendatahub-operator-controller-manager-6857545767-t5kb6\" (UID: \"d6280fd5-65fd-4c95-a616-19794456d570\") " pod="opendatahub/opendatahub-operator-controller-manager-6857545767-t5kb6"
Sep 11 23:11:27.624140 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:27.624019    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mrg2\" (UniqueName: \"kubernetes.io/projected/d6280fd5-65fd-4c95-a616-19794456d570-kube-api-access-5mrg2\") pod \"opendatahub-operator-controller-manager-6857545767-t5kb6\" (UID: \"d6280fd5-65fd-4c95-a616-19794456d570\") " pod="opendatahub/opendatahub-operator-controller-manager-6857545767-t5kb6"
Sep 11 23:11:27.624140 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:27.624049    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d6280fd5-65fd-4c95-a616-19794456d570-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6857545767-t5kb6\" (UID: \"d6280fd5-65fd-4c95-a616-19794456d570\") " pod="opendatahub/opendatahub-operator-controller-manager-6857545767-t5kb6"
Sep 11 23:11:27.624415 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:27.624389    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d6280fd5-65fd-4c95-a616-19794456d570-manifests\") pod \"opendatahub-operator-controller-manager-6857545767-t5kb6\" (UID: \"d6280fd5-65fd-4c95-a616-19794456d570\") " pod="opendatahub/opendatahub-operator-controller-manager-6857545767-t5kb6"
Sep 11 23:11:27.626528 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:27.626506    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d6280fd5-65fd-4c95-a616-19794456d570-tmp\") pod \"opendatahub-operator-controller-manager-6857545767-t5kb6\" (UID: \"d6280fd5-65fd-4c95-a616-19794456d570\") " pod="opendatahub/opendatahub-operator-controller-manager-6857545767-t5kb6"
Sep 11 23:11:27.626632 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:27.626560    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d6280fd5-65fd-4c95-a616-19794456d570-webhook-cert\") pod \"opendatahub-operator-controller-manager-6857545767-t5kb6\" (UID: \"d6280fd5-65fd-4c95-a616-19794456d570\") " pod="opendatahub/opendatahub-operator-controller-manager-6857545767-t5kb6"
Sep 11 23:11:27.626744 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:27.626727    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d6280fd5-65fd-4c95-a616-19794456d570-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6857545767-t5kb6\" (UID: \"d6280fd5-65fd-4c95-a616-19794456d570\") " pod="opendatahub/opendatahub-operator-controller-manager-6857545767-t5kb6"
Sep 11 23:11:27.632665 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:27.632643    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mrg2\" (UniqueName: \"kubernetes.io/projected/d6280fd5-65fd-4c95-a616-19794456d570-kube-api-access-5mrg2\") pod \"opendatahub-operator-controller-manager-6857545767-t5kb6\" (UID: \"d6280fd5-65fd-4c95-a616-19794456d570\") " pod="opendatahub/opendatahub-operator-controller-manager-6857545767-t5kb6"
Sep 11 23:11:27.701520 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:27.701479    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6857545767-t5kb6"
Sep 11 23:11:27.755532 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:27.755490    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ljn58"]
Sep 11 23:11:27.758987 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:11:27.758956    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc97b887f_8c90_4740_8bfd_8c5483e3f332.slice/crio-d0441f13274c8a186b4e991f52f008f90d012f00816ee38356daec659afb9ff8 WatchSource:0}: Error finding container d0441f13274c8a186b4e991f52f008f90d012f00816ee38356daec659afb9ff8: Status 404 returned error can't find the container with id d0441f13274c8a186b4e991f52f008f90d012f00816ee38356daec659afb9ff8
Sep 11 23:11:27.811295 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:27.811255    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ljn58" event={"ID":"c97b887f-8c90-4740-8bfd-8c5483e3f332","Type":"ContainerStarted","Data":"d0441f13274c8a186b4e991f52f008f90d012f00816ee38356daec659afb9ff8"}
Sep 11 23:11:28.019938 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:28.019862    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6857545767-t5kb6"]
Sep 11 23:11:28.022969 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:11:28.022945    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6280fd5_65fd_4c95_a616_19794456d570.slice/crio-a4ccdc37b6d8c8db8b4bc184ed1d819727ac7e2d38cbd754226de0ed37590a28 WatchSource:0}: Error finding container a4ccdc37b6d8c8db8b4bc184ed1d819727ac7e2d38cbd754226de0ed37590a28: Status 404 returned error can't find the container with id a4ccdc37b6d8c8db8b4bc184ed1d819727ac7e2d38cbd754226de0ed37590a28
Sep 11 23:11:28.821942 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:28.821901    2546 generic.go:356] "Generic (PLEG): container finished" podID="c97b887f-8c90-4740-8bfd-8c5483e3f332" containerID="2466b2517adc8823cb3a807d6ac38373247a86a9566807d8e14a44b670a5a370" exitCode=0
Sep 11 23:11:28.822435 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:28.821992    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ljn58" event={"ID":"c97b887f-8c90-4740-8bfd-8c5483e3f332","Type":"ContainerDied","Data":"2466b2517adc8823cb3a807d6ac38373247a86a9566807d8e14a44b670a5a370"}
Sep 11 23:11:28.824005 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:28.823974    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6857545767-t5kb6" event={"ID":"d6280fd5-65fd-4c95-a616-19794456d570","Type":"ContainerStarted","Data":"a4ccdc37b6d8c8db8b4bc184ed1d819727ac7e2d38cbd754226de0ed37590a28"}
Sep 11 23:11:31.842093 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:31.842058    2546 generic.go:356] "Generic (PLEG): container finished" podID="c97b887f-8c90-4740-8bfd-8c5483e3f332" containerID="887e086a45c04660e04b8cf30ab2a299ef388cb913d0d761230cdc8ed8888288" exitCode=0
Sep 11 23:11:31.842558 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:31.842147    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ljn58" event={"ID":"c97b887f-8c90-4740-8bfd-8c5483e3f332","Type":"ContainerDied","Data":"887e086a45c04660e04b8cf30ab2a299ef388cb913d0d761230cdc8ed8888288"}
Sep 11 23:11:31.843952 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:31.843930    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6857545767-t5kb6" event={"ID":"d6280fd5-65fd-4c95-a616-19794456d570","Type":"ContainerStarted","Data":"1cbddb47b71d8f5675a06fe49a9791e431225ef795abc11688343ce2bf7d0d2a"}
Sep 11 23:11:32.851057 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:32.851017    2546 generic.go:356] "Generic (PLEG): container finished" podID="c97b887f-8c90-4740-8bfd-8c5483e3f332" containerID="e34929d9ba339ad5420fb9261b5378a3756745cab1c0183040f61ccf474b2391" exitCode=0
Sep 11 23:11:32.851567 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:32.851065    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ljn58" event={"ID":"c97b887f-8c90-4740-8bfd-8c5483e3f332","Type":"ContainerDied","Data":"e34929d9ba339ad5420fb9261b5378a3756745cab1c0183040f61ccf474b2391"}
Sep 11 23:11:32.852906 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:32.852887    2546 generic.go:356] "Generic (PLEG): container finished" podID="d6280fd5-65fd-4c95-a616-19794456d570" containerID="1cbddb47b71d8f5675a06fe49a9791e431225ef795abc11688343ce2bf7d0d2a" exitCode=0
Sep 11 23:11:32.853026 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:32.852947    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6857545767-t5kb6" event={"ID":"d6280fd5-65fd-4c95-a616-19794456d570","Type":"ContainerDied","Data":"1cbddb47b71d8f5675a06fe49a9791e431225ef795abc11688343ce2bf7d0d2a"}
Sep 11 23:11:33.860692 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:33.860655    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6857545767-t5kb6" event={"ID":"d6280fd5-65fd-4c95-a616-19794456d570","Type":"ContainerStarted","Data":"1de553dc1a379baf377365ba446fd2fce1b20d24ad2a831780abc6944bccf9fb"}
Sep 11 23:11:33.861085 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:33.860732    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6857545767-t5kb6"
Sep 11 23:11:33.883984 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:33.883924    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6857545767-t5kb6" podStartSLOduration=2.5217378310000003 podStartE2EDuration="6.883902873s" totalImagesPullingTime="3.271749335s" totalInitContainerRuntime="1.090415707s" isStatefulPod=false podCreationTimestamp="2026-09-11 23:11:27 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:11:33.882046336 +0000 UTC m=+617.915175554" watchObservedRunningTime="2026-09-11 23:11:33.883902873 +0000 UTC m=+617.917032092"
Sep 11 23:11:33.990179 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:33.990141    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ljn58"
Sep 11 23:11:34.084620 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:34.084586    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c97b887f-8c90-4740-8bfd-8c5483e3f332-util\") pod \"c97b887f-8c90-4740-8bfd-8c5483e3f332\" (UID: \"c97b887f-8c90-4740-8bfd-8c5483e3f332\") "
Sep 11 23:11:34.084823 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:34.084658    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f79h2\" (UniqueName: \"kubernetes.io/projected/c97b887f-8c90-4740-8bfd-8c5483e3f332-kube-api-access-f79h2\") pod \"c97b887f-8c90-4740-8bfd-8c5483e3f332\" (UID: \"c97b887f-8c90-4740-8bfd-8c5483e3f332\") "
Sep 11 23:11:34.084823 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:34.084703    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c97b887f-8c90-4740-8bfd-8c5483e3f332-bundle\") pod \"c97b887f-8c90-4740-8bfd-8c5483e3f332\" (UID: \"c97b887f-8c90-4740-8bfd-8c5483e3f332\") "
Sep 11 23:11:34.085483 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:34.085448    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c97b887f-8c90-4740-8bfd-8c5483e3f332-bundle" (OuterVolumeSpecName: "bundle") pod "c97b887f-8c90-4740-8bfd-8c5483e3f332" (UID: "c97b887f-8c90-4740-8bfd-8c5483e3f332"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:11:34.087023 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:34.086984    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c97b887f-8c90-4740-8bfd-8c5483e3f332-kube-api-access-f79h2" (OuterVolumeSpecName: "kube-api-access-f79h2") pod "c97b887f-8c90-4740-8bfd-8c5483e3f332" (UID: "c97b887f-8c90-4740-8bfd-8c5483e3f332"). InnerVolumeSpecName "kube-api-access-f79h2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 23:11:34.089416 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:34.089384    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c97b887f-8c90-4740-8bfd-8c5483e3f332-util" (OuterVolumeSpecName: "util") pod "c97b887f-8c90-4740-8bfd-8c5483e3f332" (UID: "c97b887f-8c90-4740-8bfd-8c5483e3f332"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:11:34.185581 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:34.185545    2546 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c97b887f-8c90-4740-8bfd-8c5483e3f332-util\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:11:34.185581 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:34.185579    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f79h2\" (UniqueName: \"kubernetes.io/projected/c97b887f-8c90-4740-8bfd-8c5483e3f332-kube-api-access-f79h2\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:11:34.185581 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:34.185590    2546 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c97b887f-8c90-4740-8bfd-8c5483e3f332-bundle\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:11:34.868429 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:34.868349    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ljn58"
Sep 11 23:11:34.868429 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:34.868375    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ljn58" event={"ID":"c97b887f-8c90-4740-8bfd-8c5483e3f332","Type":"ContainerDied","Data":"d0441f13274c8a186b4e991f52f008f90d012f00816ee38356daec659afb9ff8"}
Sep 11 23:11:34.868429 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:34.868426    2546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0441f13274c8a186b4e991f52f008f90d012f00816ee38356daec659afb9ff8"
Sep 11 23:11:44.871067 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:44.870981    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6857545767-t5kb6"
Sep 11 23:11:47.819428 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:47.819393    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7c75bd8f97-csrr2"]
Sep 11 23:11:47.819882 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:47.819863    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c97b887f-8c90-4740-8bfd-8c5483e3f332" containerName="pull"
Sep 11 23:11:47.819882 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:47.819881    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="c97b887f-8c90-4740-8bfd-8c5483e3f332" containerName="pull"
Sep 11 23:11:47.819997 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:47.819908    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c97b887f-8c90-4740-8bfd-8c5483e3f332" containerName="extract"
Sep 11 23:11:47.819997 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:47.819917    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="c97b887f-8c90-4740-8bfd-8c5483e3f332" containerName="extract"
Sep 11 23:11:47.819997 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:47.819936    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c97b887f-8c90-4740-8bfd-8c5483e3f332" containerName="util"
Sep 11 23:11:47.819997 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:47.819945    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="c97b887f-8c90-4740-8bfd-8c5483e3f332" containerName="util"
Sep 11 23:11:47.820211 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:47.820039    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="c97b887f-8c90-4740-8bfd-8c5483e3f332" containerName="extract"
Sep 11 23:11:47.823432 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:47.823413    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7c75bd8f97-csrr2"
Sep 11 23:11:47.826435 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:47.826415    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 11 23:11:47.826682 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:47.826658    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 11 23:11:47.826807 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:47.826707    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-5jbkg\""
Sep 11 23:11:47.833102 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:47.833079    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7c75bd8f97-csrr2"]
Sep 11 23:11:48.009221 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:48.009188    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a6196e75-6ca1-42e7-9846-8ad5850e4020-tmp\") pod \"kube-auth-proxy-7c75bd8f97-csrr2\" (UID: \"a6196e75-6ca1-42e7-9846-8ad5850e4020\") " pod="openshift-ingress/kube-auth-proxy-7c75bd8f97-csrr2"
Sep 11 23:11:48.009410 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:48.009267    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdncc\" (UniqueName: \"kubernetes.io/projected/a6196e75-6ca1-42e7-9846-8ad5850e4020-kube-api-access-bdncc\") pod \"kube-auth-proxy-7c75bd8f97-csrr2\" (UID: \"a6196e75-6ca1-42e7-9846-8ad5850e4020\") " pod="openshift-ingress/kube-auth-proxy-7c75bd8f97-csrr2"
Sep 11 23:11:48.009410 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:48.009314    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6196e75-6ca1-42e7-9846-8ad5850e4020-tls-certs\") pod \"kube-auth-proxy-7c75bd8f97-csrr2\" (UID: \"a6196e75-6ca1-42e7-9846-8ad5850e4020\") " pod="openshift-ingress/kube-auth-proxy-7c75bd8f97-csrr2"
Sep 11 23:11:48.110461 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:48.110365    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdncc\" (UniqueName: \"kubernetes.io/projected/a6196e75-6ca1-42e7-9846-8ad5850e4020-kube-api-access-bdncc\") pod \"kube-auth-proxy-7c75bd8f97-csrr2\" (UID: \"a6196e75-6ca1-42e7-9846-8ad5850e4020\") " pod="openshift-ingress/kube-auth-proxy-7c75bd8f97-csrr2"
Sep 11 23:11:48.110461 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:48.110426    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6196e75-6ca1-42e7-9846-8ad5850e4020-tls-certs\") pod \"kube-auth-proxy-7c75bd8f97-csrr2\" (UID: \"a6196e75-6ca1-42e7-9846-8ad5850e4020\") " pod="openshift-ingress/kube-auth-proxy-7c75bd8f97-csrr2"
Sep 11 23:11:48.110461 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:48.110456    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a6196e75-6ca1-42e7-9846-8ad5850e4020-tmp\") pod \"kube-auth-proxy-7c75bd8f97-csrr2\" (UID: \"a6196e75-6ca1-42e7-9846-8ad5850e4020\") " pod="openshift-ingress/kube-auth-proxy-7c75bd8f97-csrr2"
Sep 11 23:11:48.110682 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:11:48.110584    2546 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Sep 11 23:11:48.110682 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:11:48.110657    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6196e75-6ca1-42e7-9846-8ad5850e4020-tls-certs podName:a6196e75-6ca1-42e7-9846-8ad5850e4020 nodeName:}" failed. No retries permitted until 2026-09-11 23:11:48.610638617 +0000 UTC m=+632.643767836 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/a6196e75-6ca1-42e7-9846-8ad5850e4020-tls-certs") pod "kube-auth-proxy-7c75bd8f97-csrr2" (UID: "a6196e75-6ca1-42e7-9846-8ad5850e4020") : secret "kube-auth-proxy-tls" not found
Sep 11 23:11:48.112833 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:48.112802    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a6196e75-6ca1-42e7-9846-8ad5850e4020-tmp\") pod \"kube-auth-proxy-7c75bd8f97-csrr2\" (UID: \"a6196e75-6ca1-42e7-9846-8ad5850e4020\") " pod="openshift-ingress/kube-auth-proxy-7c75bd8f97-csrr2"
Sep 11 23:11:48.120643 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:48.120617    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdncc\" (UniqueName: \"kubernetes.io/projected/a6196e75-6ca1-42e7-9846-8ad5850e4020-kube-api-access-bdncc\") pod \"kube-auth-proxy-7c75bd8f97-csrr2\" (UID: \"a6196e75-6ca1-42e7-9846-8ad5850e4020\") " pod="openshift-ingress/kube-auth-proxy-7c75bd8f97-csrr2"
Sep 11 23:11:48.614946 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:48.614912    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6196e75-6ca1-42e7-9846-8ad5850e4020-tls-certs\") pod \"kube-auth-proxy-7c75bd8f97-csrr2\" (UID: \"a6196e75-6ca1-42e7-9846-8ad5850e4020\") " pod="openshift-ingress/kube-auth-proxy-7c75bd8f97-csrr2"
Sep 11 23:11:48.617491 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:48.617468    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6196e75-6ca1-42e7-9846-8ad5850e4020-tls-certs\") pod \"kube-auth-proxy-7c75bd8f97-csrr2\" (UID: \"a6196e75-6ca1-42e7-9846-8ad5850e4020\") " pod="openshift-ingress/kube-auth-proxy-7c75bd8f97-csrr2"
Sep 11 23:11:48.735022 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:48.734985    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7c75bd8f97-csrr2"
Sep 11 23:11:49.010749 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:49.010722    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7c75bd8f97-csrr2"]
Sep 11 23:11:49.012644 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:11:49.012616    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6196e75_6ca1_42e7_9846_8ad5850e4020.slice/crio-c662834dcfdb0f9941a1494e5dac7a0dcb5863b069ebb388489aa0863d35a7fa WatchSource:0}: Error finding container c662834dcfdb0f9941a1494e5dac7a0dcb5863b069ebb388489aa0863d35a7fa: Status 404 returned error can't find the container with id c662834dcfdb0f9941a1494e5dac7a0dcb5863b069ebb388489aa0863d35a7fa
Sep 11 23:11:49.014364 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:49.014349    2546 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 23:11:49.949699 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:49.949647    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7c75bd8f97-csrr2" event={"ID":"a6196e75-6ca1-42e7-9846-8ad5850e4020","Type":"ContainerStarted","Data":"c662834dcfdb0f9941a1494e5dac7a0dcb5863b069ebb388489aa0863d35a7fa"}
Sep 11 23:11:51.962135 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:51.962097    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7c75bd8f97-csrr2" event={"ID":"a6196e75-6ca1-42e7-9846-8ad5850e4020","Type":"ContainerStarted","Data":"19b32ce49054e81ae96dcb7f5bf9f7ab7cfb8f14c1dfa4f7507f2ee5abd83e6e"}
Sep 11 23:11:51.980166 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:51.980107    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7c75bd8f97-csrr2" podStartSLOduration=2.251705209 podStartE2EDuration="4.980092033s" totalImagesPullingTime="2.728386824s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 23:11:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:11:51.97872148 +0000 UTC m=+636.011850696" watchObservedRunningTime="2026-09-11 23:11:51.980092033 +0000 UTC m=+636.013221249"
Sep 11 23:11:54.566929 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:54.566901    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zz8lz"]
Sep 11 23:11:54.570691 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:54.570667    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zz8lz"
Sep 11 23:11:54.573693 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:54.573666    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 11 23:11:54.573819 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:54.573666    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-lm474\""
Sep 11 23:11:54.583896 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:54.583869    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zz8lz"]
Sep 11 23:11:54.670102 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:54.670065    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvzks\" (UniqueName: \"kubernetes.io/projected/92056a3e-aacd-45ed-9f05-a360ad3c8b21-kube-api-access-rvzks\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zz8lz\" (UID: \"92056a3e-aacd-45ed-9f05-a360ad3c8b21\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zz8lz"
Sep 11 23:11:54.670353 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:54.670120    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92056a3e-aacd-45ed-9f05-a360ad3c8b21-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zz8lz\" (UID: \"92056a3e-aacd-45ed-9f05-a360ad3c8b21\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zz8lz"
Sep 11 23:11:54.670353 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:54.670146    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92056a3e-aacd-45ed-9f05-a360ad3c8b21-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zz8lz\" (UID: \"92056a3e-aacd-45ed-9f05-a360ad3c8b21\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zz8lz"
Sep 11 23:11:54.670353 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:54.670232    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92056a3e-aacd-45ed-9f05-a360ad3c8b21-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zz8lz\" (UID: \"92056a3e-aacd-45ed-9f05-a360ad3c8b21\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zz8lz"
Sep 11 23:11:54.670353 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:54.670257    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92056a3e-aacd-45ed-9f05-a360ad3c8b21-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zz8lz\" (UID: \"92056a3e-aacd-45ed-9f05-a360ad3c8b21\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zz8lz"
Sep 11 23:11:54.670353 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:54.670289    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92056a3e-aacd-45ed-9f05-a360ad3c8b21-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zz8lz\" (UID: \"92056a3e-aacd-45ed-9f05-a360ad3c8b21\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zz8lz"
Sep 11 23:11:54.670353 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:54.670340    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92056a3e-aacd-45ed-9f05-a360ad3c8b21-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zz8lz\" (UID: \"92056a3e-aacd-45ed-9f05-a360ad3c8b21\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zz8lz"
Sep 11 23:11:54.670583 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:54.670388    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92056a3e-aacd-45ed-9f05-a360ad3c8b21-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zz8lz\" (UID: \"92056a3e-aacd-45ed-9f05-a360ad3c8b21\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zz8lz"
Sep 11 23:11:54.670583 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:54.670424    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92056a3e-aacd-45ed-9f05-a360ad3c8b21-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zz8lz\" (UID: \"92056a3e-aacd-45ed-9f05-a360ad3c8b21\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zz8lz"
Sep 11 23:11:54.771322 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:54.771285    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92056a3e-aacd-45ed-9f05-a360ad3c8b21-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zz8lz\" (UID: \"92056a3e-aacd-45ed-9f05-a360ad3c8b21\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zz8lz"
Sep 11 23:11:54.771322 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:54.771319    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92056a3e-aacd-45ed-9f05-a360ad3c8b21-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zz8lz\" (UID: \"92056a3e-aacd-45ed-9f05-a360ad3c8b21\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zz8lz"
Sep 11 23:11:54.771621 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:54.771353    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92056a3e-aacd-45ed-9f05-a360ad3c8b21-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zz8lz\" (UID: \"92056a3e-aacd-45ed-9f05-a360ad3c8b21\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zz8lz"
Sep 11 23:11:54.771621 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:54.771371    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92056a3e-aacd-45ed-9f05-a360ad3c8b21-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zz8lz\" (UID: \"92056a3e-aacd-45ed-9f05-a360ad3c8b21\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zz8lz"
Sep 11 23:11:54.771621 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:54.771396    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92056a3e-aacd-45ed-9f05-a360ad3c8b21-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zz8lz\" (UID: \"92056a3e-aacd-45ed-9f05-a360ad3c8b21\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zz8lz"
Sep 11 23:11:54.771621 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:54.771432    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92056a3e-aacd-45ed-9f05-a360ad3c8b21-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zz8lz\" (UID: \"92056a3e-aacd-45ed-9f05-a360ad3c8b21\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zz8lz"
Sep 11 23:11:54.771621 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:54.771451    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92056a3e-aacd-45ed-9f05-a360ad3c8b21-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zz8lz\" (UID: \"92056a3e-aacd-45ed-9f05-a360ad3c8b21\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zz8lz"
Sep 11 23:11:54.771621 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:54.771472    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92056a3e-aacd-45ed-9f05-a360ad3c8b21-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zz8lz\" (UID: \"92056a3e-aacd-45ed-9f05-a360ad3c8b21\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zz8lz"
Sep 11 23:11:54.771621 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:54.771521    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvzks\" (UniqueName: \"kubernetes.io/projected/92056a3e-aacd-45ed-9f05-a360ad3c8b21-kube-api-access-rvzks\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zz8lz\" (UID: \"92056a3e-aacd-45ed-9f05-a360ad3c8b21\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zz8lz"
Sep 11 23:11:54.771979 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:54.771833    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92056a3e-aacd-45ed-9f05-a360ad3c8b21-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zz8lz\" (UID: \"92056a3e-aacd-45ed-9f05-a360ad3c8b21\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zz8lz"
Sep 11 23:11:54.771979 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:54.771913    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92056a3e-aacd-45ed-9f05-a360ad3c8b21-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zz8lz\" (UID: \"92056a3e-aacd-45ed-9f05-a360ad3c8b21\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zz8lz"
Sep 11 23:11:54.771979 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:54.771973    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92056a3e-aacd-45ed-9f05-a360ad3c8b21-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zz8lz\" (UID: \"92056a3e-aacd-45ed-9f05-a360ad3c8b21\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zz8lz"
Sep 11 23:11:54.772119 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:54.772011    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92056a3e-aacd-45ed-9f05-a360ad3c8b21-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zz8lz\" (UID: \"92056a3e-aacd-45ed-9f05-a360ad3c8b21\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zz8lz"
Sep 11 23:11:54.772119 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:54.772112    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92056a3e-aacd-45ed-9f05-a360ad3c8b21-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zz8lz\" (UID: \"92056a3e-aacd-45ed-9f05-a360ad3c8b21\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zz8lz"
Sep 11 23:11:54.774148 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:54.774126    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92056a3e-aacd-45ed-9f05-a360ad3c8b21-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zz8lz\" (UID: \"92056a3e-aacd-45ed-9f05-a360ad3c8b21\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zz8lz"
Sep 11 23:11:54.774256 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:54.774214    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92056a3e-aacd-45ed-9f05-a360ad3c8b21-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zz8lz\" (UID: \"92056a3e-aacd-45ed-9f05-a360ad3c8b21\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zz8lz"
Sep 11 23:11:54.781060 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:54.781025    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92056a3e-aacd-45ed-9f05-a360ad3c8b21-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zz8lz\" (UID: \"92056a3e-aacd-45ed-9f05-a360ad3c8b21\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zz8lz"
Sep 11 23:11:54.781327 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:54.781304    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvzks\" (UniqueName: \"kubernetes.io/projected/92056a3e-aacd-45ed-9f05-a360ad3c8b21-kube-api-access-rvzks\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zz8lz\" (UID: \"92056a3e-aacd-45ed-9f05-a360ad3c8b21\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zz8lz"
Sep 11 23:11:54.883714 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:54.883629    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zz8lz"
Sep 11 23:11:55.161209 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:55.161180    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zz8lz"]
Sep 11 23:11:55.162620 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:11:55.162594    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92056a3e_aacd_45ed_9f05_a360ad3c8b21.slice/crio-fbe8f19fa7985c5ddc18884ed69b4e2e14f38f19b04585019f10d5414330f631 WatchSource:0}: Error finding container fbe8f19fa7985c5ddc18884ed69b4e2e14f38f19b04585019f10d5414330f631: Status 404 returned error can't find the container with id fbe8f19fa7985c5ddc18884ed69b4e2e14f38f19b04585019f10d5414330f631
Sep 11 23:11:55.984639 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:55.984601    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zz8lz" event={"ID":"92056a3e-aacd-45ed-9f05-a360ad3c8b21","Type":"ContainerStarted","Data":"fbe8f19fa7985c5ddc18884ed69b4e2e14f38f19b04585019f10d5414330f631"}
Sep 11 23:11:57.575102 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:57.575059    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-wtlq2"]
Sep 11 23:11:57.582719 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:57.582691    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-wtlq2"
Sep 11 23:11:57.586179 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:57.586127    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-k9xt6\""
Sep 11 23:11:57.586179 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:57.586127    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Sep 11 23:11:57.590501 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:57.590475    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-wtlq2"]
Sep 11 23:11:57.696975 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:57.696923    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/66a2b731-2067-4a0d-99bd-29385ad13e94-cert\") pod \"odh-model-controller-7998c67dc4-wtlq2\" (UID: \"66a2b731-2067-4a0d-99bd-29385ad13e94\") " pod="opendatahub/odh-model-controller-7998c67dc4-wtlq2"
Sep 11 23:11:57.697189 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:57.697075    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcs7m\" (UniqueName: \"kubernetes.io/projected/66a2b731-2067-4a0d-99bd-29385ad13e94-kube-api-access-dcs7m\") pod \"odh-model-controller-7998c67dc4-wtlq2\" (UID: \"66a2b731-2067-4a0d-99bd-29385ad13e94\") " pod="opendatahub/odh-model-controller-7998c67dc4-wtlq2"
Sep 11 23:11:57.797864 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:57.797825    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/66a2b731-2067-4a0d-99bd-29385ad13e94-cert\") pod \"odh-model-controller-7998c67dc4-wtlq2\" (UID: \"66a2b731-2067-4a0d-99bd-29385ad13e94\") " pod="opendatahub/odh-model-controller-7998c67dc4-wtlq2"
Sep 11 23:11:57.798057 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:57.797912    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcs7m\" (UniqueName: \"kubernetes.io/projected/66a2b731-2067-4a0d-99bd-29385ad13e94-kube-api-access-dcs7m\") pod \"odh-model-controller-7998c67dc4-wtlq2\" (UID: \"66a2b731-2067-4a0d-99bd-29385ad13e94\") " pod="opendatahub/odh-model-controller-7998c67dc4-wtlq2"
Sep 11 23:11:57.798057 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:11:57.797991    2546 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 11 23:11:57.798206 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:11:57.798074    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66a2b731-2067-4a0d-99bd-29385ad13e94-cert podName:66a2b731-2067-4a0d-99bd-29385ad13e94 nodeName:}" failed. No retries permitted until 2026-09-11 23:11:58.298057617 +0000 UTC m=+642.331186832 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/66a2b731-2067-4a0d-99bd-29385ad13e94-cert") pod "odh-model-controller-7998c67dc4-wtlq2" (UID: "66a2b731-2067-4a0d-99bd-29385ad13e94") : secret "odh-model-controller-webhook-cert" not found
Sep 11 23:11:57.807337 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:57.807311    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcs7m\" (UniqueName: \"kubernetes.io/projected/66a2b731-2067-4a0d-99bd-29385ad13e94-kube-api-access-dcs7m\") pod \"odh-model-controller-7998c67dc4-wtlq2\" (UID: \"66a2b731-2067-4a0d-99bd-29385ad13e94\") " pod="opendatahub/odh-model-controller-7998c67dc4-wtlq2"
Sep 11 23:11:57.884163 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:57.884107    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:11:57.884251 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:57.884234    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:11:57.884289 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:57.884271    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:11:57.972939 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:57.972897    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-v8jrh"]
Sep 11 23:11:57.976874 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:57.976853    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-v8jrh"
Sep 11 23:11:57.979658 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:57.979634    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Sep 11 23:11:57.989518 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:57.989487    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-v8jrh"]
Sep 11 23:11:57.999034 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:57.998998    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zz8lz" event={"ID":"92056a3e-aacd-45ed-9f05-a360ad3c8b21","Type":"ContainerStarted","Data":"b29df88350abff0c07b40265bc4b1723d74f3b2be9d75cfc57a6a7d653b052c3"}
Sep 11 23:11:57.999215 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:57.999078    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8mns\" (UniqueName: \"kubernetes.io/projected/f4aa47f4-30c0-4e92-94d9-3b9b00889ca9-kube-api-access-t8mns\") pod \"model-serving-api-54d89bd5fc-v8jrh\" (UID: \"f4aa47f4-30c0-4e92-94d9-3b9b00889ca9\") " pod="opendatahub/model-serving-api-54d89bd5fc-v8jrh"
Sep 11 23:11:57.999215 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:57.999129    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4aa47f4-30c0-4e92-94d9-3b9b00889ca9-tls-certs\") pod \"model-serving-api-54d89bd5fc-v8jrh\" (UID: \"f4aa47f4-30c0-4e92-94d9-3b9b00889ca9\") " pod="opendatahub/model-serving-api-54d89bd5fc-v8jrh"
Sep 11 23:11:58.019590 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:58.019381    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zz8lz" podStartSLOduration=1.300138199 podStartE2EDuration="4.019358523s" totalImagesPullingTime="2.719220324s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 23:11:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:11:58.017376496 +0000 UTC m=+642.050505715" watchObservedRunningTime="2026-09-11 23:11:58.019358523 +0000 UTC m=+642.052487741"
Sep 11 23:11:58.099770 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:58.099739    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4aa47f4-30c0-4e92-94d9-3b9b00889ca9-tls-certs\") pod \"model-serving-api-54d89bd5fc-v8jrh\" (UID: \"f4aa47f4-30c0-4e92-94d9-3b9b00889ca9\") " pod="opendatahub/model-serving-api-54d89bd5fc-v8jrh"
Sep 11 23:11:58.099986 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:58.099887    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8mns\" (UniqueName: \"kubernetes.io/projected/f4aa47f4-30c0-4e92-94d9-3b9b00889ca9-kube-api-access-t8mns\") pod \"model-serving-api-54d89bd5fc-v8jrh\" (UID: \"f4aa47f4-30c0-4e92-94d9-3b9b00889ca9\") " pod="opendatahub/model-serving-api-54d89bd5fc-v8jrh"
Sep 11 23:11:58.102572 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:58.102546    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4aa47f4-30c0-4e92-94d9-3b9b00889ca9-tls-certs\") pod \"model-serving-api-54d89bd5fc-v8jrh\" (UID: \"f4aa47f4-30c0-4e92-94d9-3b9b00889ca9\") " pod="opendatahub/model-serving-api-54d89bd5fc-v8jrh"
Sep 11 23:11:58.108813 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:58.108773    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8mns\" (UniqueName: \"kubernetes.io/projected/f4aa47f4-30c0-4e92-94d9-3b9b00889ca9-kube-api-access-t8mns\") pod \"model-serving-api-54d89bd5fc-v8jrh\" (UID: \"f4aa47f4-30c0-4e92-94d9-3b9b00889ca9\") " pod="opendatahub/model-serving-api-54d89bd5fc-v8jrh"
Sep 11 23:11:58.290505 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:58.290465    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-v8jrh"
Sep 11 23:11:58.301702 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:58.301677    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/66a2b731-2067-4a0d-99bd-29385ad13e94-cert\") pod \"odh-model-controller-7998c67dc4-wtlq2\" (UID: \"66a2b731-2067-4a0d-99bd-29385ad13e94\") " pod="opendatahub/odh-model-controller-7998c67dc4-wtlq2"
Sep 11 23:11:58.304318 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:58.304293    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/66a2b731-2067-4a0d-99bd-29385ad13e94-cert\") pod \"odh-model-controller-7998c67dc4-wtlq2\" (UID: \"66a2b731-2067-4a0d-99bd-29385ad13e94\") " pod="opendatahub/odh-model-controller-7998c67dc4-wtlq2"
Sep 11 23:11:58.498656 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:58.498626    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-wtlq2"
Sep 11 23:11:58.569731 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:58.569706    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-v8jrh"]
Sep 11 23:11:58.571249 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:11:58.571219    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4aa47f4_30c0_4e92_94d9_3b9b00889ca9.slice/crio-270cbdc17b557a49ba493bdbd3e2e08ca4c66c17d641f40f380bc750fb6003f7 WatchSource:0}: Error finding container 270cbdc17b557a49ba493bdbd3e2e08ca4c66c17d641f40f380bc750fb6003f7: Status 404 returned error can't find the container with id 270cbdc17b557a49ba493bdbd3e2e08ca4c66c17d641f40f380bc750fb6003f7
Sep 11 23:11:58.772897 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:58.772868    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-wtlq2"]
Sep 11 23:11:58.775449 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:11:58.775423    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66a2b731_2067_4a0d_99bd_29385ad13e94.slice/crio-530463dc32c24b262a454de74410adcbdf5abcac5d09b7b788d2a0c70522bd3c WatchSource:0}: Error finding container 530463dc32c24b262a454de74410adcbdf5abcac5d09b7b788d2a0c70522bd3c: Status 404 returned error can't find the container with id 530463dc32c24b262a454de74410adcbdf5abcac5d09b7b788d2a0c70522bd3c
Sep 11 23:11:58.884308 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:58.884228    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zz8lz"
Sep 11 23:11:58.888595 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:58.888568    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zz8lz"
Sep 11 23:11:59.004905 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:59.004868    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-v8jrh" event={"ID":"f4aa47f4-30c0-4e92-94d9-3b9b00889ca9","Type":"ContainerStarted","Data":"270cbdc17b557a49ba493bdbd3e2e08ca4c66c17d641f40f380bc750fb6003f7"}
Sep 11 23:11:59.006168 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:59.006133    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-wtlq2" event={"ID":"66a2b731-2067-4a0d-99bd-29385ad13e94","Type":"ContainerStarted","Data":"530463dc32c24b262a454de74410adcbdf5abcac5d09b7b788d2a0c70522bd3c"}
Sep 11 23:11:59.006435 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:59.006422    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zz8lz"
Sep 11 23:11:59.007517 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:11:59.007499    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zz8lz"
Sep 11 23:12:02.034128 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:12:02.034080    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-wtlq2" event={"ID":"66a2b731-2067-4a0d-99bd-29385ad13e94","Type":"ContainerStarted","Data":"be26a2018a141b3183ecd8a8f6eaa759977f7cfa8bc097ceb16fdd5e37075cd4"}
Sep 11 23:12:02.034622 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:12:02.034288    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-wtlq2"
Sep 11 23:12:02.036178 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:12:02.036123    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-v8jrh" event={"ID":"f4aa47f4-30c0-4e92-94d9-3b9b00889ca9","Type":"ContainerStarted","Data":"b1a1d36e7012915c608cb29520662692bcceb0447f24657e2658fc2c6b4bdf7c"}
Sep 11 23:12:02.036306 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:12:02.036213    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-v8jrh"
Sep 11 23:12:02.053604 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:12:02.053557    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-wtlq2" podStartSLOduration=2.208515942 podStartE2EDuration="5.053542484s" totalImagesPullingTime="2.845026542s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 23:11:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:12:02.051170296 +0000 UTC m=+646.084299510" watchObservedRunningTime="2026-09-11 23:12:02.053542484 +0000 UTC m=+646.086671701"
Sep 11 23:12:02.074920 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:12:02.074867    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-v8jrh" podStartSLOduration=2.083388037 podStartE2EDuration="5.0748499s" totalImagesPullingTime="2.991461863s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 23:11:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:12:02.073326683 +0000 UTC m=+646.106455901" watchObservedRunningTime="2026-09-11 23:12:02.0748499 +0000 UTC m=+646.107979117"
Sep 11 23:12:04.440072 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:12:04.440034    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-qxgqr"]
Sep 11 23:12:04.446883 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:12:04.446861    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qxgqr"
Sep 11 23:12:04.449792 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:12:04.449771    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 11 23:12:04.449902 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:12:04.449827    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 11 23:12:04.449902 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:12:04.449878    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-h4lwq\""
Sep 11 23:12:04.450120 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:12:04.450106    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 11 23:12:04.457564 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:12:04.457537    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-qxgqr"]
Sep 11 23:12:04.559383 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:12:04.559342    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f445a382-8507-45a5-bc72-5d760e31efe7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-qxgqr\" (UID: \"f445a382-8507-45a5-bc72-5d760e31efe7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qxgqr"
Sep 11 23:12:04.559702 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:12:04.559679    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f445a382-8507-45a5-bc72-5d760e31efe7-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-qxgqr\" (UID: \"f445a382-8507-45a5-bc72-5d760e31efe7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qxgqr"
Sep 11 23:12:04.559886 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:12:04.559869    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f445a382-8507-45a5-bc72-5d760e31efe7-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-qxgqr\" (UID: \"f445a382-8507-45a5-bc72-5d760e31efe7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qxgqr"
Sep 11 23:12:04.560022 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:12:04.560008    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f445a382-8507-45a5-bc72-5d760e31efe7-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-qxgqr\" (UID: \"f445a382-8507-45a5-bc72-5d760e31efe7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qxgqr"
Sep 11 23:12:04.560207 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:12:04.560191    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f445a382-8507-45a5-bc72-5d760e31efe7-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-qxgqr\" (UID: \"f445a382-8507-45a5-bc72-5d760e31efe7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qxgqr"
Sep 11 23:12:04.560373 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:12:04.560336    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f445a382-8507-45a5-bc72-5d760e31efe7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-qxgqr\" (UID: \"f445a382-8507-45a5-bc72-5d760e31efe7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qxgqr"
Sep 11 23:12:04.560373 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:12:04.560372    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6h5g\" (UniqueName: \"kubernetes.io/projected/f445a382-8507-45a5-bc72-5d760e31efe7-kube-api-access-h6h5g\") pod \"istiod-openshift-gateway-5677ccb49b-qxgqr\" (UID: \"f445a382-8507-45a5-bc72-5d760e31efe7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qxgqr"
Sep 11 23:12:04.661303 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:12:04.661255    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f445a382-8507-45a5-bc72-5d760e31efe7-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-qxgqr\" (UID: \"f445a382-8507-45a5-bc72-5d760e31efe7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qxgqr"
Sep 11 23:12:04.661505 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:12:04.661309    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f445a382-8507-45a5-bc72-5d760e31efe7-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-qxgqr\" (UID: \"f445a382-8507-45a5-bc72-5d760e31efe7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qxgqr"
Sep 11 23:12:04.661505 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:12:04.661360    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f445a382-8507-45a5-bc72-5d760e31efe7-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-qxgqr\" (UID: \"f445a382-8507-45a5-bc72-5d760e31efe7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qxgqr"
Sep 11 23:12:04.661505 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:12:04.661387    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f445a382-8507-45a5-bc72-5d760e31efe7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-qxgqr\" (UID: \"f445a382-8507-45a5-bc72-5d760e31efe7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qxgqr"
Sep 11 23:12:04.661818 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:12:04.661790    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6h5g\" (UniqueName: \"kubernetes.io/projected/f445a382-8507-45a5-bc72-5d760e31efe7-kube-api-access-h6h5g\") pod \"istiod-openshift-gateway-5677ccb49b-qxgqr\" (UID: \"f445a382-8507-45a5-bc72-5d760e31efe7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qxgqr"
Sep 11 23:12:04.661996 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:12:04.661979    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f445a382-8507-45a5-bc72-5d760e31efe7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-qxgqr\" (UID: \"f445a382-8507-45a5-bc72-5d760e31efe7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qxgqr"
Sep 11 23:12:04.662125 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:12:04.662109    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f445a382-8507-45a5-bc72-5d760e31efe7-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-qxgqr\" (UID: \"f445a382-8507-45a5-bc72-5d760e31efe7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qxgqr"
Sep 11 23:12:04.662590 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:12:04.662541    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f445a382-8507-45a5-bc72-5d760e31efe7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-qxgqr\" (UID: \"f445a382-8507-45a5-bc72-5d760e31efe7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qxgqr"
Sep 11 23:12:04.664485 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:12:04.664458    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f445a382-8507-45a5-bc72-5d760e31efe7-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-qxgqr\" (UID: \"f445a382-8507-45a5-bc72-5d760e31efe7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qxgqr"
Sep 11 23:12:04.664591 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:12:04.664562    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f445a382-8507-45a5-bc72-5d760e31efe7-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-qxgqr\" (UID: \"f445a382-8507-45a5-bc72-5d760e31efe7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qxgqr"
Sep 11 23:12:04.664894 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:12:04.664875    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f445a382-8507-45a5-bc72-5d760e31efe7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-qxgqr\" (UID: \"f445a382-8507-45a5-bc72-5d760e31efe7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qxgqr"
Sep 11 23:12:04.665058 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:12:04.665026    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f445a382-8507-45a5-bc72-5d760e31efe7-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-qxgqr\" (UID: \"f445a382-8507-45a5-bc72-5d760e31efe7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qxgqr"
Sep 11 23:12:04.682768 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:12:04.682737    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f445a382-8507-45a5-bc72-5d760e31efe7-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-qxgqr\" (UID: \"f445a382-8507-45a5-bc72-5d760e31efe7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qxgqr"
Sep 11 23:12:04.687837 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:12:04.687810    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6h5g\" (UniqueName: \"kubernetes.io/projected/f445a382-8507-45a5-bc72-5d760e31efe7-kube-api-access-h6h5g\") pod \"istiod-openshift-gateway-5677ccb49b-qxgqr\" (UID: \"f445a382-8507-45a5-bc72-5d760e31efe7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qxgqr"
Sep 11 23:12:04.757598 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:12:04.757524    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qxgqr"
Sep 11 23:12:05.069106 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:12:05.069076    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-qxgqr"]
Sep 11 23:12:05.071334 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:12:05.071308    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf445a382_8507_45a5_bc72_5d760e31efe7.slice/crio-53ed22462d8eedcdc97730d8e479b590149cc9c015c3f3f12eaa8d14fee7d060 WatchSource:0}: Error finding container 53ed22462d8eedcdc97730d8e479b590149cc9c015c3f3f12eaa8d14fee7d060: Status 404 returned error can't find the container with id 53ed22462d8eedcdc97730d8e479b590149cc9c015c3f3f12eaa8d14fee7d060
Sep 11 23:12:06.062181 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:12:06.062114    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qxgqr" event={"ID":"f445a382-8507-45a5-bc72-5d760e31efe7","Type":"ContainerStarted","Data":"53ed22462d8eedcdc97730d8e479b590149cc9c015c3f3f12eaa8d14fee7d060"}
Sep 11 23:12:07.615234 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:12:07.615192    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:12:07.615502 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:12:07.615270    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:12:08.079983 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:12:08.079943    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qxgqr" event={"ID":"f445a382-8507-45a5-bc72-5d760e31efe7","Type":"ContainerStarted","Data":"a1528e268c3c263c4cf65236ef3d538f034e794dfcf263bd40bbc2e3a36cf9ac"}
Sep 11 23:12:08.080196 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:12:08.080124    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qxgqr"
Sep 11 23:12:08.082035 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:12:08.082010    2546 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-qxgqr container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 11 23:12:08.082180 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:12:08.082056    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qxgqr" podUID="f445a382-8507-45a5-bc72-5d760e31efe7" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 23:12:08.130819 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:12:08.130751    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qxgqr" podStartSLOduration=1.589298648 podStartE2EDuration="4.130733993s" totalImagesPullingTime="2.541435345s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 23:12:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:12:08.127851908 +0000 UTC m=+652.160981124" watchObservedRunningTime="2026-09-11 23:12:08.130733993 +0000 UTC m=+652.163863211"
Sep 11 23:12:09.086539 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:12:09.086511    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qxgqr"
Sep 11 23:12:13.044856 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:12:13.044824    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-wtlq2"
Sep 11 23:12:13.047917 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:12:13.047899    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-v8jrh"
Sep 11 23:13:02.949404 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:02.949365    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atlzps"]
Sep 11 23:13:02.953125 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:02.953108    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atlzps"
Sep 11 23:13:02.956231 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:02.956207    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 11 23:13:02.956373 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:02.956355    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 11 23:13:02.957856 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:02.957839    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-qbxpl\""
Sep 11 23:13:02.964448 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:02.964425    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atlzps"]
Sep 11 23:13:03.045670 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:03.045639    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cr8m2d"]
Sep 11 23:13:03.049478 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:03.049455    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cr8m2d"
Sep 11 23:13:03.058869 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:03.058843    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cr8m2d"]
Sep 11 23:13:03.059672 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:03.059644    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drzq8\" (UniqueName: \"kubernetes.io/projected/4f598cb3-2b1c-4078-ab0e-2e3fe44714e3-kube-api-access-drzq8\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atlzps\" (UID: \"4f598cb3-2b1c-4078-ab0e-2e3fe44714e3\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atlzps"
Sep 11 23:13:03.059768 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:03.059693    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4f598cb3-2b1c-4078-ab0e-2e3fe44714e3-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atlzps\" (UID: \"4f598cb3-2b1c-4078-ab0e-2e3fe44714e3\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atlzps"
Sep 11 23:13:03.059815 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:03.059793    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4f598cb3-2b1c-4078-ab0e-2e3fe44714e3-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atlzps\" (UID: \"4f598cb3-2b1c-4078-ab0e-2e3fe44714e3\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atlzps"
Sep 11 23:13:03.142779 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:03.142746    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fl9tz"]
Sep 11 23:13:03.150726 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:03.150695    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fl9tz"
Sep 11 23:13:03.156101 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:03.156074    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fl9tz"]
Sep 11 23:13:03.160539 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:03.160510    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtw25\" (UniqueName: \"kubernetes.io/projected/98b51adc-75e0-4cfd-a0ea-a1aeba71bb1e-kube-api-access-vtw25\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cr8m2d\" (UID: \"98b51adc-75e0-4cfd-a0ea-a1aeba71bb1e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cr8m2d"
Sep 11 23:13:03.160676 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:03.160633    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drzq8\" (UniqueName: \"kubernetes.io/projected/4f598cb3-2b1c-4078-ab0e-2e3fe44714e3-kube-api-access-drzq8\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atlzps\" (UID: \"4f598cb3-2b1c-4078-ab0e-2e3fe44714e3\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atlzps"
Sep 11 23:13:03.160749 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:03.160675    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/98b51adc-75e0-4cfd-a0ea-a1aeba71bb1e-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cr8m2d\" (UID: \"98b51adc-75e0-4cfd-a0ea-a1aeba71bb1e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cr8m2d"
Sep 11 23:13:03.160749 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:03.160707    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4f598cb3-2b1c-4078-ab0e-2e3fe44714e3-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atlzps\" (UID: \"4f598cb3-2b1c-4078-ab0e-2e3fe44714e3\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atlzps"
Sep 11 23:13:03.160872 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:03.160749    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/98b51adc-75e0-4cfd-a0ea-a1aeba71bb1e-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cr8m2d\" (UID: \"98b51adc-75e0-4cfd-a0ea-a1aeba71bb1e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cr8m2d"
Sep 11 23:13:03.160872 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:03.160812    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4f598cb3-2b1c-4078-ab0e-2e3fe44714e3-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atlzps\" (UID: \"4f598cb3-2b1c-4078-ab0e-2e3fe44714e3\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atlzps"
Sep 11 23:13:03.161079 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:03.161059    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4f598cb3-2b1c-4078-ab0e-2e3fe44714e3-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atlzps\" (UID: \"4f598cb3-2b1c-4078-ab0e-2e3fe44714e3\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atlzps"
Sep 11 23:13:03.161186 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:03.161134    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4f598cb3-2b1c-4078-ab0e-2e3fe44714e3-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atlzps\" (UID: \"4f598cb3-2b1c-4078-ab0e-2e3fe44714e3\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atlzps"
Sep 11 23:13:03.172831 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:03.172809    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drzq8\" (UniqueName: \"kubernetes.io/projected/4f598cb3-2b1c-4078-ab0e-2e3fe44714e3-kube-api-access-drzq8\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atlzps\" (UID: \"4f598cb3-2b1c-4078-ab0e-2e3fe44714e3\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atlzps"
Sep 11 23:13:03.252277 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:03.252191    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mhd7k"]
Sep 11 23:13:03.256031 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:03.256014    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mhd7k"
Sep 11 23:13:03.262024 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:03.261988    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtw25\" (UniqueName: \"kubernetes.io/projected/98b51adc-75e0-4cfd-a0ea-a1aeba71bb1e-kube-api-access-vtw25\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cr8m2d\" (UID: \"98b51adc-75e0-4cfd-a0ea-a1aeba71bb1e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cr8m2d"
Sep 11 23:13:03.262146 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:03.262045    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9794\" (UniqueName: \"kubernetes.io/projected/3496c41f-995e-45e8-93fc-c9491d3354c2-kube-api-access-f9794\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fl9tz\" (UID: \"3496c41f-995e-45e8-93fc-c9491d3354c2\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fl9tz"
Sep 11 23:13:03.262146 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:03.262107    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3496c41f-995e-45e8-93fc-c9491d3354c2-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fl9tz\" (UID: \"3496c41f-995e-45e8-93fc-c9491d3354c2\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fl9tz"
Sep 11 23:13:03.262243 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:03.262173    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/98b51adc-75e0-4cfd-a0ea-a1aeba71bb1e-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cr8m2d\" (UID: \"98b51adc-75e0-4cfd-a0ea-a1aeba71bb1e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cr8m2d"
Sep 11 23:13:03.262243 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:03.262212    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/98b51adc-75e0-4cfd-a0ea-a1aeba71bb1e-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cr8m2d\" (UID: \"98b51adc-75e0-4cfd-a0ea-a1aeba71bb1e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cr8m2d"
Sep 11 23:13:03.262310 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:03.262270    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3496c41f-995e-45e8-93fc-c9491d3354c2-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fl9tz\" (UID: \"3496c41f-995e-45e8-93fc-c9491d3354c2\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fl9tz"
Sep 11 23:13:03.262598 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:03.262572    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/98b51adc-75e0-4cfd-a0ea-a1aeba71bb1e-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cr8m2d\" (UID: \"98b51adc-75e0-4cfd-a0ea-a1aeba71bb1e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cr8m2d"
Sep 11 23:13:03.262721 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:03.262624    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/98b51adc-75e0-4cfd-a0ea-a1aeba71bb1e-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cr8m2d\" (UID: \"98b51adc-75e0-4cfd-a0ea-a1aeba71bb1e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cr8m2d"
Sep 11 23:13:03.263753 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:03.263727    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atlzps"
Sep 11 23:13:03.265721 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:03.265699    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mhd7k"]
Sep 11 23:13:03.271024 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:03.271000    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtw25\" (UniqueName: \"kubernetes.io/projected/98b51adc-75e0-4cfd-a0ea-a1aeba71bb1e-kube-api-access-vtw25\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cr8m2d\" (UID: \"98b51adc-75e0-4cfd-a0ea-a1aeba71bb1e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cr8m2d"
Sep 11 23:13:03.358924 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:03.358894    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cr8m2d"
Sep 11 23:13:03.362945 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:03.362795    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8606c1c6-9f26-4c93-a3f0-f8cb76ee96ab-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mhd7k\" (UID: \"8606c1c6-9f26-4c93-a3f0-f8cb76ee96ab\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mhd7k"
Sep 11 23:13:03.362945 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:03.362843    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3496c41f-995e-45e8-93fc-c9491d3354c2-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fl9tz\" (UID: \"3496c41f-995e-45e8-93fc-c9491d3354c2\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fl9tz"
Sep 11 23:13:03.362945 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:03.362926    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9794\" (UniqueName: \"kubernetes.io/projected/3496c41f-995e-45e8-93fc-c9491d3354c2-kube-api-access-f9794\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fl9tz\" (UID: \"3496c41f-995e-45e8-93fc-c9491d3354c2\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fl9tz"
Sep 11 23:13:03.363100 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:03.362987    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8606c1c6-9f26-4c93-a3f0-f8cb76ee96ab-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mhd7k\" (UID: \"8606c1c6-9f26-4c93-a3f0-f8cb76ee96ab\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mhd7k"
Sep 11 23:13:03.363100 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:03.363015    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvvpc\" (UniqueName: \"kubernetes.io/projected/8606c1c6-9f26-4c93-a3f0-f8cb76ee96ab-kube-api-access-bvvpc\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mhd7k\" (UID: \"8606c1c6-9f26-4c93-a3f0-f8cb76ee96ab\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mhd7k"
Sep 11 23:13:03.363100 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:03.363076    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3496c41f-995e-45e8-93fc-c9491d3354c2-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fl9tz\" (UID: \"3496c41f-995e-45e8-93fc-c9491d3354c2\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fl9tz"
Sep 11 23:13:03.363302 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:03.363184    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3496c41f-995e-45e8-93fc-c9491d3354c2-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fl9tz\" (UID: \"3496c41f-995e-45e8-93fc-c9491d3354c2\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fl9tz"
Sep 11 23:13:03.363428 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:03.363410    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3496c41f-995e-45e8-93fc-c9491d3354c2-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fl9tz\" (UID: \"3496c41f-995e-45e8-93fc-c9491d3354c2\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fl9tz"
Sep 11 23:13:03.372275 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:03.372253    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9794\" (UniqueName: \"kubernetes.io/projected/3496c41f-995e-45e8-93fc-c9491d3354c2-kube-api-access-f9794\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fl9tz\" (UID: \"3496c41f-995e-45e8-93fc-c9491d3354c2\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fl9tz"
Sep 11 23:13:03.461685 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:03.461654    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fl9tz"
Sep 11 23:13:03.464022 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:03.463994    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8606c1c6-9f26-4c93-a3f0-f8cb76ee96ab-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mhd7k\" (UID: \"8606c1c6-9f26-4c93-a3f0-f8cb76ee96ab\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mhd7k"
Sep 11 23:13:03.464117 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:03.464028    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvvpc\" (UniqueName: \"kubernetes.io/projected/8606c1c6-9f26-4c93-a3f0-f8cb76ee96ab-kube-api-access-bvvpc\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mhd7k\" (UID: \"8606c1c6-9f26-4c93-a3f0-f8cb76ee96ab\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mhd7k"
Sep 11 23:13:03.464117 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:03.464060    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8606c1c6-9f26-4c93-a3f0-f8cb76ee96ab-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mhd7k\" (UID: \"8606c1c6-9f26-4c93-a3f0-f8cb76ee96ab\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mhd7k"
Sep 11 23:13:03.464395 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:03.464380    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8606c1c6-9f26-4c93-a3f0-f8cb76ee96ab-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mhd7k\" (UID: \"8606c1c6-9f26-4c93-a3f0-f8cb76ee96ab\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mhd7k"
Sep 11 23:13:03.464445 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:03.464413    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8606c1c6-9f26-4c93-a3f0-f8cb76ee96ab-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mhd7k\" (UID: \"8606c1c6-9f26-4c93-a3f0-f8cb76ee96ab\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mhd7k"
Sep 11 23:13:03.473379 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:03.473352    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvvpc\" (UniqueName: \"kubernetes.io/projected/8606c1c6-9f26-4c93-a3f0-f8cb76ee96ab-kube-api-access-bvvpc\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mhd7k\" (UID: \"8606c1c6-9f26-4c93-a3f0-f8cb76ee96ab\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mhd7k"
Sep 11 23:13:03.542768 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:03.542740    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atlzps"]
Sep 11 23:13:03.544364 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:13:03.544337    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f598cb3_2b1c_4078_ab0e_2e3fe44714e3.slice/crio-c91190fff07142f94ae04a4ac5599c1e58fa86e088d231f5258eb03198b66884 WatchSource:0}: Error finding container c91190fff07142f94ae04a4ac5599c1e58fa86e088d231f5258eb03198b66884: Status 404 returned error can't find the container with id c91190fff07142f94ae04a4ac5599c1e58fa86e088d231f5258eb03198b66884
Sep 11 23:13:03.566348 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:03.566322    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mhd7k"
Sep 11 23:13:03.633679 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:03.633654    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cr8m2d"]
Sep 11 23:13:03.636313 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:13:03.636284    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98b51adc_75e0_4cfd_a0ea_a1aeba71bb1e.slice/crio-d2aead4737a2a4ccc9beb8f84e46fcdab5568c0e71dc4cfad2498395616b1054 WatchSource:0}: Error finding container d2aead4737a2a4ccc9beb8f84e46fcdab5568c0e71dc4cfad2498395616b1054: Status 404 returned error can't find the container with id d2aead4737a2a4ccc9beb8f84e46fcdab5568c0e71dc4cfad2498395616b1054
Sep 11 23:13:03.739110 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:03.739087    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fl9tz"]
Sep 11 23:13:03.740389 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:13:03.740366    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3496c41f_995e_45e8_93fc_c9491d3354c2.slice/crio-f46f4e6a05a86e38340b61ab6dacc77ed77354bae3a0cc1ec011f6d99c8b9ffc WatchSource:0}: Error finding container f46f4e6a05a86e38340b61ab6dacc77ed77354bae3a0cc1ec011f6d99c8b9ffc: Status 404 returned error can't find the container with id f46f4e6a05a86e38340b61ab6dacc77ed77354bae3a0cc1ec011f6d99c8b9ffc
Sep 11 23:13:03.843731 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:03.843707    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mhd7k"]
Sep 11 23:13:03.884951 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:13:03.884916    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8606c1c6_9f26_4c93_a3f0_f8cb76ee96ab.slice/crio-ee272d0ff959171744de262e25e01a9abd1dcd11509193f2ae50fb73fc628eb4 WatchSource:0}: Error finding container ee272d0ff959171744de262e25e01a9abd1dcd11509193f2ae50fb73fc628eb4: Status 404 returned error can't find the container with id ee272d0ff959171744de262e25e01a9abd1dcd11509193f2ae50fb73fc628eb4
Sep 11 23:13:04.373271 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:04.373237    2546 generic.go:356] "Generic (PLEG): container finished" podID="3496c41f-995e-45e8-93fc-c9491d3354c2" containerID="3a4a66b0678d71c9c3267de56dea332a473fd47595975698c2702ec6f1185e27" exitCode=0
Sep 11 23:13:04.373720 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:04.373325    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fl9tz" event={"ID":"3496c41f-995e-45e8-93fc-c9491d3354c2","Type":"ContainerDied","Data":"3a4a66b0678d71c9c3267de56dea332a473fd47595975698c2702ec6f1185e27"}
Sep 11 23:13:04.373720 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:04.373378    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fl9tz" event={"ID":"3496c41f-995e-45e8-93fc-c9491d3354c2","Type":"ContainerStarted","Data":"f46f4e6a05a86e38340b61ab6dacc77ed77354bae3a0cc1ec011f6d99c8b9ffc"}
Sep 11 23:13:04.375128 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:04.375107    2546 generic.go:356] "Generic (PLEG): container finished" podID="4f598cb3-2b1c-4078-ab0e-2e3fe44714e3" containerID="94e00b88c2ebd7203255e11d87e6cb29b1650e224cbce18f0358c0e47633fe78" exitCode=0
Sep 11 23:13:04.375226 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:04.375187    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atlzps" event={"ID":"4f598cb3-2b1c-4078-ab0e-2e3fe44714e3","Type":"ContainerDied","Data":"94e00b88c2ebd7203255e11d87e6cb29b1650e224cbce18f0358c0e47633fe78"}
Sep 11 23:13:04.375267 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:04.375239    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atlzps" event={"ID":"4f598cb3-2b1c-4078-ab0e-2e3fe44714e3","Type":"ContainerStarted","Data":"c91190fff07142f94ae04a4ac5599c1e58fa86e088d231f5258eb03198b66884"}
Sep 11 23:13:04.376952 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:04.376930    2546 generic.go:356] "Generic (PLEG): container finished" podID="98b51adc-75e0-4cfd-a0ea-a1aeba71bb1e" containerID="2ccec9c6590c63aa4e3d9bf42d88de9b5123ff08a5f57386b2d6e9c6d975cc5d" exitCode=0
Sep 11 23:13:04.377113 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:04.377014    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cr8m2d" event={"ID":"98b51adc-75e0-4cfd-a0ea-a1aeba71bb1e","Type":"ContainerDied","Data":"2ccec9c6590c63aa4e3d9bf42d88de9b5123ff08a5f57386b2d6e9c6d975cc5d"}
Sep 11 23:13:04.377113 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:04.377056    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cr8m2d" event={"ID":"98b51adc-75e0-4cfd-a0ea-a1aeba71bb1e","Type":"ContainerStarted","Data":"d2aead4737a2a4ccc9beb8f84e46fcdab5568c0e71dc4cfad2498395616b1054"}
Sep 11 23:13:04.378768 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:04.378749    2546 generic.go:356] "Generic (PLEG): container finished" podID="8606c1c6-9f26-4c93-a3f0-f8cb76ee96ab" containerID="fb7a1aea5637463b15833e7184c711a5523be598f57f248b186e336b947a6215" exitCode=0
Sep 11 23:13:04.378845 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:04.378774    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mhd7k" event={"ID":"8606c1c6-9f26-4c93-a3f0-f8cb76ee96ab","Type":"ContainerDied","Data":"fb7a1aea5637463b15833e7184c711a5523be598f57f248b186e336b947a6215"}
Sep 11 23:13:04.378845 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:04.378799    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mhd7k" event={"ID":"8606c1c6-9f26-4c93-a3f0-f8cb76ee96ab","Type":"ContainerStarted","Data":"ee272d0ff959171744de262e25e01a9abd1dcd11509193f2ae50fb73fc628eb4"}
Sep 11 23:13:05.388528 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:05.388495    2546 generic.go:356] "Generic (PLEG): container finished" podID="3496c41f-995e-45e8-93fc-c9491d3354c2" containerID="3d5ccdeac34107029791ffb74d4798ca3bc4e891e1bb21c8274bfb3917d33252" exitCode=0
Sep 11 23:13:05.389095 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:05.388647    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fl9tz" event={"ID":"3496c41f-995e-45e8-93fc-c9491d3354c2","Type":"ContainerDied","Data":"3d5ccdeac34107029791ffb74d4798ca3bc4e891e1bb21c8274bfb3917d33252"}
Sep 11 23:13:06.395551 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:06.395519    2546 generic.go:356] "Generic (PLEG): container finished" podID="8606c1c6-9f26-4c93-a3f0-f8cb76ee96ab" containerID="04700ccb2e91f84da1403958e814b8d0aaf3fa0cb287b42d876dfe13e368ea1d" exitCode=0
Sep 11 23:13:06.396009 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:06.395605    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mhd7k" event={"ID":"8606c1c6-9f26-4c93-a3f0-f8cb76ee96ab","Type":"ContainerDied","Data":"04700ccb2e91f84da1403958e814b8d0aaf3fa0cb287b42d876dfe13e368ea1d"}
Sep 11 23:13:06.398015 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:06.397992    2546 generic.go:356] "Generic (PLEG): container finished" podID="3496c41f-995e-45e8-93fc-c9491d3354c2" containerID="4177294d1901f2315ec9c6b412eb8a5e109081705e965c49529c9521e05bb342" exitCode=0
Sep 11 23:13:06.398114 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:06.398059    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fl9tz" event={"ID":"3496c41f-995e-45e8-93fc-c9491d3354c2","Type":"ContainerDied","Data":"4177294d1901f2315ec9c6b412eb8a5e109081705e965c49529c9521e05bb342"}
Sep 11 23:13:06.399976 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:06.399954    2546 generic.go:356] "Generic (PLEG): container finished" podID="4f598cb3-2b1c-4078-ab0e-2e3fe44714e3" containerID="180ff05e99a25418ad97b2c4341ee7300e79a7a07a7cfc5bdc3a97b8ef34b94b" exitCode=0
Sep 11 23:13:06.400071 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:06.400041    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atlzps" event={"ID":"4f598cb3-2b1c-4078-ab0e-2e3fe44714e3","Type":"ContainerDied","Data":"180ff05e99a25418ad97b2c4341ee7300e79a7a07a7cfc5bdc3a97b8ef34b94b"}
Sep 11 23:13:06.402239 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:06.402216    2546 generic.go:356] "Generic (PLEG): container finished" podID="98b51adc-75e0-4cfd-a0ea-a1aeba71bb1e" containerID="59550743fdeb0fc42a6beb420c01939257435473cee654701866ae880d9bf7c4" exitCode=0
Sep 11 23:13:06.402319 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:06.402296    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cr8m2d" event={"ID":"98b51adc-75e0-4cfd-a0ea-a1aeba71bb1e","Type":"ContainerDied","Data":"59550743fdeb0fc42a6beb420c01939257435473cee654701866ae880d9bf7c4"}
Sep 11 23:13:07.409522 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:07.409489    2546 generic.go:356] "Generic (PLEG): container finished" podID="4f598cb3-2b1c-4078-ab0e-2e3fe44714e3" containerID="e59310a099937194b96c6cd2659038cfc46175497b2226a22e6ae31d8ce0b7be" exitCode=0
Sep 11 23:13:07.409929 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:07.409546    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atlzps" event={"ID":"4f598cb3-2b1c-4078-ab0e-2e3fe44714e3","Type":"ContainerDied","Data":"e59310a099937194b96c6cd2659038cfc46175497b2226a22e6ae31d8ce0b7be"}
Sep 11 23:13:07.411705 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:07.411683    2546 generic.go:356] "Generic (PLEG): container finished" podID="98b51adc-75e0-4cfd-a0ea-a1aeba71bb1e" containerID="7193dc1ad6115405b87f145c5a5f46b837f15a1e9b96d2f6f8c44741ce929f8e" exitCode=0
Sep 11 23:13:07.411828 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:07.411753    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cr8m2d" event={"ID":"98b51adc-75e0-4cfd-a0ea-a1aeba71bb1e","Type":"ContainerDied","Data":"7193dc1ad6115405b87f145c5a5f46b837f15a1e9b96d2f6f8c44741ce929f8e"}
Sep 11 23:13:07.413902 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:07.413879    2546 generic.go:356] "Generic (PLEG): container finished" podID="8606c1c6-9f26-4c93-a3f0-f8cb76ee96ab" containerID="f47a8bcd6fb584b7a12136fae5df078a45ac2fb12d2891672f2b249e648377ea" exitCode=0
Sep 11 23:13:07.414013 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:07.413932    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mhd7k" event={"ID":"8606c1c6-9f26-4c93-a3f0-f8cb76ee96ab","Type":"ContainerDied","Data":"f47a8bcd6fb584b7a12136fae5df078a45ac2fb12d2891672f2b249e648377ea"}
Sep 11 23:13:07.548520 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:07.548489    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fl9tz"
Sep 11 23:13:07.599482 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:07.599447    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f9794\" (UniqueName: \"kubernetes.io/projected/3496c41f-995e-45e8-93fc-c9491d3354c2-kube-api-access-f9794\") pod \"3496c41f-995e-45e8-93fc-c9491d3354c2\" (UID: \"3496c41f-995e-45e8-93fc-c9491d3354c2\") "
Sep 11 23:13:07.599654 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:07.599547    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3496c41f-995e-45e8-93fc-c9491d3354c2-bundle\") pod \"3496c41f-995e-45e8-93fc-c9491d3354c2\" (UID: \"3496c41f-995e-45e8-93fc-c9491d3354c2\") "
Sep 11 23:13:07.599654 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:07.599581    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3496c41f-995e-45e8-93fc-c9491d3354c2-util\") pod \"3496c41f-995e-45e8-93fc-c9491d3354c2\" (UID: \"3496c41f-995e-45e8-93fc-c9491d3354c2\") "
Sep 11 23:13:07.600042 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:07.600017    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3496c41f-995e-45e8-93fc-c9491d3354c2-bundle" (OuterVolumeSpecName: "bundle") pod "3496c41f-995e-45e8-93fc-c9491d3354c2" (UID: "3496c41f-995e-45e8-93fc-c9491d3354c2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:13:07.601759 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:07.601733    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3496c41f-995e-45e8-93fc-c9491d3354c2-kube-api-access-f9794" (OuterVolumeSpecName: "kube-api-access-f9794") pod "3496c41f-995e-45e8-93fc-c9491d3354c2" (UID: "3496c41f-995e-45e8-93fc-c9491d3354c2"). InnerVolumeSpecName "kube-api-access-f9794". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 23:13:07.604890 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:07.604871    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3496c41f-995e-45e8-93fc-c9491d3354c2-util" (OuterVolumeSpecName: "util") pod "3496c41f-995e-45e8-93fc-c9491d3354c2" (UID: "3496c41f-995e-45e8-93fc-c9491d3354c2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:13:07.701169 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:07.701130    2546 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3496c41f-995e-45e8-93fc-c9491d3354c2-bundle\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:13:07.701340 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:07.701186    2546 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3496c41f-995e-45e8-93fc-c9491d3354c2-util\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:13:07.701340 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:07.701203    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9794\" (UniqueName: \"kubernetes.io/projected/3496c41f-995e-45e8-93fc-c9491d3354c2-kube-api-access-f9794\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:13:08.421876 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:08.421789    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fl9tz" event={"ID":"3496c41f-995e-45e8-93fc-c9491d3354c2","Type":"ContainerDied","Data":"f46f4e6a05a86e38340b61ab6dacc77ed77354bae3a0cc1ec011f6d99c8b9ffc"}
Sep 11 23:13:08.421876 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:08.421870    2546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f46f4e6a05a86e38340b61ab6dacc77ed77354bae3a0cc1ec011f6d99c8b9ffc"
Sep 11 23:13:08.422499 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:08.421812    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fl9tz"
Sep 11 23:13:08.558455 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:08.558420    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mhd7k"
Sep 11 23:13:08.597782 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:08.597756    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cr8m2d"
Sep 11 23:13:08.613748 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:08.613724    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8606c1c6-9f26-4c93-a3f0-f8cb76ee96ab-util\") pod \"8606c1c6-9f26-4c93-a3f0-f8cb76ee96ab\" (UID: \"8606c1c6-9f26-4c93-a3f0-f8cb76ee96ab\") "
Sep 11 23:13:08.613924 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:08.613847    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bvvpc\" (UniqueName: \"kubernetes.io/projected/8606c1c6-9f26-4c93-a3f0-f8cb76ee96ab-kube-api-access-bvvpc\") pod \"8606c1c6-9f26-4c93-a3f0-f8cb76ee96ab\" (UID: \"8606c1c6-9f26-4c93-a3f0-f8cb76ee96ab\") "
Sep 11 23:13:08.613924 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:08.613880    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8606c1c6-9f26-4c93-a3f0-f8cb76ee96ab-bundle\") pod \"8606c1c6-9f26-4c93-a3f0-f8cb76ee96ab\" (UID: \"8606c1c6-9f26-4c93-a3f0-f8cb76ee96ab\") "
Sep 11 23:13:08.614445 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:08.614417    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8606c1c6-9f26-4c93-a3f0-f8cb76ee96ab-bundle" (OuterVolumeSpecName: "bundle") pod "8606c1c6-9f26-4c93-a3f0-f8cb76ee96ab" (UID: "8606c1c6-9f26-4c93-a3f0-f8cb76ee96ab"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:13:08.616982 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:08.616955    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8606c1c6-9f26-4c93-a3f0-f8cb76ee96ab-kube-api-access-bvvpc" (OuterVolumeSpecName: "kube-api-access-bvvpc") pod "8606c1c6-9f26-4c93-a3f0-f8cb76ee96ab" (UID: "8606c1c6-9f26-4c93-a3f0-f8cb76ee96ab"). InnerVolumeSpecName "kube-api-access-bvvpc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 23:13:08.620739 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:08.620716    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8606c1c6-9f26-4c93-a3f0-f8cb76ee96ab-util" (OuterVolumeSpecName: "util") pod "8606c1c6-9f26-4c93-a3f0-f8cb76ee96ab" (UID: "8606c1c6-9f26-4c93-a3f0-f8cb76ee96ab"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:13:08.621540 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:08.621512    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atlzps"
Sep 11 23:13:08.714790 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:08.714703    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4f598cb3-2b1c-4078-ab0e-2e3fe44714e3-util\") pod \"4f598cb3-2b1c-4078-ab0e-2e3fe44714e3\" (UID: \"4f598cb3-2b1c-4078-ab0e-2e3fe44714e3\") "
Sep 11 23:13:08.714790 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:08.714748    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-drzq8\" (UniqueName: \"kubernetes.io/projected/4f598cb3-2b1c-4078-ab0e-2e3fe44714e3-kube-api-access-drzq8\") pod \"4f598cb3-2b1c-4078-ab0e-2e3fe44714e3\" (UID: \"4f598cb3-2b1c-4078-ab0e-2e3fe44714e3\") "
Sep 11 23:13:08.714790 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:08.714763    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4f598cb3-2b1c-4078-ab0e-2e3fe44714e3-bundle\") pod \"4f598cb3-2b1c-4078-ab0e-2e3fe44714e3\" (UID: \"4f598cb3-2b1c-4078-ab0e-2e3fe44714e3\") "
Sep 11 23:13:08.715054 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:08.714802    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vtw25\" (UniqueName: \"kubernetes.io/projected/98b51adc-75e0-4cfd-a0ea-a1aeba71bb1e-kube-api-access-vtw25\") pod \"98b51adc-75e0-4cfd-a0ea-a1aeba71bb1e\" (UID: \"98b51adc-75e0-4cfd-a0ea-a1aeba71bb1e\") "
Sep 11 23:13:08.715054 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:08.714854    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/98b51adc-75e0-4cfd-a0ea-a1aeba71bb1e-util\") pod \"98b51adc-75e0-4cfd-a0ea-a1aeba71bb1e\" (UID: \"98b51adc-75e0-4cfd-a0ea-a1aeba71bb1e\") "
Sep 11 23:13:08.715054 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:08.714910    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/98b51adc-75e0-4cfd-a0ea-a1aeba71bb1e-bundle\") pod \"98b51adc-75e0-4cfd-a0ea-a1aeba71bb1e\" (UID: \"98b51adc-75e0-4cfd-a0ea-a1aeba71bb1e\") "
Sep 11 23:13:08.715255 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:08.715221    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvvpc\" (UniqueName: \"kubernetes.io/projected/8606c1c6-9f26-4c93-a3f0-f8cb76ee96ab-kube-api-access-bvvpc\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:13:08.715255 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:08.715242    2546 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8606c1c6-9f26-4c93-a3f0-f8cb76ee96ab-bundle\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:13:08.715255 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:08.715255    2546 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8606c1c6-9f26-4c93-a3f0-f8cb76ee96ab-util\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:13:08.715617 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:08.715591    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f598cb3-2b1c-4078-ab0e-2e3fe44714e3-bundle" (OuterVolumeSpecName: "bundle") pod "4f598cb3-2b1c-4078-ab0e-2e3fe44714e3" (UID: "4f598cb3-2b1c-4078-ab0e-2e3fe44714e3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:13:08.715844 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:08.715809    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98b51adc-75e0-4cfd-a0ea-a1aeba71bb1e-bundle" (OuterVolumeSpecName: "bundle") pod "98b51adc-75e0-4cfd-a0ea-a1aeba71bb1e" (UID: "98b51adc-75e0-4cfd-a0ea-a1aeba71bb1e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:13:08.717225 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:08.717194    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f598cb3-2b1c-4078-ab0e-2e3fe44714e3-kube-api-access-drzq8" (OuterVolumeSpecName: "kube-api-access-drzq8") pod "4f598cb3-2b1c-4078-ab0e-2e3fe44714e3" (UID: "4f598cb3-2b1c-4078-ab0e-2e3fe44714e3"). InnerVolumeSpecName "kube-api-access-drzq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 23:13:08.717467 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:08.717444    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98b51adc-75e0-4cfd-a0ea-a1aeba71bb1e-kube-api-access-vtw25" (OuterVolumeSpecName: "kube-api-access-vtw25") pod "98b51adc-75e0-4cfd-a0ea-a1aeba71bb1e" (UID: "98b51adc-75e0-4cfd-a0ea-a1aeba71bb1e"). InnerVolumeSpecName "kube-api-access-vtw25". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 23:13:08.720718 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:08.720692    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98b51adc-75e0-4cfd-a0ea-a1aeba71bb1e-util" (OuterVolumeSpecName: "util") pod "98b51adc-75e0-4cfd-a0ea-a1aeba71bb1e" (UID: "98b51adc-75e0-4cfd-a0ea-a1aeba71bb1e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:13:08.721815 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:08.721795    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f598cb3-2b1c-4078-ab0e-2e3fe44714e3-util" (OuterVolumeSpecName: "util") pod "4f598cb3-2b1c-4078-ab0e-2e3fe44714e3" (UID: "4f598cb3-2b1c-4078-ab0e-2e3fe44714e3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:13:08.816383 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:08.816349    2546 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/98b51adc-75e0-4cfd-a0ea-a1aeba71bb1e-util\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:13:08.816383 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:08.816377    2546 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/98b51adc-75e0-4cfd-a0ea-a1aeba71bb1e-bundle\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:13:08.816383 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:08.816387    2546 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4f598cb3-2b1c-4078-ab0e-2e3fe44714e3-util\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:13:08.816383 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:08.816395    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drzq8\" (UniqueName: \"kubernetes.io/projected/4f598cb3-2b1c-4078-ab0e-2e3fe44714e3-kube-api-access-drzq8\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:13:08.816645 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:08.816404    2546 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4f598cb3-2b1c-4078-ab0e-2e3fe44714e3-bundle\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:13:08.816645 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:08.816413    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtw25\" (UniqueName: \"kubernetes.io/projected/98b51adc-75e0-4cfd-a0ea-a1aeba71bb1e-kube-api-access-vtw25\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:13:09.429347 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:09.429316    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cr8m2d"
Sep 11 23:13:09.429826 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:09.429354    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cr8m2d" event={"ID":"98b51adc-75e0-4cfd-a0ea-a1aeba71bb1e","Type":"ContainerDied","Data":"d2aead4737a2a4ccc9beb8f84e46fcdab5568c0e71dc4cfad2498395616b1054"}
Sep 11 23:13:09.429826 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:09.429413    2546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2aead4737a2a4ccc9beb8f84e46fcdab5568c0e71dc4cfad2498395616b1054"
Sep 11 23:13:09.431411 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:09.431392    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mhd7k"
Sep 11 23:13:09.431546 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:09.431426    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mhd7k" event={"ID":"8606c1c6-9f26-4c93-a3f0-f8cb76ee96ab","Type":"ContainerDied","Data":"ee272d0ff959171744de262e25e01a9abd1dcd11509193f2ae50fb73fc628eb4"}
Sep 11 23:13:09.431546 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:09.431458    2546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee272d0ff959171744de262e25e01a9abd1dcd11509193f2ae50fb73fc628eb4"
Sep 11 23:13:09.433514 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:09.433489    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atlzps" event={"ID":"4f598cb3-2b1c-4078-ab0e-2e3fe44714e3","Type":"ContainerDied","Data":"c91190fff07142f94ae04a4ac5599c1e58fa86e088d231f5258eb03198b66884"}
Sep 11 23:13:09.433639 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:09.433529    2546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c91190fff07142f94ae04a4ac5599c1e58fa86e088d231f5258eb03198b66884"
Sep 11 23:13:09.433639 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:09.433535    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atlzps"
Sep 11 23:13:18.698570 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:18.698533    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-h82hl"]
Sep 11 23:13:18.698951 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:18.698886    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8606c1c6-9f26-4c93-a3f0-f8cb76ee96ab" containerName="pull"
Sep 11 23:13:18.698951 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:18.698896    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="8606c1c6-9f26-4c93-a3f0-f8cb76ee96ab" containerName="pull"
Sep 11 23:13:18.698951 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:18.698904    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98b51adc-75e0-4cfd-a0ea-a1aeba71bb1e" containerName="extract"
Sep 11 23:13:18.698951 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:18.698909    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="98b51adc-75e0-4cfd-a0ea-a1aeba71bb1e" containerName="extract"
Sep 11 23:13:18.698951 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:18.698920    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3496c41f-995e-45e8-93fc-c9491d3354c2" containerName="util"
Sep 11 23:13:18.698951 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:18.698925    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="3496c41f-995e-45e8-93fc-c9491d3354c2" containerName="util"
Sep 11 23:13:18.698951 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:18.698932    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3496c41f-995e-45e8-93fc-c9491d3354c2" containerName="pull"
Sep 11 23:13:18.698951 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:18.698937    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="3496c41f-995e-45e8-93fc-c9491d3354c2" containerName="pull"
Sep 11 23:13:18.698951 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:18.698943    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f598cb3-2b1c-4078-ab0e-2e3fe44714e3" containerName="extract"
Sep 11 23:13:18.698951 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:18.698948    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f598cb3-2b1c-4078-ab0e-2e3fe44714e3" containerName="extract"
Sep 11 23:13:18.698951 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:18.698958    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f598cb3-2b1c-4078-ab0e-2e3fe44714e3" containerName="util"
Sep 11 23:13:18.699305 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:18.698963    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f598cb3-2b1c-4078-ab0e-2e3fe44714e3" containerName="util"
Sep 11 23:13:18.699305 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:18.698969    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f598cb3-2b1c-4078-ab0e-2e3fe44714e3" containerName="pull"
Sep 11 23:13:18.699305 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:18.698974    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f598cb3-2b1c-4078-ab0e-2e3fe44714e3" containerName="pull"
Sep 11 23:13:18.699305 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:18.698986    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98b51adc-75e0-4cfd-a0ea-a1aeba71bb1e" containerName="util"
Sep 11 23:13:18.699305 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:18.698991    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="98b51adc-75e0-4cfd-a0ea-a1aeba71bb1e" containerName="util"
Sep 11 23:13:18.699305 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:18.698999    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3496c41f-995e-45e8-93fc-c9491d3354c2" containerName="extract"
Sep 11 23:13:18.699305 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:18.699004    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="3496c41f-995e-45e8-93fc-c9491d3354c2" containerName="extract"
Sep 11 23:13:18.699305 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:18.699009    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98b51adc-75e0-4cfd-a0ea-a1aeba71bb1e" containerName="pull"
Sep 11 23:13:18.699305 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:18.699014    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="98b51adc-75e0-4cfd-a0ea-a1aeba71bb1e" containerName="pull"
Sep 11 23:13:18.699305 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:18.699018    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8606c1c6-9f26-4c93-a3f0-f8cb76ee96ab" containerName="extract"
Sep 11 23:13:18.699305 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:18.699022    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="8606c1c6-9f26-4c93-a3f0-f8cb76ee96ab" containerName="extract"
Sep 11 23:13:18.699305 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:18.699047    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8606c1c6-9f26-4c93-a3f0-f8cb76ee96ab" containerName="util"
Sep 11 23:13:18.699305 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:18.699052    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="8606c1c6-9f26-4c93-a3f0-f8cb76ee96ab" containerName="util"
Sep 11 23:13:18.699305 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:18.699101    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="3496c41f-995e-45e8-93fc-c9491d3354c2" containerName="extract"
Sep 11 23:13:18.699305 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:18.699109    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="98b51adc-75e0-4cfd-a0ea-a1aeba71bb1e" containerName="extract"
Sep 11 23:13:18.699305 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:18.699115    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f598cb3-2b1c-4078-ab0e-2e3fe44714e3" containerName="extract"
Sep 11 23:13:18.699305 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:18.699123    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="8606c1c6-9f26-4c93-a3f0-f8cb76ee96ab" containerName="extract"
Sep 11 23:13:18.702264 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:18.702246    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-h82hl"
Sep 11 23:13:18.705523 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:18.705499    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 11 23:13:18.705652 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:18.705637    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 11 23:13:18.705755 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:18.705738    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-q84zq\""
Sep 11 23:13:18.718767 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:18.718737    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-h82hl"]
Sep 11 23:13:18.801249 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:18.801210    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/620ad12b-d0fb-43aa-aa8b-e2f875713526-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-h82hl\" (UID: \"620ad12b-d0fb-43aa-aa8b-e2f875713526\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-h82hl"
Sep 11 23:13:18.801428 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:18.801287    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lp78\" (UniqueName: \"kubernetes.io/projected/620ad12b-d0fb-43aa-aa8b-e2f875713526-kube-api-access-9lp78\") pod \"kuadrant-operator-controller-manager-76f8f686bf-h82hl\" (UID: \"620ad12b-d0fb-43aa-aa8b-e2f875713526\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-h82hl"
Sep 11 23:13:18.902805 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:18.902768    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lp78\" (UniqueName: \"kubernetes.io/projected/620ad12b-d0fb-43aa-aa8b-e2f875713526-kube-api-access-9lp78\") pod \"kuadrant-operator-controller-manager-76f8f686bf-h82hl\" (UID: \"620ad12b-d0fb-43aa-aa8b-e2f875713526\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-h82hl"
Sep 11 23:13:18.903013 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:18.902839    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/620ad12b-d0fb-43aa-aa8b-e2f875713526-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-h82hl\" (UID: \"620ad12b-d0fb-43aa-aa8b-e2f875713526\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-h82hl"
Sep 11 23:13:18.903292 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:18.903272    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/620ad12b-d0fb-43aa-aa8b-e2f875713526-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-h82hl\" (UID: \"620ad12b-d0fb-43aa-aa8b-e2f875713526\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-h82hl"
Sep 11 23:13:18.921135 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:18.921106    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lp78\" (UniqueName: \"kubernetes.io/projected/620ad12b-d0fb-43aa-aa8b-e2f875713526-kube-api-access-9lp78\") pod \"kuadrant-operator-controller-manager-76f8f686bf-h82hl\" (UID: \"620ad12b-d0fb-43aa-aa8b-e2f875713526\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-h82hl"
Sep 11 23:13:19.013104 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:19.013026    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-h82hl"
Sep 11 23:13:19.294672 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:19.294616    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-h82hl"]
Sep 11 23:13:19.296914 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:13:19.296882    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod620ad12b_d0fb_43aa_aa8b_e2f875713526.slice/crio-256bf8c030d5f8d2744a0c859d6931cef9a2b475facfa080c514e245d8d6c1bd WatchSource:0}: Error finding container 256bf8c030d5f8d2744a0c859d6931cef9a2b475facfa080c514e245d8d6c1bd: Status 404 returned error can't find the container with id 256bf8c030d5f8d2744a0c859d6931cef9a2b475facfa080c514e245d8d6c1bd
Sep 11 23:13:19.488923 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:19.488886    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-h82hl" event={"ID":"620ad12b-d0fb-43aa-aa8b-e2f875713526","Type":"ContainerStarted","Data":"256bf8c030d5f8d2744a0c859d6931cef9a2b475facfa080c514e245d8d6c1bd"}
Sep 11 23:13:23.426972 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:23.426930    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-dbvlh"]
Sep 11 23:13:23.430402 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:23.430377    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dbvlh"
Sep 11 23:13:23.434530 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:23.434509    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-fvnkm\""
Sep 11 23:13:23.447814 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:23.447753    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-dbvlh"]
Sep 11 23:13:23.518626 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:23.518520    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-h82hl" event={"ID":"620ad12b-d0fb-43aa-aa8b-e2f875713526","Type":"ContainerStarted","Data":"afb61ae93a0fbbc6a22064d7258cd3867f6082023dc48a9bf82bd4975141f427"}
Sep 11 23:13:23.518626 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:23.518597    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-h82hl"
Sep 11 23:13:23.546701 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:23.546666    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52wjd\" (UniqueName: \"kubernetes.io/projected/2da1844c-9399-453b-aa0c-56505f1b495a-kube-api-access-52wjd\") pod \"limitador-operator-controller-manager-74495c69f8-dbvlh\" (UID: \"2da1844c-9399-453b-aa0c-56505f1b495a\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dbvlh"
Sep 11 23:13:23.553705 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:23.553657    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-h82hl" podStartSLOduration=1.618640791 podStartE2EDuration="5.553639677s" totalImagesPullingTime="3.934998886s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 23:13:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:13:23.551508275 +0000 UTC m=+727.584637494" watchObservedRunningTime="2026-09-11 23:13:23.553639677 +0000 UTC m=+727.586768894"
Sep 11 23:13:23.647985 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:23.647947    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52wjd\" (UniqueName: \"kubernetes.io/projected/2da1844c-9399-453b-aa0c-56505f1b495a-kube-api-access-52wjd\") pod \"limitador-operator-controller-manager-74495c69f8-dbvlh\" (UID: \"2da1844c-9399-453b-aa0c-56505f1b495a\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dbvlh"
Sep 11 23:13:23.663411 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:23.663377    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52wjd\" (UniqueName: \"kubernetes.io/projected/2da1844c-9399-453b-aa0c-56505f1b495a-kube-api-access-52wjd\") pod \"limitador-operator-controller-manager-74495c69f8-dbvlh\" (UID: \"2da1844c-9399-453b-aa0c-56505f1b495a\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dbvlh"
Sep 11 23:13:23.740924 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:23.740884    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dbvlh"
Sep 11 23:13:24.025383 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:13:24.025343    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2da1844c_9399_453b_aa0c_56505f1b495a.slice/crio-74e3eb00a1a16220d29922dab80b2b74731e8640ef6d5333bbe733e146f4943e WatchSource:0}: Error finding container 74e3eb00a1a16220d29922dab80b2b74731e8640ef6d5333bbe733e146f4943e: Status 404 returned error can't find the container with id 74e3eb00a1a16220d29922dab80b2b74731e8640ef6d5333bbe733e146f4943e
Sep 11 23:13:24.025383 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:24.025371    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-dbvlh"]
Sep 11 23:13:24.526091 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:24.526048    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dbvlh" event={"ID":"2da1844c-9399-453b-aa0c-56505f1b495a","Type":"ContainerStarted","Data":"74e3eb00a1a16220d29922dab80b2b74731e8640ef6d5333bbe733e146f4943e"}
Sep 11 23:13:26.539464 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:26.539423    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dbvlh" event={"ID":"2da1844c-9399-453b-aa0c-56505f1b495a","Type":"ContainerStarted","Data":"078a3fea554ebf792d0cb7004d0a2ac51b2934a56c6b404d1497c5a560b2d6e3"}
Sep 11 23:13:26.539830 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:26.539576    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dbvlh"
Sep 11 23:13:26.568320 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:26.568264    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dbvlh" podStartSLOduration=1.492887214 podStartE2EDuration="3.568249601s" totalImagesPullingTime="2.075362387s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 23:13:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:13:26.56572434 +0000 UTC m=+730.598853558" watchObservedRunningTime="2026-09-11 23:13:26.568249601 +0000 UTC m=+730.601378817"
Sep 11 23:13:34.529235 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:34.529200    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-h82hl"
Sep 11 23:13:36.583528 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:36.583489    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-h82hl"]
Sep 11 23:13:36.584004 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:36.583705    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-h82hl" podUID="620ad12b-d0fb-43aa-aa8b-e2f875713526" containerName="manager" containerID="cri-o://afb61ae93a0fbbc6a22064d7258cd3867f6082023dc48a9bf82bd4975141f427" gracePeriod=2
Sep 11 23:13:36.595762 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:13:36.595699    2546 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"620ad12b-d0fb-43aa-aa8b-e2f875713526\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-76f8f686bf-h82hl\": pods \"kuadrant-operator-controller-manager-76f8f686bf-h82hl\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-h82hl"
Sep 11 23:13:36.597275 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:36.597243    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-h82hl"]
Sep 11 23:13:36.606860 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:36.606830    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nc6jh"]
Sep 11 23:13:36.607387 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:36.607366    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="620ad12b-d0fb-43aa-aa8b-e2f875713526" containerName="manager"
Sep 11 23:13:36.607774 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:36.607523    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="620ad12b-d0fb-43aa-aa8b-e2f875713526" containerName="manager"
Sep 11 23:13:36.607774 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:36.607678    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="620ad12b-d0fb-43aa-aa8b-e2f875713526" containerName="manager"
Sep 11 23:13:36.611031 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:36.611009    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nc6jh"
Sep 11 23:13:36.625647 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:36.625617    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nc6jh"]
Sep 11 23:13:36.638209 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:36.638176    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-dbvlh"]
Sep 11 23:13:36.638447 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:36.638421    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dbvlh" podUID="2da1844c-9399-453b-aa0c-56505f1b495a" containerName="manager" containerID="cri-o://078a3fea554ebf792d0cb7004d0a2ac51b2934a56c6b404d1497c5a560b2d6e3" gracePeriod=2
Sep 11 23:13:36.640433 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:36.640406    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dbvlh"
Sep 11 23:13:36.650043 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:36.648280    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-dbvlh"]
Sep 11 23:13:36.650043 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:13:36.647575    2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-h82hl\" is forbidden: User \"system:node:ip-10-0-128-161.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-161.ec2.internal' and this object" podUID="620ad12b-d0fb-43aa-aa8b-e2f875713526" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-h82hl"
Sep 11 23:13:36.651185 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:13:36.651134    2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-dbvlh\" is forbidden: User \"system:node:ip-10-0-128-161.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-161.ec2.internal' and this object" podUID="2da1844c-9399-453b-aa0c-56505f1b495a" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dbvlh"
Sep 11 23:13:36.655330 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:13:36.655287    2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-h82hl\" is forbidden: User \"system:node:ip-10-0-128-161.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-161.ec2.internal' and this object" podUID="620ad12b-d0fb-43aa-aa8b-e2f875713526" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-h82hl"
Sep 11 23:13:36.670775 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:36.670741    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-lf82j"]
Sep 11 23:13:36.671367 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:36.671340    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2da1844c-9399-453b-aa0c-56505f1b495a" containerName="manager"
Sep 11 23:13:36.671367 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:36.671364    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="2da1844c-9399-453b-aa0c-56505f1b495a" containerName="manager"
Sep 11 23:13:36.671516 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:36.671491    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="2da1844c-9399-453b-aa0c-56505f1b495a" containerName="manager"
Sep 11 23:13:36.675050 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:36.675019    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-lf82j"
Sep 11 23:13:36.679477 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:13:36.679448    2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-h82hl\" is forbidden: User \"system:node:ip-10-0-128-161.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-161.ec2.internal' and this object" podUID="620ad12b-d0fb-43aa-aa8b-e2f875713526" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-h82hl"
Sep 11 23:13:36.681897 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:13:36.681874    2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-dbvlh\" is forbidden: User \"system:node:ip-10-0-128-161.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-161.ec2.internal' and this object" podUID="2da1844c-9399-453b-aa0c-56505f1b495a" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dbvlh"
Sep 11 23:13:36.692700 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:36.692671    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-lf82j"]
Sep 11 23:13:36.768485 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:36.768430    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwxq7\" (UniqueName: \"kubernetes.io/projected/5644f03d-9297-4d55-86fc-9db18e38cdae-kube-api-access-dwxq7\") pod \"limitador-operator-controller-manager-74495c69f8-lf82j\" (UID: \"5644f03d-9297-4d55-86fc-9db18e38cdae\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-lf82j"
Sep 11 23:13:36.768648 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:36.768611    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/30576cfe-0e2a-4986-ba12-eab2bdf54160-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-nc6jh\" (UID: \"30576cfe-0e2a-4986-ba12-eab2bdf54160\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nc6jh"
Sep 11 23:13:36.768708 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:36.768647    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbwc4\" (UniqueName: \"kubernetes.io/projected/30576cfe-0e2a-4986-ba12-eab2bdf54160-kube-api-access-cbwc4\") pod \"kuadrant-operator-controller-manager-76f8f686bf-nc6jh\" (UID: \"30576cfe-0e2a-4986-ba12-eab2bdf54160\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nc6jh"
Sep 11 23:13:36.869133 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:36.869103    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/30576cfe-0e2a-4986-ba12-eab2bdf54160-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-nc6jh\" (UID: \"30576cfe-0e2a-4986-ba12-eab2bdf54160\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nc6jh"
Sep 11 23:13:36.869133 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:36.869140    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbwc4\" (UniqueName: \"kubernetes.io/projected/30576cfe-0e2a-4986-ba12-eab2bdf54160-kube-api-access-cbwc4\") pod \"kuadrant-operator-controller-manager-76f8f686bf-nc6jh\" (UID: \"30576cfe-0e2a-4986-ba12-eab2bdf54160\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nc6jh"
Sep 11 23:13:36.869372 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:36.869206    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwxq7\" (UniqueName: \"kubernetes.io/projected/5644f03d-9297-4d55-86fc-9db18e38cdae-kube-api-access-dwxq7\") pod \"limitador-operator-controller-manager-74495c69f8-lf82j\" (UID: \"5644f03d-9297-4d55-86fc-9db18e38cdae\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-lf82j"
Sep 11 23:13:36.869602 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:36.869540    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/30576cfe-0e2a-4986-ba12-eab2bdf54160-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-nc6jh\" (UID: \"30576cfe-0e2a-4986-ba12-eab2bdf54160\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nc6jh"
Sep 11 23:13:36.879939 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:36.879904    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwxq7\" (UniqueName: \"kubernetes.io/projected/5644f03d-9297-4d55-86fc-9db18e38cdae-kube-api-access-dwxq7\") pod \"limitador-operator-controller-manager-74495c69f8-lf82j\" (UID: \"5644f03d-9297-4d55-86fc-9db18e38cdae\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-lf82j"
Sep 11 23:13:36.880036 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:36.879904    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbwc4\" (UniqueName: \"kubernetes.io/projected/30576cfe-0e2a-4986-ba12-eab2bdf54160-kube-api-access-cbwc4\") pod \"kuadrant-operator-controller-manager-76f8f686bf-nc6jh\" (UID: \"30576cfe-0e2a-4986-ba12-eab2bdf54160\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nc6jh"
Sep 11 23:13:36.888276 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:36.888251    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-h82hl"
Sep 11 23:13:36.890411 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:36.890386    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dbvlh"
Sep 11 23:13:36.891237 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:13:36.891214    2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-h82hl\" is forbidden: User \"system:node:ip-10-0-128-161.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-161.ec2.internal' and this object" podUID="620ad12b-d0fb-43aa-aa8b-e2f875713526" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-h82hl"
Sep 11 23:13:36.893558 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:13:36.893537    2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-dbvlh\" is forbidden: User \"system:node:ip-10-0-128-161.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-161.ec2.internal' and this object" podUID="2da1844c-9399-453b-aa0c-56505f1b495a" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dbvlh"
Sep 11 23:13:36.895865 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:13:36.895846    2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-h82hl\" is forbidden: User \"system:node:ip-10-0-128-161.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-161.ec2.internal' and this object" podUID="620ad12b-d0fb-43aa-aa8b-e2f875713526" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-h82hl"
Sep 11 23:13:36.898164 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:13:36.898136    2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-dbvlh\" is forbidden: User \"system:node:ip-10-0-128-161.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-161.ec2.internal' and this object" podUID="2da1844c-9399-453b-aa0c-56505f1b495a" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dbvlh"
Sep 11 23:13:36.970396 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:36.970360    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9lp78\" (UniqueName: \"kubernetes.io/projected/620ad12b-d0fb-43aa-aa8b-e2f875713526-kube-api-access-9lp78\") pod \"620ad12b-d0fb-43aa-aa8b-e2f875713526\" (UID: \"620ad12b-d0fb-43aa-aa8b-e2f875713526\") "
Sep 11 23:13:36.970562 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:36.970416    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/620ad12b-d0fb-43aa-aa8b-e2f875713526-extensions-socket-volume\") pod \"620ad12b-d0fb-43aa-aa8b-e2f875713526\" (UID: \"620ad12b-d0fb-43aa-aa8b-e2f875713526\") "
Sep 11 23:13:36.970927 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:36.970897    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/620ad12b-d0fb-43aa-aa8b-e2f875713526-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "620ad12b-d0fb-43aa-aa8b-e2f875713526" (UID: "620ad12b-d0fb-43aa-aa8b-e2f875713526"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:13:36.972677 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:36.972653    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/620ad12b-d0fb-43aa-aa8b-e2f875713526-kube-api-access-9lp78" (OuterVolumeSpecName: "kube-api-access-9lp78") pod "620ad12b-d0fb-43aa-aa8b-e2f875713526" (UID: "620ad12b-d0fb-43aa-aa8b-e2f875713526"). InnerVolumeSpecName "kube-api-access-9lp78". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 23:13:37.011073 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:37.011033    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nc6jh"
Sep 11 23:13:37.016178 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:37.016141    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-lf82j"
Sep 11 23:13:37.076040 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:37.071108    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-52wjd\" (UniqueName: \"kubernetes.io/projected/2da1844c-9399-453b-aa0c-56505f1b495a-kube-api-access-52wjd\") pod \"2da1844c-9399-453b-aa0c-56505f1b495a\" (UID: \"2da1844c-9399-453b-aa0c-56505f1b495a\") "
Sep 11 23:13:37.076040 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:37.071437    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lp78\" (UniqueName: \"kubernetes.io/projected/620ad12b-d0fb-43aa-aa8b-e2f875713526-kube-api-access-9lp78\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:13:37.076040 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:37.071464    2546 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/620ad12b-d0fb-43aa-aa8b-e2f875713526-extensions-socket-volume\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:13:37.076040 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:37.073343    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2da1844c-9399-453b-aa0c-56505f1b495a-kube-api-access-52wjd" (OuterVolumeSpecName: "kube-api-access-52wjd") pod "2da1844c-9399-453b-aa0c-56505f1b495a" (UID: "2da1844c-9399-453b-aa0c-56505f1b495a"). InnerVolumeSpecName "kube-api-access-52wjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 23:13:37.171919 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:37.171881    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52wjd\" (UniqueName: \"kubernetes.io/projected/2da1844c-9399-453b-aa0c-56505f1b495a-kube-api-access-52wjd\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:13:37.293527 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:37.293493    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nc6jh"]
Sep 11 23:13:37.297267 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:13:37.297238    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30576cfe_0e2a_4986_ba12_eab2bdf54160.slice/crio-1ba5176f6f9ca29311429899031e8219b4bdfe326b93fa5ea3e8c734eebd6541 WatchSource:0}: Error finding container 1ba5176f6f9ca29311429899031e8219b4bdfe326b93fa5ea3e8c734eebd6541: Status 404 returned error can't find the container with id 1ba5176f6f9ca29311429899031e8219b4bdfe326b93fa5ea3e8c734eebd6541
Sep 11 23:13:37.297393 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:37.297368    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-lf82j"]
Sep 11 23:13:37.298033 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:13:37.298014    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5644f03d_9297_4d55_86fc_9db18e38cdae.slice/crio-66ac5b44119eaae894fa8148f8d43979c405deb33b543a7c2123c255f79de95e WatchSource:0}: Error finding container 66ac5b44119eaae894fa8148f8d43979c405deb33b543a7c2123c255f79de95e: Status 404 returned error can't find the container with id 66ac5b44119eaae894fa8148f8d43979c405deb33b543a7c2123c255f79de95e
Sep 11 23:13:37.608375 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:37.608277    2546 generic.go:356] "Generic (PLEG): container finished" podID="2da1844c-9399-453b-aa0c-56505f1b495a" containerID="078a3fea554ebf792d0cb7004d0a2ac51b2934a56c6b404d1497c5a560b2d6e3" exitCode=0
Sep 11 23:13:37.608375 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:37.608345    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dbvlh"
Sep 11 23:13:37.609172 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:37.608385    2546 scope.go:117] "RemoveContainer" containerID="078a3fea554ebf792d0cb7004d0a2ac51b2934a56c6b404d1497c5a560b2d6e3"
Sep 11 23:13:37.610600 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:37.610447    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-lf82j" event={"ID":"5644f03d-9297-4d55-86fc-9db18e38cdae","Type":"ContainerStarted","Data":"5bd5c2331364225c0ab31bb784d8a832785062771bad8413ae6e5ddbb5f28bdc"}
Sep 11 23:13:37.610600 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:37.610495    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-lf82j" event={"ID":"5644f03d-9297-4d55-86fc-9db18e38cdae","Type":"ContainerStarted","Data":"66ac5b44119eaae894fa8148f8d43979c405deb33b543a7c2123c255f79de95e"}
Sep 11 23:13:37.610600 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:37.610523    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-lf82j"
Sep 11 23:13:37.611109 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:13:37.611084    2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-h82hl\" is forbidden: User \"system:node:ip-10-0-128-161.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-161.ec2.internal' and this object" podUID="620ad12b-d0fb-43aa-aa8b-e2f875713526" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-h82hl"
Sep 11 23:13:37.612297 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:37.612272    2546 generic.go:356] "Generic (PLEG): container finished" podID="620ad12b-d0fb-43aa-aa8b-e2f875713526" containerID="afb61ae93a0fbbc6a22064d7258cd3867f6082023dc48a9bf82bd4975141f427" exitCode=0
Sep 11 23:13:37.612397 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:37.612315    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-h82hl"
Sep 11 23:13:37.613533 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:13:37.613506    2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-dbvlh\" is forbidden: User \"system:node:ip-10-0-128-161.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-161.ec2.internal' and this object" podUID="2da1844c-9399-453b-aa0c-56505f1b495a" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dbvlh"
Sep 11 23:13:37.614658 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:37.614628    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nc6jh" event={"ID":"30576cfe-0e2a-4986-ba12-eab2bdf54160","Type":"ContainerStarted","Data":"47b9ebfa97cc02bf7f8aedf165978b8f2eaeaa1d2c22d40ab35a7a8a507b31cb"}
Sep 11 23:13:37.614801 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:37.614685    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nc6jh" event={"ID":"30576cfe-0e2a-4986-ba12-eab2bdf54160","Type":"ContainerStarted","Data":"1ba5176f6f9ca29311429899031e8219b4bdfe326b93fa5ea3e8c734eebd6541"}
Sep 11 23:13:37.614941 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:37.614923    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nc6jh"
Sep 11 23:13:37.621801 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:37.621776    2546 scope.go:117] "RemoveContainer" containerID="078a3fea554ebf792d0cb7004d0a2ac51b2934a56c6b404d1497c5a560b2d6e3"
Sep 11 23:13:37.622109 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:13:37.622088    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"078a3fea554ebf792d0cb7004d0a2ac51b2934a56c6b404d1497c5a560b2d6e3\": container with ID starting with 078a3fea554ebf792d0cb7004d0a2ac51b2934a56c6b404d1497c5a560b2d6e3 not found: ID does not exist" containerID="078a3fea554ebf792d0cb7004d0a2ac51b2934a56c6b404d1497c5a560b2d6e3"
Sep 11 23:13:37.622252 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:37.622123    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"078a3fea554ebf792d0cb7004d0a2ac51b2934a56c6b404d1497c5a560b2d6e3"} err="failed to get container status \"078a3fea554ebf792d0cb7004d0a2ac51b2934a56c6b404d1497c5a560b2d6e3\": rpc error: code = NotFound desc = could not find container \"078a3fea554ebf792d0cb7004d0a2ac51b2934a56c6b404d1497c5a560b2d6e3\": container with ID starting with 078a3fea554ebf792d0cb7004d0a2ac51b2934a56c6b404d1497c5a560b2d6e3 not found: ID does not exist"
Sep 11 23:13:37.622252 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:37.622174    2546 scope.go:117] "RemoveContainer" containerID="afb61ae93a0fbbc6a22064d7258cd3867f6082023dc48a9bf82bd4975141f427"
Sep 11 23:13:37.631927 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:37.631907    2546 scope.go:117] "RemoveContainer" containerID="afb61ae93a0fbbc6a22064d7258cd3867f6082023dc48a9bf82bd4975141f427"
Sep 11 23:13:37.632417 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:13:37.632392    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afb61ae93a0fbbc6a22064d7258cd3867f6082023dc48a9bf82bd4975141f427\": container with ID starting with afb61ae93a0fbbc6a22064d7258cd3867f6082023dc48a9bf82bd4975141f427 not found: ID does not exist" containerID="afb61ae93a0fbbc6a22064d7258cd3867f6082023dc48a9bf82bd4975141f427"
Sep 11 23:13:37.632496 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:37.632423    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afb61ae93a0fbbc6a22064d7258cd3867f6082023dc48a9bf82bd4975141f427"} err="failed to get container status \"afb61ae93a0fbbc6a22064d7258cd3867f6082023dc48a9bf82bd4975141f427\": rpc error: code = NotFound desc = could not find container \"afb61ae93a0fbbc6a22064d7258cd3867f6082023dc48a9bf82bd4975141f427\": container with ID starting with afb61ae93a0fbbc6a22064d7258cd3867f6082023dc48a9bf82bd4975141f427 not found: ID does not exist"
Sep 11 23:13:37.633642 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:13:37.633602    2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-h82hl\" is forbidden: User \"system:node:ip-10-0-128-161.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-161.ec2.internal' and this object" podUID="620ad12b-d0fb-43aa-aa8b-e2f875713526" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-h82hl"
Sep 11 23:13:37.633642 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:37.633607    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-lf82j" podStartSLOduration=1.633591362 podStartE2EDuration="1.633591362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 23:13:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:13:37.63123158 +0000 UTC m=+741.664360802" watchObservedRunningTime="2026-09-11 23:13:37.633591362 +0000 UTC m=+741.666720581"
Sep 11 23:13:37.635959 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:13:37.635936    2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-dbvlh\" is forbidden: User \"system:node:ip-10-0-128-161.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-161.ec2.internal' and this object" podUID="2da1844c-9399-453b-aa0c-56505f1b495a" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dbvlh"
Sep 11 23:13:37.638218 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:13:37.638192    2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-h82hl\" is forbidden: User \"system:node:ip-10-0-128-161.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-161.ec2.internal' and this object" podUID="620ad12b-d0fb-43aa-aa8b-e2f875713526" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-h82hl"
Sep 11 23:13:37.640356 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:13:37.640329    2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-dbvlh\" is forbidden: User \"system:node:ip-10-0-128-161.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-161.ec2.internal' and this object" podUID="2da1844c-9399-453b-aa0c-56505f1b495a" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dbvlh"
Sep 11 23:13:37.666999 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:37.663169    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nc6jh" podStartSLOduration=1.663123579 podStartE2EDuration="1.663123579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 23:13:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:13:37.659907483 +0000 UTC m=+741.693036699" watchObservedRunningTime="2026-09-11 23:13:37.663123579 +0000 UTC m=+741.696252796"
Sep 11 23:13:38.494342 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:38.494305    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2da1844c-9399-453b-aa0c-56505f1b495a" path="/var/lib/kubelet/pods/2da1844c-9399-453b-aa0c-56505f1b495a/volumes"
Sep 11 23:13:38.494719 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:38.494704    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="620ad12b-d0fb-43aa-aa8b-e2f875713526" path="/var/lib/kubelet/pods/620ad12b-d0fb-43aa-aa8b-e2f875713526/volumes"
Sep 11 23:13:48.625230 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:48.625193    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-lf82j"
Sep 11 23:13:48.625629 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:48.625391    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nc6jh"
Sep 11 23:13:52.599224 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:52.599177    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nc6jh"]
Sep 11 23:13:52.599726 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:52.599474    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nc6jh" podUID="30576cfe-0e2a-4986-ba12-eab2bdf54160" containerName="manager" containerID="cri-o://47b9ebfa97cc02bf7f8aedf165978b8f2eaeaa1d2c22d40ab35a7a8a507b31cb" gracePeriod=10
Sep 11 23:13:52.865202 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:52.865046    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nc6jh"
Sep 11 23:13:52.868086 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:52.868055    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c56f6"]
Sep 11 23:13:52.868461 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:52.868446    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30576cfe-0e2a-4986-ba12-eab2bdf54160" containerName="manager"
Sep 11 23:13:52.868506 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:52.868464    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="30576cfe-0e2a-4986-ba12-eab2bdf54160" containerName="manager"
Sep 11 23:13:52.868549 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:52.868539    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="30576cfe-0e2a-4986-ba12-eab2bdf54160" containerName="manager"
Sep 11 23:13:52.873483 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:52.873451    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c56f6"
Sep 11 23:13:52.887889 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:52.887848    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c56f6"]
Sep 11 23:13:52.906633 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:52.906591    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpdp8\" (UniqueName: \"kubernetes.io/projected/a5a83b12-239d-4342-9f10-dc236f6fbb2b-kube-api-access-dpdp8\") pod \"kuadrant-operator-controller-manager-5769f99d9f-c56f6\" (UID: \"a5a83b12-239d-4342-9f10-dc236f6fbb2b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c56f6"
Sep 11 23:13:52.906816 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:52.906689    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a5a83b12-239d-4342-9f10-dc236f6fbb2b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-c56f6\" (UID: \"a5a83b12-239d-4342-9f10-dc236f6fbb2b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c56f6"
Sep 11 23:13:53.007070 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:53.007036    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/30576cfe-0e2a-4986-ba12-eab2bdf54160-extensions-socket-volume\") pod \"30576cfe-0e2a-4986-ba12-eab2bdf54160\" (UID: \"30576cfe-0e2a-4986-ba12-eab2bdf54160\") "
Sep 11 23:13:53.007302 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:53.007108    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cbwc4\" (UniqueName: \"kubernetes.io/projected/30576cfe-0e2a-4986-ba12-eab2bdf54160-kube-api-access-cbwc4\") pod \"30576cfe-0e2a-4986-ba12-eab2bdf54160\" (UID: \"30576cfe-0e2a-4986-ba12-eab2bdf54160\") "
Sep 11 23:13:53.007302 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:53.007262    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a5a83b12-239d-4342-9f10-dc236f6fbb2b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-c56f6\" (UID: \"a5a83b12-239d-4342-9f10-dc236f6fbb2b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c56f6"
Sep 11 23:13:53.007432 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:53.007313    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpdp8\" (UniqueName: \"kubernetes.io/projected/a5a83b12-239d-4342-9f10-dc236f6fbb2b-kube-api-access-dpdp8\") pod \"kuadrant-operator-controller-manager-5769f99d9f-c56f6\" (UID: \"a5a83b12-239d-4342-9f10-dc236f6fbb2b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c56f6"
Sep 11 23:13:53.007552 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:53.007526    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30576cfe-0e2a-4986-ba12-eab2bdf54160-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "30576cfe-0e2a-4986-ba12-eab2bdf54160" (UID: "30576cfe-0e2a-4986-ba12-eab2bdf54160"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:13:53.007691 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:53.007667    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a5a83b12-239d-4342-9f10-dc236f6fbb2b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-c56f6\" (UID: \"a5a83b12-239d-4342-9f10-dc236f6fbb2b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c56f6"
Sep 11 23:13:53.009615 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:53.009591    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30576cfe-0e2a-4986-ba12-eab2bdf54160-kube-api-access-cbwc4" (OuterVolumeSpecName: "kube-api-access-cbwc4") pod "30576cfe-0e2a-4986-ba12-eab2bdf54160" (UID: "30576cfe-0e2a-4986-ba12-eab2bdf54160"). InnerVolumeSpecName "kube-api-access-cbwc4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 23:13:53.017577 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:53.017551    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpdp8\" (UniqueName: \"kubernetes.io/projected/a5a83b12-239d-4342-9f10-dc236f6fbb2b-kube-api-access-dpdp8\") pod \"kuadrant-operator-controller-manager-5769f99d9f-c56f6\" (UID: \"a5a83b12-239d-4342-9f10-dc236f6fbb2b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c56f6"
Sep 11 23:13:53.108114 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:53.108080    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbwc4\" (UniqueName: \"kubernetes.io/projected/30576cfe-0e2a-4986-ba12-eab2bdf54160-kube-api-access-cbwc4\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:13:53.108114 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:53.108108    2546 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/30576cfe-0e2a-4986-ba12-eab2bdf54160-extensions-socket-volume\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:13:53.185307 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:53.185260    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c56f6"
Sep 11 23:13:53.473767 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:53.473732    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c56f6"]
Sep 11 23:13:53.475297 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:13:53.475269    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5a83b12_239d_4342_9f10_dc236f6fbb2b.slice/crio-01559d6b88d4ad2dccf16c1332ba4c425688c85ba2fea216ade7cdf0810135e7 WatchSource:0}: Error finding container 01559d6b88d4ad2dccf16c1332ba4c425688c85ba2fea216ade7cdf0810135e7: Status 404 returned error can't find the container with id 01559d6b88d4ad2dccf16c1332ba4c425688c85ba2fea216ade7cdf0810135e7
Sep 11 23:13:53.718078 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:53.718025    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c56f6" event={"ID":"a5a83b12-239d-4342-9f10-dc236f6fbb2b","Type":"ContainerStarted","Data":"489dfc2189f4ab2947303b73fff8bbe6dd1bdc12afe712592429797c4478aa27"}
Sep 11 23:13:53.718629 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:53.718096    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c56f6" event={"ID":"a5a83b12-239d-4342-9f10-dc236f6fbb2b","Type":"ContainerStarted","Data":"01559d6b88d4ad2dccf16c1332ba4c425688c85ba2fea216ade7cdf0810135e7"}
Sep 11 23:13:53.718629 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:53.718615    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c56f6"
Sep 11 23:13:53.720025 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:53.719994    2546 generic.go:356] "Generic (PLEG): container finished" podID="30576cfe-0e2a-4986-ba12-eab2bdf54160" containerID="47b9ebfa97cc02bf7f8aedf165978b8f2eaeaa1d2c22d40ab35a7a8a507b31cb" exitCode=0
Sep 11 23:13:53.720139 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:53.720052    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nc6jh" event={"ID":"30576cfe-0e2a-4986-ba12-eab2bdf54160","Type":"ContainerDied","Data":"47b9ebfa97cc02bf7f8aedf165978b8f2eaeaa1d2c22d40ab35a7a8a507b31cb"}
Sep 11 23:13:53.720139 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:53.720045    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nc6jh"
Sep 11 23:13:53.720139 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:53.720103    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nc6jh" event={"ID":"30576cfe-0e2a-4986-ba12-eab2bdf54160","Type":"ContainerDied","Data":"1ba5176f6f9ca29311429899031e8219b4bdfe326b93fa5ea3e8c734eebd6541"}
Sep 11 23:13:53.720139 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:53.720133    2546 scope.go:117] "RemoveContainer" containerID="47b9ebfa97cc02bf7f8aedf165978b8f2eaeaa1d2c22d40ab35a7a8a507b31cb"
Sep 11 23:13:53.730735 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:53.730703    2546 scope.go:117] "RemoveContainer" containerID="47b9ebfa97cc02bf7f8aedf165978b8f2eaeaa1d2c22d40ab35a7a8a507b31cb"
Sep 11 23:13:53.731794 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:13:53.731704    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47b9ebfa97cc02bf7f8aedf165978b8f2eaeaa1d2c22d40ab35a7a8a507b31cb\": container with ID starting with 47b9ebfa97cc02bf7f8aedf165978b8f2eaeaa1d2c22d40ab35a7a8a507b31cb not found: ID does not exist" containerID="47b9ebfa97cc02bf7f8aedf165978b8f2eaeaa1d2c22d40ab35a7a8a507b31cb"
Sep 11 23:13:53.731924 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:53.731810    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47b9ebfa97cc02bf7f8aedf165978b8f2eaeaa1d2c22d40ab35a7a8a507b31cb"} err="failed to get container status \"47b9ebfa97cc02bf7f8aedf165978b8f2eaeaa1d2c22d40ab35a7a8a507b31cb\": rpc error: code = NotFound desc = could not find container \"47b9ebfa97cc02bf7f8aedf165978b8f2eaeaa1d2c22d40ab35a7a8a507b31cb\": container with ID starting with 47b9ebfa97cc02bf7f8aedf165978b8f2eaeaa1d2c22d40ab35a7a8a507b31cb not found: ID does not exist"
Sep 11 23:13:53.756329 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:53.756264    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c56f6" podStartSLOduration=1.756243422 podStartE2EDuration="1.756243422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 23:13:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:13:53.753758493 +0000 UTC m=+757.786887711" watchObservedRunningTime="2026-09-11 23:13:53.756243422 +0000 UTC m=+757.789372640"
Sep 11 23:13:53.779084 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:53.779028    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nc6jh"]
Sep 11 23:13:53.781877 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:53.781837    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nc6jh"]
Sep 11 23:13:54.495033 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:13:54.494992    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30576cfe-0e2a-4986-ba12-eab2bdf54160" path="/var/lib/kubelet/pods/30576cfe-0e2a-4986-ba12-eab2bdf54160/volumes"
Sep 11 23:14:05.736896 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:14:05.736864    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c56f6"
Sep 11 23:14:08.910614 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:14:08.910577    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vdztn"]
Sep 11 23:14:08.917290 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:14:08.917262    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vdztn"
Sep 11 23:14:08.920369 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:14:08.920341    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-m6wch\""
Sep 11 23:14:08.925797 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:14:08.925344    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vdztn"]
Sep 11 23:14:08.943352 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:14:08.943305    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a09df99-b62e-4670-9852-a452fa1c6cba-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vdztn\" (UID: \"4a09df99-b62e-4670-9852-a452fa1c6cba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vdztn"
Sep 11 23:14:08.943570 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:14:08.943371    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a09df99-b62e-4670-9852-a452fa1c6cba-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vdztn\" (UID: \"4a09df99-b62e-4670-9852-a452fa1c6cba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vdztn"
Sep 11 23:14:08.943570 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:14:08.943404    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a09df99-b62e-4670-9852-a452fa1c6cba-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vdztn\" (UID: \"4a09df99-b62e-4670-9852-a452fa1c6cba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vdztn"
Sep 11 23:14:08.943570 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:14:08.943445    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a09df99-b62e-4670-9852-a452fa1c6cba-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vdztn\" (UID: \"4a09df99-b62e-4670-9852-a452fa1c6cba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vdztn"
Sep 11 23:14:08.943570 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:14:08.943472    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a09df99-b62e-4670-9852-a452fa1c6cba-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vdztn\" (UID: \"4a09df99-b62e-4670-9852-a452fa1c6cba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vdztn"
Sep 11 23:14:08.943570 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:14:08.943504    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a09df99-b62e-4670-9852-a452fa1c6cba-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vdztn\" (UID: \"4a09df99-b62e-4670-9852-a452fa1c6cba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vdztn"
Sep 11 23:14:08.943570 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:14:08.943537    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a09df99-b62e-4670-9852-a452fa1c6cba-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vdztn\" (UID: \"4a09df99-b62e-4670-9852-a452fa1c6cba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vdztn"
Sep 11 23:14:08.943803 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:14:08.943631    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwwwm\" (UniqueName: \"kubernetes.io/projected/4a09df99-b62e-4670-9852-a452fa1c6cba-kube-api-access-bwwwm\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vdztn\" (UID: \"4a09df99-b62e-4670-9852-a452fa1c6cba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vdztn"
Sep 11 23:14:08.943803 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:14:08.943676    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a09df99-b62e-4670-9852-a452fa1c6cba-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vdztn\" (UID: \"4a09df99-b62e-4670-9852-a452fa1c6cba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vdztn"
Sep 11 23:14:09.044566 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:14:09.044523    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a09df99-b62e-4670-9852-a452fa1c6cba-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vdztn\" (UID: \"4a09df99-b62e-4670-9852-a452fa1c6cba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vdztn"
Sep 11 23:14:09.044795 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:14:09.044591    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwwwm\" (UniqueName: \"kubernetes.io/projected/4a09df99-b62e-4670-9852-a452fa1c6cba-kube-api-access-bwwwm\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vdztn\" (UID: \"4a09df99-b62e-4670-9852-a452fa1c6cba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vdztn"
Sep 11 23:14:09.044795 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:14:09.044635    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a09df99-b62e-4670-9852-a452fa1c6cba-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vdztn\" (UID: \"4a09df99-b62e-4670-9852-a452fa1c6cba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vdztn"
Sep 11 23:14:09.044795 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:14:09.044676    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a09df99-b62e-4670-9852-a452fa1c6cba-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vdztn\" (UID: \"4a09df99-b62e-4670-9852-a452fa1c6cba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vdztn"
Sep 11 23:14:09.044795 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:14:09.044726    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a09df99-b62e-4670-9852-a452fa1c6cba-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vdztn\" (UID: \"4a09df99-b62e-4670-9852-a452fa1c6cba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vdztn"
Sep 11 23:14:09.044795 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:14:09.044756    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a09df99-b62e-4670-9852-a452fa1c6cba-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vdztn\" (UID: \"4a09df99-b62e-4670-9852-a452fa1c6cba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vdztn"
Sep 11 23:14:09.044795 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:14:09.044786    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a09df99-b62e-4670-9852-a452fa1c6cba-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vdztn\" (UID: \"4a09df99-b62e-4670-9852-a452fa1c6cba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vdztn"
Sep 11 23:14:09.045117 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:14:09.044818    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a09df99-b62e-4670-9852-a452fa1c6cba-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vdztn\" (UID: \"4a09df99-b62e-4670-9852-a452fa1c6cba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vdztn"
Sep 11 23:14:09.045117 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:14:09.044861    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a09df99-b62e-4670-9852-a452fa1c6cba-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vdztn\" (UID: \"4a09df99-b62e-4670-9852-a452fa1c6cba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vdztn"
Sep 11 23:14:09.045117 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:14:09.045024    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a09df99-b62e-4670-9852-a452fa1c6cba-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vdztn\" (UID: \"4a09df99-b62e-4670-9852-a452fa1c6cba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vdztn"
Sep 11 23:14:09.045117 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:14:09.045086    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a09df99-b62e-4670-9852-a452fa1c6cba-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vdztn\" (UID: \"4a09df99-b62e-4670-9852-a452fa1c6cba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vdztn"
Sep 11 23:14:09.045344 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:14:09.045198    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a09df99-b62e-4670-9852-a452fa1c6cba-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vdztn\" (UID: \"4a09df99-b62e-4670-9852-a452fa1c6cba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vdztn"
Sep 11 23:14:09.045344 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:14:09.045268    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a09df99-b62e-4670-9852-a452fa1c6cba-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vdztn\" (UID: \"4a09df99-b62e-4670-9852-a452fa1c6cba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vdztn"
Sep 11 23:14:09.045526 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:14:09.045502    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a09df99-b62e-4670-9852-a452fa1c6cba-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vdztn\" (UID: \"4a09df99-b62e-4670-9852-a452fa1c6cba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vdztn"
Sep 11 23:14:09.047416 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:14:09.047392    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a09df99-b62e-4670-9852-a452fa1c6cba-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vdztn\" (UID: \"4a09df99-b62e-4670-9852-a452fa1c6cba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vdztn"
Sep 11 23:14:09.047625 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:14:09.047608    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a09df99-b62e-4670-9852-a452fa1c6cba-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vdztn\" (UID: \"4a09df99-b62e-4670-9852-a452fa1c6cba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vdztn"
Sep 11 23:14:09.055552 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:14:09.055515    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a09df99-b62e-4670-9852-a452fa1c6cba-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vdztn\" (UID: \"4a09df99-b62e-4670-9852-a452fa1c6cba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vdztn"
Sep 11 23:14:09.055743 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:14:09.055713    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwwwm\" (UniqueName: \"kubernetes.io/projected/4a09df99-b62e-4670-9852-a452fa1c6cba-kube-api-access-bwwwm\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vdztn\" (UID: \"4a09df99-b62e-4670-9852-a452fa1c6cba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vdztn"
Sep 11 23:14:09.231708 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:14:09.231605    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vdztn"
Sep 11 23:14:09.515189 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:14:09.515133    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vdztn"]
Sep 11 23:14:09.516271 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:14:09.516243    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a09df99_b62e_4670_9852_a452fa1c6cba.slice/crio-5c54b92c2f0918a17cad9e0743c37120abb0e56ab39580341c8aeed62cb01289 WatchSource:0}: Error finding container 5c54b92c2f0918a17cad9e0743c37120abb0e56ab39580341c8aeed62cb01289: Status 404 returned error can't find the container with id 5c54b92c2f0918a17cad9e0743c37120abb0e56ab39580341c8aeed62cb01289
Sep 11 23:14:09.518643 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:14:09.518609    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:14:09.518757 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:14:09.518681    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:14:09.518757 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:14:09.518713    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:14:09.825526 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:14:09.825420    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vdztn" event={"ID":"4a09df99-b62e-4670-9852-a452fa1c6cba","Type":"ContainerStarted","Data":"6ae0a3d0c3a369ddd78d7f6f77aa594be63918f554069a52277912d3e1205cc1"}
Sep 11 23:14:09.825526 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:14:09.825517    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vdztn" event={"ID":"4a09df99-b62e-4670-9852-a452fa1c6cba","Type":"ContainerStarted","Data":"5c54b92c2f0918a17cad9e0743c37120abb0e56ab39580341c8aeed62cb01289"}
Sep 11 23:14:09.848987 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:14:09.848902    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vdztn" podStartSLOduration=1.848880949 podStartE2EDuration="1.848880949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 23:14:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:14:09.847330664 +0000 UTC m=+773.880459894" watchObservedRunningTime="2026-09-11 23:14:09.848880949 +0000 UTC m=+773.882010169"
Sep 11 23:14:10.232598 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:14:10.232552    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vdztn"
Sep 11 23:14:10.237455 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:14:10.237424    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vdztn"
Sep 11 23:14:10.831708 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:14:10.831669    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vdztn"
Sep 11 23:14:10.832713 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:14:10.832691    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vdztn"
Sep 11 23:14:24.928267 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:14:24.928224    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"]
Sep 11 23:14:24.933235 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:14:24.933214    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"
Sep 11 23:14:24.936573 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:14:24.936542    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-mzkwj\""
Sep 11 23:14:24.936573 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:14:24.936563    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 11 23:14:24.943099 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:14:24.943071    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"]
Sep 11 23:14:24.973115 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:14:24.973080    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"]
Sep 11 23:14:24.996200 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:14:24.996142    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c13e18e8-e890-44b5-988e-049d0b5cc447-config-file\") pod \"limitador-limitador-84994f7ffc-pwbzk\" (UID: \"c13e18e8-e890-44b5-988e-049d0b5cc447\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"
Sep 11 23:14:24.996381 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:14:24.996268    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbqp2\" (UniqueName: \"kubernetes.io/projected/c13e18e8-e890-44b5-988e-049d0b5cc447-kube-api-access-qbqp2\") pod \"limitador-limitador-84994f7ffc-pwbzk\" (UID: \"c13e18e8-e890-44b5-988e-049d0b5cc447\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"
Sep 11 23:14:25.097377 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:14:25.097341    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c13e18e8-e890-44b5-988e-049d0b5cc447-config-file\") pod \"limitador-limitador-84994f7ffc-pwbzk\" (UID: \"c13e18e8-e890-44b5-988e-049d0b5cc447\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"
Sep 11 23:14:25.097560 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:14:25.097402    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbqp2\" (UniqueName: \"kubernetes.io/projected/c13e18e8-e890-44b5-988e-049d0b5cc447-kube-api-access-qbqp2\") pod \"limitador-limitador-84994f7ffc-pwbzk\" (UID: \"c13e18e8-e890-44b5-988e-049d0b5cc447\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"
Sep 11 23:14:25.097997 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:14:25.097977    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c13e18e8-e890-44b5-988e-049d0b5cc447-config-file\") pod \"limitador-limitador-84994f7ffc-pwbzk\" (UID: \"c13e18e8-e890-44b5-988e-049d0b5cc447\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"
Sep 11 23:14:25.108533 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:14:25.108506    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbqp2\" (UniqueName: \"kubernetes.io/projected/c13e18e8-e890-44b5-988e-049d0b5cc447-kube-api-access-qbqp2\") pod \"limitador-limitador-84994f7ffc-pwbzk\" (UID: \"c13e18e8-e890-44b5-988e-049d0b5cc447\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"
Sep 11 23:14:25.245990 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:14:25.245898    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"
Sep 11 23:14:25.520959 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:14:25.520927    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"]
Sep 11 23:14:25.522391 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:14:25.522362    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc13e18e8_e890_44b5_988e_049d0b5cc447.slice/crio-bd0987476e5299752d4f45c6746a34502783a85bd56ecb8407fb6b2f1d6ce385 WatchSource:0}: Error finding container bd0987476e5299752d4f45c6746a34502783a85bd56ecb8407fb6b2f1d6ce385: Status 404 returned error can't find the container with id bd0987476e5299752d4f45c6746a34502783a85bd56ecb8407fb6b2f1d6ce385
Sep 11 23:14:25.916995 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:14:25.916947    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-pwbzk" event={"ID":"c13e18e8-e890-44b5-988e-049d0b5cc447","Type":"ContainerStarted","Data":"bd0987476e5299752d4f45c6746a34502783a85bd56ecb8407fb6b2f1d6ce385"}
Sep 11 23:14:26.923824 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:14:26.923790    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-pwbzk" event={"ID":"c13e18e8-e890-44b5-988e-049d0b5cc447","Type":"ContainerStarted","Data":"640dcf1460537412aa2e52258299bf6deebe8bd3a451e9914e92e22d22e5d1bf"}
Sep 11 23:14:26.924251 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:14:26.923933    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"
Sep 11 23:14:26.942705 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:14:26.942655    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-pwbzk" podStartSLOduration=1.984881122 podStartE2EDuration="2.942638911s" totalImagesPullingTime="957.757789ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 23:14:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:14:26.940512455 +0000 UTC m=+790.973641697" watchObservedRunningTime="2026-09-11 23:14:26.942638911 +0000 UTC m=+790.975768127"
Sep 11 23:14:37.931409 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:14:37.931325    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"
Sep 11 23:15:40.208615 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:15:40.208547    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"]
Sep 11 23:15:50.410241 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:15:50.410209    2546 generic.go:356] "Generic (PLEG): container finished" podID="4a09df99-b62e-4670-9852-a452fa1c6cba" containerID="6ae0a3d0c3a369ddd78d7f6f77aa594be63918f554069a52277912d3e1205cc1" exitCode=0
Sep 11 23:15:50.410670 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:15:50.410262    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vdztn" event={"ID":"4a09df99-b62e-4670-9852-a452fa1c6cba","Type":"ContainerDied","Data":"6ae0a3d0c3a369ddd78d7f6f77aa594be63918f554069a52277912d3e1205cc1"}
Sep 11 23:15:50.410670 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:15:50.410627    2546 scope.go:117] "RemoveContainer" containerID="6ae0a3d0c3a369ddd78d7f6f77aa594be63918f554069a52277912d3e1205cc1"
Sep 11 23:15:50.411689 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:15:50.411658    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:15:50.411784 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:15:50.411742    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:15:50.411849 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:15:50.411793    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:15:51.418329 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:15:51.418294    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vdztn" event={"ID":"4a09df99-b62e-4670-9852-a452fa1c6cba","Type":"ContainerStarted","Data":"b49d82247b19b64963133e6ee80e2c286ce2e75881e3f57e74ef653e81dd669d"}
Sep 11 23:15:52.233620 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:15:52.233581    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vdztn"
Sep 11 23:15:52.233822 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:15:52.233770    2546 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-vdztn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Sep 11 23:15:52.233910 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:15:52.233823    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vdztn" podUID="4a09df99-b62e-4670-9852-a452fa1c6cba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Sep 11 23:15:53.232591 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:15:53.232545    2546 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-vdztn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Sep 11 23:15:53.233123 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:15:53.232625    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vdztn" podUID="4a09df99-b62e-4670-9852-a452fa1c6cba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Sep 11 23:15:54.232359 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:15:54.232307    2546 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-vdztn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Sep 11 23:15:54.232718 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:15:54.232399    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vdztn" podUID="4a09df99-b62e-4670-9852-a452fa1c6cba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Sep 11 23:15:55.232915 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:15:55.232867    2546 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-vdztn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Sep 11 23:15:55.233458 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:15:55.232938    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vdztn" podUID="4a09df99-b62e-4670-9852-a452fa1c6cba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Sep 11 23:15:55.832084 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:15:55.832035    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vdztn"
Sep 11 23:15:56.232856 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:15:56.232812    2546 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-vdztn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Sep 11 23:15:56.233199 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:15:56.233139    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vdztn" podUID="4a09df99-b62e-4670-9852-a452fa1c6cba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Sep 11 23:15:57.236764 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:15:57.236732    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vdztn"
Sep 11 23:15:57.237758 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:15:57.237740    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vdztn"
Sep 11 23:16:12.784714 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:16:12.784632    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"]
Sep 11 23:16:16.480674 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:16:16.480640    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7244_6b0a8af3-932c-4091-b7a0-180fe1751a3a/ovn-acl-logging/0.log"
Sep 11 23:16:16.481479 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:16:16.481458    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7244_6b0a8af3-932c-4091-b7a0-180fe1751a3a/ovn-acl-logging/0.log"
Sep 11 23:16:20.593753 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:16:20.593716    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"]
Sep 11 23:16:28.287492 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:16:28.287454    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"]
Sep 11 23:16:31.886020 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:16:31.885980    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"]
Sep 11 23:16:37.884185 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:16:37.884131    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"]
Sep 11 23:16:41.818722 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:16:41.818689    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-n87mz"]
Sep 11 23:16:41.822612 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:16:41.822580    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n87mz"
Sep 11 23:16:41.826093 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:16:41.826070    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 11 23:16:41.826265 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:16:41.826145    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-5pwjc\""
Sep 11 23:16:41.826349 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:16:41.826070    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 11 23:16:41.827463 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:16:41.827444    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 11 23:16:41.834491 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:16:41.834468    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-n87mz"]
Sep 11 23:16:41.959259 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:16:41.959223    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6c49bb6-490d-4570-8f86-212c566c5b90-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-n87mz\" (UID: \"a6c49bb6-490d-4570-8f86-212c566c5b90\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n87mz"
Sep 11 23:16:41.959441 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:16:41.959263    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6c49bb6-490d-4570-8f86-212c566c5b90-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-n87mz\" (UID: \"a6c49bb6-490d-4570-8f86-212c566c5b90\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n87mz"
Sep 11 23:16:41.959441 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:16:41.959323    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6c49bb6-490d-4570-8f86-212c566c5b90-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-n87mz\" (UID: \"a6c49bb6-490d-4570-8f86-212c566c5b90\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n87mz"
Sep 11 23:16:41.959441 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:16:41.959422    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l29gf\" (UniqueName: \"kubernetes.io/projected/a6c49bb6-490d-4570-8f86-212c566c5b90-kube-api-access-l29gf\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-n87mz\" (UID: \"a6c49bb6-490d-4570-8f86-212c566c5b90\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n87mz"
Sep 11 23:16:41.959571 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:16:41.959456    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6c49bb6-490d-4570-8f86-212c566c5b90-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-n87mz\" (UID: \"a6c49bb6-490d-4570-8f86-212c566c5b90\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n87mz"
Sep 11 23:16:41.959571 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:16:41.959481    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6c49bb6-490d-4570-8f86-212c566c5b90-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-n87mz\" (UID: \"a6c49bb6-490d-4570-8f86-212c566c5b90\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n87mz"
Sep 11 23:16:42.060552 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:16:42.060511    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6c49bb6-490d-4570-8f86-212c566c5b90-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-n87mz\" (UID: \"a6c49bb6-490d-4570-8f86-212c566c5b90\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n87mz"
Sep 11 23:16:42.060747 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:16:42.060577    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6c49bb6-490d-4570-8f86-212c566c5b90-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-n87mz\" (UID: \"a6c49bb6-490d-4570-8f86-212c566c5b90\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n87mz"
Sep 11 23:16:42.060747 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:16:42.060598    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6c49bb6-490d-4570-8f86-212c566c5b90-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-n87mz\" (UID: \"a6c49bb6-490d-4570-8f86-212c566c5b90\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n87mz"
Sep 11 23:16:42.060747 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:16:42.060625    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6c49bb6-490d-4570-8f86-212c566c5b90-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-n87mz\" (UID: \"a6c49bb6-490d-4570-8f86-212c566c5b90\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n87mz"
Sep 11 23:16:42.060747 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:16:42.060673    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l29gf\" (UniqueName: \"kubernetes.io/projected/a6c49bb6-490d-4570-8f86-212c566c5b90-kube-api-access-l29gf\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-n87mz\" (UID: \"a6c49bb6-490d-4570-8f86-212c566c5b90\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n87mz"
Sep 11 23:16:42.060747 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:16:42.060712    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6c49bb6-490d-4570-8f86-212c566c5b90-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-n87mz\" (UID: \"a6c49bb6-490d-4570-8f86-212c566c5b90\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n87mz"
Sep 11 23:16:42.061003 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:16:42.060980    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6c49bb6-490d-4570-8f86-212c566c5b90-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-n87mz\" (UID: \"a6c49bb6-490d-4570-8f86-212c566c5b90\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n87mz"
Sep 11 23:16:42.061051 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:16:42.061017    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6c49bb6-490d-4570-8f86-212c566c5b90-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-n87mz\" (UID: \"a6c49bb6-490d-4570-8f86-212c566c5b90\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n87mz"
Sep 11 23:16:42.061094 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:16:42.061050    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6c49bb6-490d-4570-8f86-212c566c5b90-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-n87mz\" (UID: \"a6c49bb6-490d-4570-8f86-212c566c5b90\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n87mz"
Sep 11 23:16:42.063146 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:16:42.063125    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6c49bb6-490d-4570-8f86-212c566c5b90-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-n87mz\" (UID: \"a6c49bb6-490d-4570-8f86-212c566c5b90\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n87mz"
Sep 11 23:16:42.063408 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:16:42.063390    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6c49bb6-490d-4570-8f86-212c566c5b90-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-n87mz\" (UID: \"a6c49bb6-490d-4570-8f86-212c566c5b90\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n87mz"
Sep 11 23:16:42.069246 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:16:42.069191    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l29gf\" (UniqueName: \"kubernetes.io/projected/a6c49bb6-490d-4570-8f86-212c566c5b90-kube-api-access-l29gf\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-n87mz\" (UID: \"a6c49bb6-490d-4570-8f86-212c566c5b90\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n87mz"
Sep 11 23:16:42.136168 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:16:42.136124    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n87mz"
Sep 11 23:16:42.414143 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:16:42.414116    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-n87mz"]
Sep 11 23:16:42.416000 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:16:42.415972    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6c49bb6_490d_4570_8f86_212c566c5b90.slice/crio-d52a20fce72bb1968fa54cf36699d104419ce9a3afe934b0ea41274a69b50693 WatchSource:0}: Error finding container d52a20fce72bb1968fa54cf36699d104419ce9a3afe934b0ea41274a69b50693: Status 404 returned error can't find the container with id d52a20fce72bb1968fa54cf36699d104419ce9a3afe934b0ea41274a69b50693
Sep 11 23:16:42.724548 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:16:42.724454    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n87mz" event={"ID":"a6c49bb6-490d-4570-8f86-212c566c5b90","Type":"ContainerStarted","Data":"d52a20fce72bb1968fa54cf36699d104419ce9a3afe934b0ea41274a69b50693"}
Sep 11 23:16:43.732965 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:16:43.732917    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n87mz" event={"ID":"a6c49bb6-490d-4570-8f86-212c566c5b90","Type":"ContainerStarted","Data":"8fc2874bdca1b1863aa1394bd2672b537b34d5e067cedd068094f23256fe907b"}
Sep 11 23:16:43.752188 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:16:43.752119    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n87mz" podStartSLOduration=1.766555259 podStartE2EDuration="2.752102214s" totalImagesPullingTime="985.546955ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 23:16:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:16:43.750748478 +0000 UTC m=+927.783877698" watchObservedRunningTime="2026-09-11 23:16:43.752102214 +0000 UTC m=+927.785231477"
Sep 11 23:16:44.392253 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:16:44.392220    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"]
Sep 11 23:16:52.137137 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:16:52.137091    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n87mz"
Sep 11 23:16:52.137560 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:16:52.137193    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n87mz"
Sep 11 23:16:52.149866 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:16:52.149831    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n87mz"
Sep 11 23:16:52.498252 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:16:52.498216    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"]
Sep 11 23:16:52.800109 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:16:52.800020    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n87mz"
Sep 11 23:18:20.996487 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:20.996453    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp"]
Sep 11 23:18:20.999999 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:20.999975    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp"
Sep 11 23:18:21.003458 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:21.003436    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-12fb6b-openshift-default-dockercfg-8mggn\""
Sep 11 23:18:21.012401 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:21.012374    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp"]
Sep 11 23:18:21.144476 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:21.144444    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48592704-31d9-4e6f-bad1-ac862a6dfa72-istio-podinfo\") pod \"e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp\" (UID: \"48592704-31d9-4e6f-bad1-ac862a6dfa72\") " pod="openshift-ingress/e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp"
Sep 11 23:18:21.144476 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:21.144486    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48592704-31d9-4e6f-bad1-ac862a6dfa72-workload-socket\") pod \"e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp\" (UID: \"48592704-31d9-4e6f-bad1-ac862a6dfa72\") " pod="openshift-ingress/e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp"
Sep 11 23:18:21.144770 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:21.144530    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48592704-31d9-4e6f-bad1-ac862a6dfa72-credential-socket\") pod \"e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp\" (UID: \"48592704-31d9-4e6f-bad1-ac862a6dfa72\") " pod="openshift-ingress/e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp"
Sep 11 23:18:21.144770 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:21.144594    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6hq2\" (UniqueName: \"kubernetes.io/projected/48592704-31d9-4e6f-bad1-ac862a6dfa72-kube-api-access-t6hq2\") pod \"e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp\" (UID: \"48592704-31d9-4e6f-bad1-ac862a6dfa72\") " pod="openshift-ingress/e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp"
Sep 11 23:18:21.144770 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:21.144671    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48592704-31d9-4e6f-bad1-ac862a6dfa72-workload-certs\") pod \"e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp\" (UID: \"48592704-31d9-4e6f-bad1-ac862a6dfa72\") " pod="openshift-ingress/e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp"
Sep 11 23:18:21.144770 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:21.144715    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48592704-31d9-4e6f-bad1-ac862a6dfa72-istio-token\") pod \"e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp\" (UID: \"48592704-31d9-4e6f-bad1-ac862a6dfa72\") " pod="openshift-ingress/e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp"
Sep 11 23:18:21.144770 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:21.144739    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48592704-31d9-4e6f-bad1-ac862a6dfa72-istiod-ca-cert\") pod \"e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp\" (UID: \"48592704-31d9-4e6f-bad1-ac862a6dfa72\") " pod="openshift-ingress/e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp"
Sep 11 23:18:21.145035 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:21.144846    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48592704-31d9-4e6f-bad1-ac862a6dfa72-istio-envoy\") pod \"e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp\" (UID: \"48592704-31d9-4e6f-bad1-ac862a6dfa72\") " pod="openshift-ingress/e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp"
Sep 11 23:18:21.145035 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:21.144877    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48592704-31d9-4e6f-bad1-ac862a6dfa72-istio-data\") pod \"e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp\" (UID: \"48592704-31d9-4e6f-bad1-ac862a6dfa72\") " pod="openshift-ingress/e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp"
Sep 11 23:18:21.167940 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:21.167912    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-92d8ce-openshift-default-5ff8db57b8-8xvzw"]
Sep 11 23:18:21.171504 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:21.171484    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-92d8ce-openshift-default-5ff8db57b8-8xvzw"
Sep 11 23:18:21.174627 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:21.174608    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-92d8ce-openshift-default-dockercfg-gw6zj\""
Sep 11 23:18:21.184267 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:21.184246    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-92d8ce-openshift-default-5ff8db57b8-8xvzw"]
Sep 11 23:18:21.246254 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:21.246222    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2304922d-c8d4-4a95-a81c-4af2ccea36b0-istio-token\") pod \"e2e-inf-a-92d8ce-openshift-default-5ff8db57b8-8xvzw\" (UID: \"2304922d-c8d4-4a95-a81c-4af2ccea36b0\") " pod="openshift-ingress/e2e-inf-a-92d8ce-openshift-default-5ff8db57b8-8xvzw"
Sep 11 23:18:21.246422 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:21.246269    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48592704-31d9-4e6f-bad1-ac862a6dfa72-istio-token\") pod \"e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp\" (UID: \"48592704-31d9-4e6f-bad1-ac862a6dfa72\") " pod="openshift-ingress/e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp"
Sep 11 23:18:21.246422 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:21.246291    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48592704-31d9-4e6f-bad1-ac862a6dfa72-workload-certs\") pod \"e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp\" (UID: \"48592704-31d9-4e6f-bad1-ac862a6dfa72\") " pod="openshift-ingress/e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp"
Sep 11 23:18:21.246422 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:21.246403    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2304922d-c8d4-4a95-a81c-4af2ccea36b0-workload-certs\") pod \"e2e-inf-a-92d8ce-openshift-default-5ff8db57b8-8xvzw\" (UID: \"2304922d-c8d4-4a95-a81c-4af2ccea36b0\") " pod="openshift-ingress/e2e-inf-a-92d8ce-openshift-default-5ff8db57b8-8xvzw"
Sep 11 23:18:21.246577 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:21.246470    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2304922d-c8d4-4a95-a81c-4af2ccea36b0-istio-data\") pod \"e2e-inf-a-92d8ce-openshift-default-5ff8db57b8-8xvzw\" (UID: \"2304922d-c8d4-4a95-a81c-4af2ccea36b0\") " pod="openshift-ingress/e2e-inf-a-92d8ce-openshift-default-5ff8db57b8-8xvzw"
Sep 11 23:18:21.246577 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:21.246498    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2304922d-c8d4-4a95-a81c-4af2ccea36b0-istio-podinfo\") pod \"e2e-inf-a-92d8ce-openshift-default-5ff8db57b8-8xvzw\" (UID: \"2304922d-c8d4-4a95-a81c-4af2ccea36b0\") " pod="openshift-ingress/e2e-inf-a-92d8ce-openshift-default-5ff8db57b8-8xvzw"
Sep 11 23:18:21.246577 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:21.246530    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48592704-31d9-4e6f-bad1-ac862a6dfa72-workload-socket\") pod \"e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp\" (UID: \"48592704-31d9-4e6f-bad1-ac862a6dfa72\") " pod="openshift-ingress/e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp"
Sep 11 23:18:21.246577 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:21.246553    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46bbf\" (UniqueName: \"kubernetes.io/projected/2304922d-c8d4-4a95-a81c-4af2ccea36b0-kube-api-access-46bbf\") pod \"e2e-inf-a-92d8ce-openshift-default-5ff8db57b8-8xvzw\" (UID: \"2304922d-c8d4-4a95-a81c-4af2ccea36b0\") " pod="openshift-ingress/e2e-inf-a-92d8ce-openshift-default-5ff8db57b8-8xvzw"
Sep 11 23:18:21.246732 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:21.246611    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48592704-31d9-4e6f-bad1-ac862a6dfa72-credential-socket\") pod \"e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp\" (UID: \"48592704-31d9-4e6f-bad1-ac862a6dfa72\") " pod="openshift-ingress/e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp"
Sep 11 23:18:21.246732 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:21.246638    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2304922d-c8d4-4a95-a81c-4af2ccea36b0-istio-envoy\") pod \"e2e-inf-a-92d8ce-openshift-default-5ff8db57b8-8xvzw\" (UID: \"2304922d-c8d4-4a95-a81c-4af2ccea36b0\") " pod="openshift-ingress/e2e-inf-a-92d8ce-openshift-default-5ff8db57b8-8xvzw"
Sep 11 23:18:21.246732 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:21.246665    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2304922d-c8d4-4a95-a81c-4af2ccea36b0-istiod-ca-cert\") pod \"e2e-inf-a-92d8ce-openshift-default-5ff8db57b8-8xvzw\" (UID: \"2304922d-c8d4-4a95-a81c-4af2ccea36b0\") " pod="openshift-ingress/e2e-inf-a-92d8ce-openshift-default-5ff8db57b8-8xvzw"
Sep 11 23:18:21.246732 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:21.246617    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48592704-31d9-4e6f-bad1-ac862a6dfa72-workload-certs\") pod \"e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp\" (UID: \"48592704-31d9-4e6f-bad1-ac862a6dfa72\") " pod="openshift-ingress/e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp"
Sep 11 23:18:21.246732 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:21.246687    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6hq2\" (UniqueName: \"kubernetes.io/projected/48592704-31d9-4e6f-bad1-ac862a6dfa72-kube-api-access-t6hq2\") pod \"e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp\" (UID: \"48592704-31d9-4e6f-bad1-ac862a6dfa72\") " pod="openshift-ingress/e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp"
Sep 11 23:18:21.246944 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:21.246799    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48592704-31d9-4e6f-bad1-ac862a6dfa72-istiod-ca-cert\") pod \"e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp\" (UID: \"48592704-31d9-4e6f-bad1-ac862a6dfa72\") " pod="openshift-ingress/e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp"
Sep 11 23:18:21.246944 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:21.246829    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48592704-31d9-4e6f-bad1-ac862a6dfa72-workload-socket\") pod \"e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp\" (UID: \"48592704-31d9-4e6f-bad1-ac862a6dfa72\") " pod="openshift-ingress/e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp"
Sep 11 23:18:21.246944 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:21.246841    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2304922d-c8d4-4a95-a81c-4af2ccea36b0-workload-socket\") pod \"e2e-inf-a-92d8ce-openshift-default-5ff8db57b8-8xvzw\" (UID: \"2304922d-c8d4-4a95-a81c-4af2ccea36b0\") " pod="openshift-ingress/e2e-inf-a-92d8ce-openshift-default-5ff8db57b8-8xvzw"
Sep 11 23:18:21.246944 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:21.246913    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48592704-31d9-4e6f-bad1-ac862a6dfa72-credential-socket\") pod \"e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp\" (UID: \"48592704-31d9-4e6f-bad1-ac862a6dfa72\") " pod="openshift-ingress/e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp"
Sep 11 23:18:21.246944 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:21.246917    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48592704-31d9-4e6f-bad1-ac862a6dfa72-istio-envoy\") pod \"e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp\" (UID: \"48592704-31d9-4e6f-bad1-ac862a6dfa72\") " pod="openshift-ingress/e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp"
Sep 11 23:18:21.247141 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:21.246958    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48592704-31d9-4e6f-bad1-ac862a6dfa72-istio-data\") pod \"e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp\" (UID: \"48592704-31d9-4e6f-bad1-ac862a6dfa72\") " pod="openshift-ingress/e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp"
Sep 11 23:18:21.247141 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:21.246994    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2304922d-c8d4-4a95-a81c-4af2ccea36b0-credential-socket\") pod \"e2e-inf-a-92d8ce-openshift-default-5ff8db57b8-8xvzw\" (UID: \"2304922d-c8d4-4a95-a81c-4af2ccea36b0\") " pod="openshift-ingress/e2e-inf-a-92d8ce-openshift-default-5ff8db57b8-8xvzw"
Sep 11 23:18:21.247141 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:21.247033    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48592704-31d9-4e6f-bad1-ac862a6dfa72-istio-podinfo\") pod \"e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp\" (UID: \"48592704-31d9-4e6f-bad1-ac862a6dfa72\") " pod="openshift-ingress/e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp"
Sep 11 23:18:21.247312 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:21.247292    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48592704-31d9-4e6f-bad1-ac862a6dfa72-istiod-ca-cert\") pod \"e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp\" (UID: \"48592704-31d9-4e6f-bad1-ac862a6dfa72\") " pod="openshift-ingress/e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp"
Sep 11 23:18:21.247373 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:21.247294    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48592704-31d9-4e6f-bad1-ac862a6dfa72-istio-data\") pod \"e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp\" (UID: \"48592704-31d9-4e6f-bad1-ac862a6dfa72\") " pod="openshift-ingress/e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp"
Sep 11 23:18:21.249412 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:21.249385    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48592704-31d9-4e6f-bad1-ac862a6dfa72-istio-envoy\") pod \"e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp\" (UID: \"48592704-31d9-4e6f-bad1-ac862a6dfa72\") " pod="openshift-ingress/e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp"
Sep 11 23:18:21.249619 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:21.249602    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48592704-31d9-4e6f-bad1-ac862a6dfa72-istio-podinfo\") pod \"e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp\" (UID: \"48592704-31d9-4e6f-bad1-ac862a6dfa72\") " pod="openshift-ingress/e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp"
Sep 11 23:18:21.256283 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:21.256257    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48592704-31d9-4e6f-bad1-ac862a6dfa72-istio-token\") pod \"e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp\" (UID: \"48592704-31d9-4e6f-bad1-ac862a6dfa72\") " pod="openshift-ingress/e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp"
Sep 11 23:18:21.260639 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:21.260612    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6hq2\" (UniqueName: \"kubernetes.io/projected/48592704-31d9-4e6f-bad1-ac862a6dfa72-kube-api-access-t6hq2\") pod \"e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp\" (UID: \"48592704-31d9-4e6f-bad1-ac862a6dfa72\") " pod="openshift-ingress/e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp"
Sep 11 23:18:21.311932 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:21.311894    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp"
Sep 11 23:18:21.348440 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:21.348398    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2304922d-c8d4-4a95-a81c-4af2ccea36b0-workload-certs\") pod \"e2e-inf-a-92d8ce-openshift-default-5ff8db57b8-8xvzw\" (UID: \"2304922d-c8d4-4a95-a81c-4af2ccea36b0\") " pod="openshift-ingress/e2e-inf-a-92d8ce-openshift-default-5ff8db57b8-8xvzw"
Sep 11 23:18:21.348617 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:21.348474    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2304922d-c8d4-4a95-a81c-4af2ccea36b0-istio-data\") pod \"e2e-inf-a-92d8ce-openshift-default-5ff8db57b8-8xvzw\" (UID: \"2304922d-c8d4-4a95-a81c-4af2ccea36b0\") " pod="openshift-ingress/e2e-inf-a-92d8ce-openshift-default-5ff8db57b8-8xvzw"
Sep 11 23:18:21.348617 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:21.348520    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2304922d-c8d4-4a95-a81c-4af2ccea36b0-istio-podinfo\") pod \"e2e-inf-a-92d8ce-openshift-default-5ff8db57b8-8xvzw\" (UID: \"2304922d-c8d4-4a95-a81c-4af2ccea36b0\") " pod="openshift-ingress/e2e-inf-a-92d8ce-openshift-default-5ff8db57b8-8xvzw"
Sep 11 23:18:21.348617 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:21.348554    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46bbf\" (UniqueName: \"kubernetes.io/projected/2304922d-c8d4-4a95-a81c-4af2ccea36b0-kube-api-access-46bbf\") pod \"e2e-inf-a-92d8ce-openshift-default-5ff8db57b8-8xvzw\" (UID: \"2304922d-c8d4-4a95-a81c-4af2ccea36b0\") " pod="openshift-ingress/e2e-inf-a-92d8ce-openshift-default-5ff8db57b8-8xvzw"
Sep 11 23:18:21.348617 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:21.348592    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2304922d-c8d4-4a95-a81c-4af2ccea36b0-istio-envoy\") pod \"e2e-inf-a-92d8ce-openshift-default-5ff8db57b8-8xvzw\" (UID: \"2304922d-c8d4-4a95-a81c-4af2ccea36b0\") " pod="openshift-ingress/e2e-inf-a-92d8ce-openshift-default-5ff8db57b8-8xvzw"
Sep 11 23:18:21.348860 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:21.348630    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2304922d-c8d4-4a95-a81c-4af2ccea36b0-istiod-ca-cert\") pod \"e2e-inf-a-92d8ce-openshift-default-5ff8db57b8-8xvzw\" (UID: \"2304922d-c8d4-4a95-a81c-4af2ccea36b0\") " pod="openshift-ingress/e2e-inf-a-92d8ce-openshift-default-5ff8db57b8-8xvzw"
Sep 11 23:18:21.348860 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:21.348672    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2304922d-c8d4-4a95-a81c-4af2ccea36b0-workload-socket\") pod \"e2e-inf-a-92d8ce-openshift-default-5ff8db57b8-8xvzw\" (UID: \"2304922d-c8d4-4a95-a81c-4af2ccea36b0\") " pod="openshift-ingress/e2e-inf-a-92d8ce-openshift-default-5ff8db57b8-8xvzw"
Sep 11 23:18:21.348860 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:21.348720    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2304922d-c8d4-4a95-a81c-4af2ccea36b0-credential-socket\") pod \"e2e-inf-a-92d8ce-openshift-default-5ff8db57b8-8xvzw\" (UID: \"2304922d-c8d4-4a95-a81c-4af2ccea36b0\") " pod="openshift-ingress/e2e-inf-a-92d8ce-openshift-default-5ff8db57b8-8xvzw"
Sep 11 23:18:21.348860 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:21.348789    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2304922d-c8d4-4a95-a81c-4af2ccea36b0-istio-token\") pod \"e2e-inf-a-92d8ce-openshift-default-5ff8db57b8-8xvzw\" (UID: \"2304922d-c8d4-4a95-a81c-4af2ccea36b0\") " pod="openshift-ingress/e2e-inf-a-92d8ce-openshift-default-5ff8db57b8-8xvzw"
Sep 11 23:18:21.349318 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:21.349292    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2304922d-c8d4-4a95-a81c-4af2ccea36b0-workload-certs\") pod \"e2e-inf-a-92d8ce-openshift-default-5ff8db57b8-8xvzw\" (UID: \"2304922d-c8d4-4a95-a81c-4af2ccea36b0\") " pod="openshift-ingress/e2e-inf-a-92d8ce-openshift-default-5ff8db57b8-8xvzw"
Sep 11 23:18:21.349413 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:21.349388    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2304922d-c8d4-4a95-a81c-4af2ccea36b0-workload-socket\") pod \"e2e-inf-a-92d8ce-openshift-default-5ff8db57b8-8xvzw\" (UID: \"2304922d-c8d4-4a95-a81c-4af2ccea36b0\") " pod="openshift-ingress/e2e-inf-a-92d8ce-openshift-default-5ff8db57b8-8xvzw"
Sep 11 23:18:21.349565 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:21.349543    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2304922d-c8d4-4a95-a81c-4af2ccea36b0-credential-socket\") pod \"e2e-inf-a-92d8ce-openshift-default-5ff8db57b8-8xvzw\" (UID: \"2304922d-c8d4-4a95-a81c-4af2ccea36b0\") " pod="openshift-ingress/e2e-inf-a-92d8ce-openshift-default-5ff8db57b8-8xvzw"
Sep 11 23:18:21.349718 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:21.349691    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2304922d-c8d4-4a95-a81c-4af2ccea36b0-istiod-ca-cert\") pod \"e2e-inf-a-92d8ce-openshift-default-5ff8db57b8-8xvzw\" (UID: \"2304922d-c8d4-4a95-a81c-4af2ccea36b0\") " pod="openshift-ingress/e2e-inf-a-92d8ce-openshift-default-5ff8db57b8-8xvzw"
Sep 11 23:18:21.349932 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:21.349858    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2304922d-c8d4-4a95-a81c-4af2ccea36b0-istio-data\") pod \"e2e-inf-a-92d8ce-openshift-default-5ff8db57b8-8xvzw\" (UID: \"2304922d-c8d4-4a95-a81c-4af2ccea36b0\") " pod="openshift-ingress/e2e-inf-a-92d8ce-openshift-default-5ff8db57b8-8xvzw"
Sep 11 23:18:21.352331 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:21.352259    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2304922d-c8d4-4a95-a81c-4af2ccea36b0-istio-podinfo\") pod \"e2e-inf-a-92d8ce-openshift-default-5ff8db57b8-8xvzw\" (UID: \"2304922d-c8d4-4a95-a81c-4af2ccea36b0\") " pod="openshift-ingress/e2e-inf-a-92d8ce-openshift-default-5ff8db57b8-8xvzw"
Sep 11 23:18:21.352602 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:21.352536    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2304922d-c8d4-4a95-a81c-4af2ccea36b0-istio-envoy\") pod \"e2e-inf-a-92d8ce-openshift-default-5ff8db57b8-8xvzw\" (UID: \"2304922d-c8d4-4a95-a81c-4af2ccea36b0\") " pod="openshift-ingress/e2e-inf-a-92d8ce-openshift-default-5ff8db57b8-8xvzw"
Sep 11 23:18:21.361370 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:21.361324    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46bbf\" (UniqueName: \"kubernetes.io/projected/2304922d-c8d4-4a95-a81c-4af2ccea36b0-kube-api-access-46bbf\") pod \"e2e-inf-a-92d8ce-openshift-default-5ff8db57b8-8xvzw\" (UID: \"2304922d-c8d4-4a95-a81c-4af2ccea36b0\") " pod="openshift-ingress/e2e-inf-a-92d8ce-openshift-default-5ff8db57b8-8xvzw"
Sep 11 23:18:21.361627 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:21.361597    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2304922d-c8d4-4a95-a81c-4af2ccea36b0-istio-token\") pod \"e2e-inf-a-92d8ce-openshift-default-5ff8db57b8-8xvzw\" (UID: \"2304922d-c8d4-4a95-a81c-4af2ccea36b0\") " pod="openshift-ingress/e2e-inf-a-92d8ce-openshift-default-5ff8db57b8-8xvzw"
Sep 11 23:18:21.481266 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:21.481235    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-92d8ce-openshift-default-5ff8db57b8-8xvzw"
Sep 11 23:18:21.605852 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:21.605820    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp"]
Sep 11 23:18:21.607300 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:18:21.607267    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48592704_31d9_4e6f_bad1_ac862a6dfa72.slice/crio-69e2b66a45ce76d3c745c018d4efec12145cb5e4a3a45246511f7736cbdb684a WatchSource:0}: Error finding container 69e2b66a45ce76d3c745c018d4efec12145cb5e4a3a45246511f7736cbdb684a: Status 404 returned error can't find the container with id 69e2b66a45ce76d3c745c018d4efec12145cb5e4a3a45246511f7736cbdb684a
Sep 11 23:18:21.609599 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:21.609568    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:18:21.609680 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:21.609631    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:18:21.609680 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:21.609661    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:18:21.772367 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:21.772336    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-92d8ce-openshift-default-5ff8db57b8-8xvzw"]
Sep 11 23:18:21.774965 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:18:21.774935    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2304922d_c8d4_4a95_a81c_4af2ccea36b0.slice/crio-c09cd6895a18a7914d1752a70535cabe1e2f3ec13f70b3d8bffe669a4b2d2904 WatchSource:0}: Error finding container c09cd6895a18a7914d1752a70535cabe1e2f3ec13f70b3d8bffe669a4b2d2904: Status 404 returned error can't find the container with id c09cd6895a18a7914d1752a70535cabe1e2f3ec13f70b3d8bffe669a4b2d2904
Sep 11 23:18:21.777628 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:21.777597    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:18:21.777698 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:21.777663    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:18:21.777698 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:21.777690    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:18:22.309651 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:22.309613    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-92d8ce-openshift-default-5ff8db57b8-8xvzw" event={"ID":"2304922d-c8d4-4a95-a81c-4af2ccea36b0","Type":"ContainerStarted","Data":"bf23cb14ab75188574c3d578b03c4980958fb1aa6d7f7abf440d6a2f07f1bcec"}
Sep 11 23:18:22.310126 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:22.309669    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-92d8ce-openshift-default-5ff8db57b8-8xvzw" event={"ID":"2304922d-c8d4-4a95-a81c-4af2ccea36b0","Type":"ContainerStarted","Data":"c09cd6895a18a7914d1752a70535cabe1e2f3ec13f70b3d8bffe669a4b2d2904"}
Sep 11 23:18:22.311345 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:22.311324    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp" event={"ID":"48592704-31d9-4e6f-bad1-ac862a6dfa72","Type":"ContainerStarted","Data":"037fb0ae3d3ce58a7da3acd0f1d575db3e324cf70b9536808ea6eeff2f2e363e"}
Sep 11 23:18:22.311441 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:22.311366    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp" event={"ID":"48592704-31d9-4e6f-bad1-ac862a6dfa72","Type":"ContainerStarted","Data":"69e2b66a45ce76d3c745c018d4efec12145cb5e4a3a45246511f7736cbdb684a"}
Sep 11 23:18:22.312567 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:22.312550    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp"
Sep 11 23:18:22.317580 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:22.317559    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp"
Sep 11 23:18:22.334731 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:22.334681    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-92d8ce-openshift-default-5ff8db57b8-8xvzw" podStartSLOduration=1.334666229 podStartE2EDuration="1.334666229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 23:18:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:18:22.333384193 +0000 UTC m=+1026.366513413" watchObservedRunningTime="2026-09-11 23:18:22.334666229 +0000 UTC m=+1026.367795445"
Sep 11 23:18:22.362169 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:22.362109    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp" podStartSLOduration=2.36209476 podStartE2EDuration="2.36209476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 23:18:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:18:22.360597239 +0000 UTC m=+1026.393726448" watchObservedRunningTime="2026-09-11 23:18:22.36209476 +0000 UTC m=+1026.395223977"
Sep 11 23:18:22.481590 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:22.481544    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-92d8ce-openshift-default-5ff8db57b8-8xvzw"
Sep 11 23:18:22.497918 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:22.497892    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-92d8ce-openshift-default-5ff8db57b8-8xvzw"
Sep 11 23:18:23.318287 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:23.318254    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp"
Sep 11 23:18:23.318841 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:23.318320    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-92d8ce-openshift-default-5ff8db57b8-8xvzw"
Sep 11 23:18:23.319169 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:23.319129    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp"
Sep 11 23:18:23.319270 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:23.319222    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-92d8ce-openshift-default-5ff8db57b8-8xvzw"
Sep 11 23:18:24.712596 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:24.712565    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"]
Sep 11 23:18:29.279636 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:29.279607    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-a0c7930e-openshift-default-865d96c648-r2kng"]
Sep 11 23:18:29.284456 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:29.284436    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a0c7930e-openshift-default-865d96c648-r2kng"
Sep 11 23:18:29.287741 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:29.287720    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-a0c7930e-openshift-default-dockercfg-dxsdw\""
Sep 11 23:18:29.303428 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:29.303405    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a0c7930e-openshift-default-865d96c648-r2kng"]
Sep 11 23:18:29.430510 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:29.430474    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd91316e-27f3-4f22-81cd-59e6479cf614-istiod-ca-cert\") pod \"e2e-ait-a0c7930e-openshift-default-865d96c648-r2kng\" (UID: \"cd91316e-27f3-4f22-81cd-59e6479cf614\") " pod="openshift-ingress/e2e-ait-a0c7930e-openshift-default-865d96c648-r2kng"
Sep 11 23:18:29.430510 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:29.430513    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd91316e-27f3-4f22-81cd-59e6479cf614-workload-socket\") pod \"e2e-ait-a0c7930e-openshift-default-865d96c648-r2kng\" (UID: \"cd91316e-27f3-4f22-81cd-59e6479cf614\") " pod="openshift-ingress/e2e-ait-a0c7930e-openshift-default-865d96c648-r2kng"
Sep 11 23:18:29.430761 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:29.430572    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42jp8\" (UniqueName: \"kubernetes.io/projected/cd91316e-27f3-4f22-81cd-59e6479cf614-kube-api-access-42jp8\") pod \"e2e-ait-a0c7930e-openshift-default-865d96c648-r2kng\" (UID: \"cd91316e-27f3-4f22-81cd-59e6479cf614\") " pod="openshift-ingress/e2e-ait-a0c7930e-openshift-default-865d96c648-r2kng"
Sep 11 23:18:29.430761 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:29.430633    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd91316e-27f3-4f22-81cd-59e6479cf614-credential-socket\") pod \"e2e-ait-a0c7930e-openshift-default-865d96c648-r2kng\" (UID: \"cd91316e-27f3-4f22-81cd-59e6479cf614\") " pod="openshift-ingress/e2e-ait-a0c7930e-openshift-default-865d96c648-r2kng"
Sep 11 23:18:29.430761 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:29.430685    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd91316e-27f3-4f22-81cd-59e6479cf614-istio-podinfo\") pod \"e2e-ait-a0c7930e-openshift-default-865d96c648-r2kng\" (UID: \"cd91316e-27f3-4f22-81cd-59e6479cf614\") " pod="openshift-ingress/e2e-ait-a0c7930e-openshift-default-865d96c648-r2kng"
Sep 11 23:18:29.430761 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:29.430711    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd91316e-27f3-4f22-81cd-59e6479cf614-workload-certs\") pod \"e2e-ait-a0c7930e-openshift-default-865d96c648-r2kng\" (UID: \"cd91316e-27f3-4f22-81cd-59e6479cf614\") " pod="openshift-ingress/e2e-ait-a0c7930e-openshift-default-865d96c648-r2kng"
Sep 11 23:18:29.430761 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:29.430727    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd91316e-27f3-4f22-81cd-59e6479cf614-istio-envoy\") pod \"e2e-ait-a0c7930e-openshift-default-865d96c648-r2kng\" (UID: \"cd91316e-27f3-4f22-81cd-59e6479cf614\") " pod="openshift-ingress/e2e-ait-a0c7930e-openshift-default-865d96c648-r2kng"
Sep 11 23:18:29.430761 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:29.430752    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd91316e-27f3-4f22-81cd-59e6479cf614-istio-data\") pod \"e2e-ait-a0c7930e-openshift-default-865d96c648-r2kng\" (UID: \"cd91316e-27f3-4f22-81cd-59e6479cf614\") " pod="openshift-ingress/e2e-ait-a0c7930e-openshift-default-865d96c648-r2kng"
Sep 11 23:18:29.430959 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:29.430775    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd91316e-27f3-4f22-81cd-59e6479cf614-istio-token\") pod \"e2e-ait-a0c7930e-openshift-default-865d96c648-r2kng\" (UID: \"cd91316e-27f3-4f22-81cd-59e6479cf614\") " pod="openshift-ingress/e2e-ait-a0c7930e-openshift-default-865d96c648-r2kng"
Sep 11 23:18:29.485672 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:29.485638    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8"]
Sep 11 23:18:29.489491 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:29.489468    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8"
Sep 11 23:18:29.492555 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:29.492533    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-0a7a66-openshift-default-dockercfg-bw7vv\""
Sep 11 23:18:29.501600 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:29.501576    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8"]
Sep 11 23:18:29.532076 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:29.531987    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd91316e-27f3-4f22-81cd-59e6479cf614-istiod-ca-cert\") pod \"e2e-ait-a0c7930e-openshift-default-865d96c648-r2kng\" (UID: \"cd91316e-27f3-4f22-81cd-59e6479cf614\") " pod="openshift-ingress/e2e-ait-a0c7930e-openshift-default-865d96c648-r2kng"
Sep 11 23:18:29.532076 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:29.532036    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd91316e-27f3-4f22-81cd-59e6479cf614-workload-socket\") pod \"e2e-ait-a0c7930e-openshift-default-865d96c648-r2kng\" (UID: \"cd91316e-27f3-4f22-81cd-59e6479cf614\") " pod="openshift-ingress/e2e-ait-a0c7930e-openshift-default-865d96c648-r2kng"
Sep 11 23:18:29.532304 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:29.532124    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42jp8\" (UniqueName: \"kubernetes.io/projected/cd91316e-27f3-4f22-81cd-59e6479cf614-kube-api-access-42jp8\") pod \"e2e-ait-a0c7930e-openshift-default-865d96c648-r2kng\" (UID: \"cd91316e-27f3-4f22-81cd-59e6479cf614\") " pod="openshift-ingress/e2e-ait-a0c7930e-openshift-default-865d96c648-r2kng"
Sep 11 23:18:29.532304 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:29.532185    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd91316e-27f3-4f22-81cd-59e6479cf614-credential-socket\") pod \"e2e-ait-a0c7930e-openshift-default-865d96c648-r2kng\" (UID: \"cd91316e-27f3-4f22-81cd-59e6479cf614\") " pod="openshift-ingress/e2e-ait-a0c7930e-openshift-default-865d96c648-r2kng"
Sep 11 23:18:29.532304 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:29.532238    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd91316e-27f3-4f22-81cd-59e6479cf614-istio-podinfo\") pod \"e2e-ait-a0c7930e-openshift-default-865d96c648-r2kng\" (UID: \"cd91316e-27f3-4f22-81cd-59e6479cf614\") " pod="openshift-ingress/e2e-ait-a0c7930e-openshift-default-865d96c648-r2kng"
Sep 11 23:18:29.532304 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:29.532277    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd91316e-27f3-4f22-81cd-59e6479cf614-workload-certs\") pod \"e2e-ait-a0c7930e-openshift-default-865d96c648-r2kng\" (UID: \"cd91316e-27f3-4f22-81cd-59e6479cf614\") " pod="openshift-ingress/e2e-ait-a0c7930e-openshift-default-865d96c648-r2kng"
Sep 11 23:18:29.532304 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:29.532301    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd91316e-27f3-4f22-81cd-59e6479cf614-istio-envoy\") pod \"e2e-ait-a0c7930e-openshift-default-865d96c648-r2kng\" (UID: \"cd91316e-27f3-4f22-81cd-59e6479cf614\") " pod="openshift-ingress/e2e-ait-a0c7930e-openshift-default-865d96c648-r2kng"
Sep 11 23:18:29.532571 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:29.532341    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd91316e-27f3-4f22-81cd-59e6479cf614-istio-data\") pod \"e2e-ait-a0c7930e-openshift-default-865d96c648-r2kng\" (UID: \"cd91316e-27f3-4f22-81cd-59e6479cf614\") " pod="openshift-ingress/e2e-ait-a0c7930e-openshift-default-865d96c648-r2kng"
Sep 11 23:18:29.532571 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:29.532382    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd91316e-27f3-4f22-81cd-59e6479cf614-istio-token\") pod \"e2e-ait-a0c7930e-openshift-default-865d96c648-r2kng\" (UID: \"cd91316e-27f3-4f22-81cd-59e6479cf614\") " pod="openshift-ingress/e2e-ait-a0c7930e-openshift-default-865d96c648-r2kng"
Sep 11 23:18:29.532726 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:29.532567    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd91316e-27f3-4f22-81cd-59e6479cf614-credential-socket\") pod \"e2e-ait-a0c7930e-openshift-default-865d96c648-r2kng\" (UID: \"cd91316e-27f3-4f22-81cd-59e6479cf614\") " pod="openshift-ingress/e2e-ait-a0c7930e-openshift-default-865d96c648-r2kng"
Sep 11 23:18:29.532843 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:29.532817    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd91316e-27f3-4f22-81cd-59e6479cf614-istio-data\") pod \"e2e-ait-a0c7930e-openshift-default-865d96c648-r2kng\" (UID: \"cd91316e-27f3-4f22-81cd-59e6479cf614\") " pod="openshift-ingress/e2e-ait-a0c7930e-openshift-default-865d96c648-r2kng"
Sep 11 23:18:29.532943 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:29.532856    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd91316e-27f3-4f22-81cd-59e6479cf614-istiod-ca-cert\") pod \"e2e-ait-a0c7930e-openshift-default-865d96c648-r2kng\" (UID: \"cd91316e-27f3-4f22-81cd-59e6479cf614\") " pod="openshift-ingress/e2e-ait-a0c7930e-openshift-default-865d96c648-r2kng"
Sep 11 23:18:29.532943 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:29.532896    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd91316e-27f3-4f22-81cd-59e6479cf614-workload-certs\") pod \"e2e-ait-a0c7930e-openshift-default-865d96c648-r2kng\" (UID: \"cd91316e-27f3-4f22-81cd-59e6479cf614\") " pod="openshift-ingress/e2e-ait-a0c7930e-openshift-default-865d96c648-r2kng"
Sep 11 23:18:29.533249 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:29.533003    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd91316e-27f3-4f22-81cd-59e6479cf614-workload-socket\") pod \"e2e-ait-a0c7930e-openshift-default-865d96c648-r2kng\" (UID: \"cd91316e-27f3-4f22-81cd-59e6479cf614\") " pod="openshift-ingress/e2e-ait-a0c7930e-openshift-default-865d96c648-r2kng"
Sep 11 23:18:29.534826 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:29.534807    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd91316e-27f3-4f22-81cd-59e6479cf614-istio-envoy\") pod \"e2e-ait-a0c7930e-openshift-default-865d96c648-r2kng\" (UID: \"cd91316e-27f3-4f22-81cd-59e6479cf614\") " pod="openshift-ingress/e2e-ait-a0c7930e-openshift-default-865d96c648-r2kng"
Sep 11 23:18:29.534952 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:29.534933    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd91316e-27f3-4f22-81cd-59e6479cf614-istio-podinfo\") pod \"e2e-ait-a0c7930e-openshift-default-865d96c648-r2kng\" (UID: \"cd91316e-27f3-4f22-81cd-59e6479cf614\") " pod="openshift-ingress/e2e-ait-a0c7930e-openshift-default-865d96c648-r2kng"
Sep 11 23:18:29.541015 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:29.540992    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd91316e-27f3-4f22-81cd-59e6479cf614-istio-token\") pod \"e2e-ait-a0c7930e-openshift-default-865d96c648-r2kng\" (UID: \"cd91316e-27f3-4f22-81cd-59e6479cf614\") " pod="openshift-ingress/e2e-ait-a0c7930e-openshift-default-865d96c648-r2kng"
Sep 11 23:18:29.541629 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:29.541610    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42jp8\" (UniqueName: \"kubernetes.io/projected/cd91316e-27f3-4f22-81cd-59e6479cf614-kube-api-access-42jp8\") pod \"e2e-ait-a0c7930e-openshift-default-865d96c648-r2kng\" (UID: \"cd91316e-27f3-4f22-81cd-59e6479cf614\") " pod="openshift-ingress/e2e-ait-a0c7930e-openshift-default-865d96c648-r2kng"
Sep 11 23:18:29.595868 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:29.595812    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a0c7930e-openshift-default-865d96c648-r2kng"
Sep 11 23:18:29.632951 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:29.632908    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad22eafb-8cf0-41a7-9286-ee3d645405c4-credential-socket\") pod \"e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8\" (UID: \"ad22eafb-8cf0-41a7-9286-ee3d645405c4\") " pod="openshift-ingress/e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8"
Sep 11 23:18:29.632951 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:29.632953    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad22eafb-8cf0-41a7-9286-ee3d645405c4-istiod-ca-cert\") pod \"e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8\" (UID: \"ad22eafb-8cf0-41a7-9286-ee3d645405c4\") " pod="openshift-ingress/e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8"
Sep 11 23:18:29.633212 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:29.633047    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad22eafb-8cf0-41a7-9286-ee3d645405c4-workload-certs\") pod \"e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8\" (UID: \"ad22eafb-8cf0-41a7-9286-ee3d645405c4\") " pod="openshift-ingress/e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8"
Sep 11 23:18:29.633212 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:29.633097    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad22eafb-8cf0-41a7-9286-ee3d645405c4-workload-socket\") pod \"e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8\" (UID: \"ad22eafb-8cf0-41a7-9286-ee3d645405c4\") " pod="openshift-ingress/e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8"
Sep 11 23:18:29.633283 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:29.633220    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad22eafb-8cf0-41a7-9286-ee3d645405c4-istio-data\") pod \"e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8\" (UID: \"ad22eafb-8cf0-41a7-9286-ee3d645405c4\") " pod="openshift-ingress/e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8"
Sep 11 23:18:29.633611 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:29.633569    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad22eafb-8cf0-41a7-9286-ee3d645405c4-istio-podinfo\") pod \"e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8\" (UID: \"ad22eafb-8cf0-41a7-9286-ee3d645405c4\") " pod="openshift-ingress/e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8"
Sep 11 23:18:29.633687 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:29.633636    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54bwf\" (UniqueName: \"kubernetes.io/projected/ad22eafb-8cf0-41a7-9286-ee3d645405c4-kube-api-access-54bwf\") pod \"e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8\" (UID: \"ad22eafb-8cf0-41a7-9286-ee3d645405c4\") " pod="openshift-ingress/e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8"
Sep 11 23:18:29.633725 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:29.633693    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad22eafb-8cf0-41a7-9286-ee3d645405c4-istio-envoy\") pod \"e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8\" (UID: \"ad22eafb-8cf0-41a7-9286-ee3d645405c4\") " pod="openshift-ingress/e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8"
Sep 11 23:18:29.633761 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:29.633732    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad22eafb-8cf0-41a7-9286-ee3d645405c4-istio-token\") pod \"e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8\" (UID: \"ad22eafb-8cf0-41a7-9286-ee3d645405c4\") " pod="openshift-ingress/e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8"
Sep 11 23:18:29.735341 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:29.735306    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad22eafb-8cf0-41a7-9286-ee3d645405c4-istio-podinfo\") pod \"e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8\" (UID: \"ad22eafb-8cf0-41a7-9286-ee3d645405c4\") " pod="openshift-ingress/e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8"
Sep 11 23:18:29.735341 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:29.735345    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54bwf\" (UniqueName: \"kubernetes.io/projected/ad22eafb-8cf0-41a7-9286-ee3d645405c4-kube-api-access-54bwf\") pod \"e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8\" (UID: \"ad22eafb-8cf0-41a7-9286-ee3d645405c4\") " pod="openshift-ingress/e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8"
Sep 11 23:18:29.735580 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:29.735370    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad22eafb-8cf0-41a7-9286-ee3d645405c4-istio-envoy\") pod \"e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8\" (UID: \"ad22eafb-8cf0-41a7-9286-ee3d645405c4\") " pod="openshift-ingress/e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8"
Sep 11 23:18:29.735580 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:29.735385    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad22eafb-8cf0-41a7-9286-ee3d645405c4-istio-token\") pod \"e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8\" (UID: \"ad22eafb-8cf0-41a7-9286-ee3d645405c4\") " pod="openshift-ingress/e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8"
Sep 11 23:18:29.735580 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:29.735408    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad22eafb-8cf0-41a7-9286-ee3d645405c4-credential-socket\") pod \"e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8\" (UID: \"ad22eafb-8cf0-41a7-9286-ee3d645405c4\") " pod="openshift-ingress/e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8"
Sep 11 23:18:29.735580 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:29.735442    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad22eafb-8cf0-41a7-9286-ee3d645405c4-istiod-ca-cert\") pod \"e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8\" (UID: \"ad22eafb-8cf0-41a7-9286-ee3d645405c4\") " pod="openshift-ingress/e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8"
Sep 11 23:18:29.735580 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:29.735478    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad22eafb-8cf0-41a7-9286-ee3d645405c4-workload-certs\") pod \"e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8\" (UID: \"ad22eafb-8cf0-41a7-9286-ee3d645405c4\") " pod="openshift-ingress/e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8"
Sep 11 23:18:29.735580 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:29.735509    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad22eafb-8cf0-41a7-9286-ee3d645405c4-workload-socket\") pod \"e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8\" (UID: \"ad22eafb-8cf0-41a7-9286-ee3d645405c4\") " pod="openshift-ingress/e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8"
Sep 11 23:18:29.735580 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:29.735560    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad22eafb-8cf0-41a7-9286-ee3d645405c4-istio-data\") pod \"e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8\" (UID: \"ad22eafb-8cf0-41a7-9286-ee3d645405c4\") " pod="openshift-ingress/e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8"
Sep 11 23:18:29.735962 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:29.735883    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad22eafb-8cf0-41a7-9286-ee3d645405c4-credential-socket\") pod \"e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8\" (UID: \"ad22eafb-8cf0-41a7-9286-ee3d645405c4\") " pod="openshift-ingress/e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8"
Sep 11 23:18:29.736091 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:29.736062    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad22eafb-8cf0-41a7-9286-ee3d645405c4-workload-certs\") pod \"e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8\" (UID: \"ad22eafb-8cf0-41a7-9286-ee3d645405c4\") " pod="openshift-ingress/e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8"
Sep 11 23:18:29.736347 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:29.736132    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad22eafb-8cf0-41a7-9286-ee3d645405c4-workload-socket\") pod \"e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8\" (UID: \"ad22eafb-8cf0-41a7-9286-ee3d645405c4\") " pod="openshift-ingress/e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8"
Sep 11 23:18:29.736347 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:29.736279    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad22eafb-8cf0-41a7-9286-ee3d645405c4-istio-data\") pod \"e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8\" (UID: \"ad22eafb-8cf0-41a7-9286-ee3d645405c4\") " pod="openshift-ingress/e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8"
Sep 11 23:18:29.736471 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:29.736411    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad22eafb-8cf0-41a7-9286-ee3d645405c4-istiod-ca-cert\") pod \"e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8\" (UID: \"ad22eafb-8cf0-41a7-9286-ee3d645405c4\") " pod="openshift-ingress/e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8"
Sep 11 23:18:29.738005 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:29.737980    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad22eafb-8cf0-41a7-9286-ee3d645405c4-istio-envoy\") pod \"e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8\" (UID: \"ad22eafb-8cf0-41a7-9286-ee3d645405c4\") " pod="openshift-ingress/e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8"
Sep 11 23:18:29.738119 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:29.738105    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad22eafb-8cf0-41a7-9286-ee3d645405c4-istio-podinfo\") pod \"e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8\" (UID: \"ad22eafb-8cf0-41a7-9286-ee3d645405c4\") " pod="openshift-ingress/e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8"
Sep 11 23:18:29.743551 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:29.743524    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad22eafb-8cf0-41a7-9286-ee3d645405c4-istio-token\") pod \"e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8\" (UID: \"ad22eafb-8cf0-41a7-9286-ee3d645405c4\") " pod="openshift-ingress/e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8"
Sep 11 23:18:29.743674 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:29.743657    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54bwf\" (UniqueName: \"kubernetes.io/projected/ad22eafb-8cf0-41a7-9286-ee3d645405c4-kube-api-access-54bwf\") pod \"e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8\" (UID: \"ad22eafb-8cf0-41a7-9286-ee3d645405c4\") " pod="openshift-ingress/e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8"
Sep 11 23:18:29.800778 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:29.800694    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8"
Sep 11 23:18:29.884373 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:29.884338    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a0c7930e-openshift-default-865d96c648-r2kng"]
Sep 11 23:18:29.886662 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:18:29.886630    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd91316e_27f3_4f22_81cd_59e6479cf614.slice/crio-19751ccd1f1e11ad5430613005cff16df08fe0a52b50b6963c57d9966d3765b0 WatchSource:0}: Error finding container 19751ccd1f1e11ad5430613005cff16df08fe0a52b50b6963c57d9966d3765b0: Status 404 returned error can't find the container with id 19751ccd1f1e11ad5430613005cff16df08fe0a52b50b6963c57d9966d3765b0
Sep 11 23:18:29.889417 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:29.889380    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:18:29.889500 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:29.889466    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:18:29.889541 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:29.889510    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:18:30.086585 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:30.086553    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8"]
Sep 11 23:18:30.088075 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:18:30.088051    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad22eafb_8cf0_41a7_9286_ee3d645405c4.slice/crio-7f5c0c3f0404ecbcb01dab5db7f7a40e9c7498a764e6ae512c6e4b15c122569d WatchSource:0}: Error finding container 7f5c0c3f0404ecbcb01dab5db7f7a40e9c7498a764e6ae512c6e4b15c122569d: Status 404 returned error can't find the container with id 7f5c0c3f0404ecbcb01dab5db7f7a40e9c7498a764e6ae512c6e4b15c122569d
Sep 11 23:18:30.090954 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:30.090923    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:18:30.091042 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:30.091002    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:18:30.091042 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:30.091029    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:18:30.372096 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:30.371992    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8" event={"ID":"ad22eafb-8cf0-41a7-9286-ee3d645405c4","Type":"ContainerStarted","Data":"a7ee56d8e50965ff0d1d6f5cd7f0856129ced129f7ad9a59d2372bee656adffe"}
Sep 11 23:18:30.372096 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:30.372059    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8" event={"ID":"ad22eafb-8cf0-41a7-9286-ee3d645405c4","Type":"ContainerStarted","Data":"7f5c0c3f0404ecbcb01dab5db7f7a40e9c7498a764e6ae512c6e4b15c122569d"}
Sep 11 23:18:30.374217 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:30.374184    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a0c7930e-openshift-default-865d96c648-r2kng" event={"ID":"cd91316e-27f3-4f22-81cd-59e6479cf614","Type":"ContainerStarted","Data":"c1cd33ea46d6fec9c54b50c7d85f7705c401cae46b4ed14d45b76e9ebb960f62"}
Sep 11 23:18:30.374359 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:30.374247    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a0c7930e-openshift-default-865d96c648-r2kng" event={"ID":"cd91316e-27f3-4f22-81cd-59e6479cf614","Type":"ContainerStarted","Data":"19751ccd1f1e11ad5430613005cff16df08fe0a52b50b6963c57d9966d3765b0"}
Sep 11 23:18:30.416177 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:30.416089    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8" podStartSLOduration=1.416070133 podStartE2EDuration="1.416070133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 23:18:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:18:30.396147916 +0000 UTC m=+1034.429277134" watchObservedRunningTime="2026-09-11 23:18:30.416070133 +0000 UTC m=+1034.449199353"
Sep 11 23:18:30.418055 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:30.418018    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-a0c7930e-openshift-default-865d96c648-r2kng" podStartSLOduration=1.418005819 podStartE2EDuration="1.418005819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 23:18:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:18:30.414977567 +0000 UTC m=+1034.448106783" watchObservedRunningTime="2026-09-11 23:18:30.418005819 +0000 UTC m=+1034.451135035"
Sep 11 23:18:30.596075 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:30.596034    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a0c7930e-openshift-default-865d96c648-r2kng"
Sep 11 23:18:30.601462 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:30.601432    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a0c7930e-openshift-default-865d96c648-r2kng"
Sep 11 23:18:30.801264 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:30.801235    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8"
Sep 11 23:18:31.380077 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:31.380041    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a0c7930e-openshift-default-865d96c648-r2kng"
Sep 11 23:18:31.381188 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:31.381167    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a0c7930e-openshift-default-865d96c648-r2kng"
Sep 11 23:18:31.807416 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:31.807384    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8"
Sep 11 23:18:32.387055 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:32.387020    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8"
Sep 11 23:18:32.387906 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:32.387888    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8"
Sep 11 23:18:41.373863 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:41.373817    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-06ef79-77bfb74cfd-5mr6z"]
Sep 11 23:18:41.377769 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:41.377739    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-06ef79-77bfb74cfd-5mr6z"
Sep 11 23:18:41.381264 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:41.381229    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-06ef79-dockercfg-hw82d\""
Sep 11 23:18:41.381530 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:41.381513    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-06ef79\""
Sep 11 23:18:41.399841 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:41.399813    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-06ef79-77bfb74cfd-5mr6z"]
Sep 11 23:18:41.467424 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:41.467373    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jc79d\" (UniqueName: \"kubernetes.io/projected/2f226bf5-fc00-493d-92c2-0c59e9b33dd1-kube-api-access-jc79d\") pod \"payload-pre-processing-e2e-worker-w4-06ef79-77bfb74cfd-5mr6z\" (UID: \"2f226bf5-fc00-493d-92c2-0c59e9b33dd1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-06ef79-77bfb74cfd-5mr6z"
Sep 11 23:18:41.467615 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:41.467484    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f226bf5-fc00-493d-92c2-0c59e9b33dd1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-06ef79-77bfb74cfd-5mr6z\" (UID: \"2f226bf5-fc00-493d-92c2-0c59e9b33dd1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-06ef79-77bfb74cfd-5mr6z"
Sep 11 23:18:41.568402 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:41.568366    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc79d\" (UniqueName: \"kubernetes.io/projected/2f226bf5-fc00-493d-92c2-0c59e9b33dd1-kube-api-access-jc79d\") pod \"payload-pre-processing-e2e-worker-w4-06ef79-77bfb74cfd-5mr6z\" (UID: \"2f226bf5-fc00-493d-92c2-0c59e9b33dd1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-06ef79-77bfb74cfd-5mr6z"
Sep 11 23:18:41.568597 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:41.568439    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f226bf5-fc00-493d-92c2-0c59e9b33dd1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-06ef79-77bfb74cfd-5mr6z\" (UID: \"2f226bf5-fc00-493d-92c2-0c59e9b33dd1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-06ef79-77bfb74cfd-5mr6z"
Sep 11 23:18:41.569190 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:41.569167    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f226bf5-fc00-493d-92c2-0c59e9b33dd1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-06ef79-77bfb74cfd-5mr6z\" (UID: \"2f226bf5-fc00-493d-92c2-0c59e9b33dd1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-06ef79-77bfb74cfd-5mr6z"
Sep 11 23:18:41.578609 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:41.578584    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc79d\" (UniqueName: \"kubernetes.io/projected/2f226bf5-fc00-493d-92c2-0c59e9b33dd1-kube-api-access-jc79d\") pod \"payload-pre-processing-e2e-worker-w4-06ef79-77bfb74cfd-5mr6z\" (UID: \"2f226bf5-fc00-493d-92c2-0c59e9b33dd1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-06ef79-77bfb74cfd-5mr6z"
Sep 11 23:18:41.690899 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:41.690844    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-06ef79-77bfb74cfd-5mr6z"
Sep 11 23:18:41.986411 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:41.983037    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-06ef79-77bfb74cfd-5mr6z"]
Sep 11 23:18:41.989521 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:41.989498    2546 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 23:18:42.461503 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:42.461461    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-06ef79-77bfb74cfd-5mr6z" event={"ID":"2f226bf5-fc00-493d-92c2-0c59e9b33dd1","Type":"ContainerStarted","Data":"32ace1bbefbe65c2ed545b9ef4a38f7c45b8c5c0c1ec4779f387d9d4810ee63e"}
Sep 11 23:18:44.477334 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:44.477230    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-06ef79-77bfb74cfd-5mr6z" event={"ID":"2f226bf5-fc00-493d-92c2-0c59e9b33dd1","Type":"ContainerStarted","Data":"c9f9ba0be686326d252ae9229c4ee4b263eb57c7a201a53ded15a0fe9cc114d9"}
Sep 11 23:18:44.477334 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:44.477298    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-06ef79-77bfb74cfd-5mr6z"
Sep 11 23:18:44.478810 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:44.478787    2546 generic.go:356] "Generic (PLEG): container finished" podID="48592704-31d9-4e6f-bad1-ac862a6dfa72" containerID="037fb0ae3d3ce58a7da3acd0f1d575db3e324cf70b9536808ea6eeff2f2e363e" exitCode=0
Sep 11 23:18:44.478921 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:44.478854    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp" event={"ID":"48592704-31d9-4e6f-bad1-ac862a6dfa72","Type":"ContainerDied","Data":"037fb0ae3d3ce58a7da3acd0f1d575db3e324cf70b9536808ea6eeff2f2e363e"}
Sep 11 23:18:44.479266 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:44.479249    2546 scope.go:117] "RemoveContainer" containerID="037fb0ae3d3ce58a7da3acd0f1d575db3e324cf70b9536808ea6eeff2f2e363e"
Sep 11 23:18:44.480371 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:44.480343    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:18:44.480475 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:44.480428    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:18:44.480542 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:44.480483    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:18:44.480542 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:44.480514    2546 generic.go:356] "Generic (PLEG): container finished" podID="2304922d-c8d4-4a95-a81c-4af2ccea36b0" containerID="bf23cb14ab75188574c3d578b03c4980958fb1aa6d7f7abf440d6a2f07f1bcec" exitCode=0
Sep 11 23:18:44.480647 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:44.480585    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-92d8ce-openshift-default-5ff8db57b8-8xvzw" event={"ID":"2304922d-c8d4-4a95-a81c-4af2ccea36b0","Type":"ContainerDied","Data":"bf23cb14ab75188574c3d578b03c4980958fb1aa6d7f7abf440d6a2f07f1bcec"}
Sep 11 23:18:44.480910 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:44.480887    2546 scope.go:117] "RemoveContainer" containerID="bf23cb14ab75188574c3d578b03c4980958fb1aa6d7f7abf440d6a2f07f1bcec"
Sep 11 23:18:44.482039 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:44.482012    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:18:44.482121 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:44.482067    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:18:44.482121 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:44.482101    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:18:44.501751 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:44.501704    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-06ef79-77bfb74cfd-5mr6z" podStartSLOduration=1.926424642 podStartE2EDuration="3.501686557s" totalImagesPullingTime="1.575261915s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 23:18:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:18:44.497938198 +0000 UTC m=+1048.531067416" watchObservedRunningTime="2026-09-11 23:18:44.501686557 +0000 UTC m=+1048.534815777"
Sep 11 23:18:44.742255 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:44.739537    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq"]
Sep 11 23:18:44.745533 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:44.744466    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq"
Sep 11 23:18:44.749129 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:44.748954    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-bfa694-openshift-default-dockercfg-nqpcw\""
Sep 11 23:18:44.764661 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:44.764639    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq"]
Sep 11 23:18:44.807703 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:44.807669    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcd4c593-5a49-4cdc-bd1a-27c49cbe594c-istio-token\") pod \"e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq\" (UID: \"dcd4c593-5a49-4cdc-bd1a-27c49cbe594c\") " pod="openshift-ingress/e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq"
Sep 11 23:18:44.807859 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:44.807734    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcd4c593-5a49-4cdc-bd1a-27c49cbe594c-istio-data\") pod \"e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq\" (UID: \"dcd4c593-5a49-4cdc-bd1a-27c49cbe594c\") " pod="openshift-ingress/e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq"
Sep 11 23:18:44.807859 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:44.807756    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcd4c593-5a49-4cdc-bd1a-27c49cbe594c-istio-podinfo\") pod \"e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq\" (UID: \"dcd4c593-5a49-4cdc-bd1a-27c49cbe594c\") " pod="openshift-ingress/e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq"
Sep 11 23:18:44.807859 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:44.807792    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcd4c593-5a49-4cdc-bd1a-27c49cbe594c-istio-envoy\") pod \"e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq\" (UID: \"dcd4c593-5a49-4cdc-bd1a-27c49cbe594c\") " pod="openshift-ingress/e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq"
Sep 11 23:18:44.807859 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:44.807825    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcd4c593-5a49-4cdc-bd1a-27c49cbe594c-workload-socket\") pod \"e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq\" (UID: \"dcd4c593-5a49-4cdc-bd1a-27c49cbe594c\") " pod="openshift-ingress/e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq"
Sep 11 23:18:44.808039 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:44.807893    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcd4c593-5a49-4cdc-bd1a-27c49cbe594c-istiod-ca-cert\") pod \"e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq\" (UID: \"dcd4c593-5a49-4cdc-bd1a-27c49cbe594c\") " pod="openshift-ingress/e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq"
Sep 11 23:18:44.808039 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:44.807927    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcd4c593-5a49-4cdc-bd1a-27c49cbe594c-credential-socket\") pod \"e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq\" (UID: \"dcd4c593-5a49-4cdc-bd1a-27c49cbe594c\") " pod="openshift-ingress/e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq"
Sep 11 23:18:44.808039 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:44.807942    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwbk2\" (UniqueName: \"kubernetes.io/projected/dcd4c593-5a49-4cdc-bd1a-27c49cbe594c-kube-api-access-zwbk2\") pod \"e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq\" (UID: \"dcd4c593-5a49-4cdc-bd1a-27c49cbe594c\") " pod="openshift-ingress/e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq"
Sep 11 23:18:44.808039 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:44.807981    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcd4c593-5a49-4cdc-bd1a-27c49cbe594c-workload-certs\") pod \"e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq\" (UID: \"dcd4c593-5a49-4cdc-bd1a-27c49cbe594c\") " pod="openshift-ingress/e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq"
Sep 11 23:18:44.909021 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:44.908984    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcd4c593-5a49-4cdc-bd1a-27c49cbe594c-credential-socket\") pod \"e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq\" (UID: \"dcd4c593-5a49-4cdc-bd1a-27c49cbe594c\") " pod="openshift-ingress/e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq"
Sep 11 23:18:44.909355 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:44.909315    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwbk2\" (UniqueName: \"kubernetes.io/projected/dcd4c593-5a49-4cdc-bd1a-27c49cbe594c-kube-api-access-zwbk2\") pod \"e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq\" (UID: \"dcd4c593-5a49-4cdc-bd1a-27c49cbe594c\") " pod="openshift-ingress/e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq"
Sep 11 23:18:44.909582 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:44.909550    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcd4c593-5a49-4cdc-bd1a-27c49cbe594c-workload-certs\") pod \"e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq\" (UID: \"dcd4c593-5a49-4cdc-bd1a-27c49cbe594c\") " pod="openshift-ingress/e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq"
Sep 11 23:18:44.909825 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:44.909805    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcd4c593-5a49-4cdc-bd1a-27c49cbe594c-istio-token\") pod \"e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq\" (UID: \"dcd4c593-5a49-4cdc-bd1a-27c49cbe594c\") " pod="openshift-ingress/e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq"
Sep 11 23:18:44.910298 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:44.910270    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcd4c593-5a49-4cdc-bd1a-27c49cbe594c-credential-socket\") pod \"e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq\" (UID: \"dcd4c593-5a49-4cdc-bd1a-27c49cbe594c\") " pod="openshift-ingress/e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq"
Sep 11 23:18:44.910417 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:44.910052    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcd4c593-5a49-4cdc-bd1a-27c49cbe594c-workload-certs\") pod \"e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq\" (UID: \"dcd4c593-5a49-4cdc-bd1a-27c49cbe594c\") " pod="openshift-ingress/e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq"
Sep 11 23:18:44.910606 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:44.910583    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcd4c593-5a49-4cdc-bd1a-27c49cbe594c-istio-data\") pod \"e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq\" (UID: \"dcd4c593-5a49-4cdc-bd1a-27c49cbe594c\") " pod="openshift-ingress/e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq"
Sep 11 23:18:44.911094 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:44.911073    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcd4c593-5a49-4cdc-bd1a-27c49cbe594c-istio-podinfo\") pod \"e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq\" (UID: \"dcd4c593-5a49-4cdc-bd1a-27c49cbe594c\") " pod="openshift-ingress/e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq"
Sep 11 23:18:44.911702 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:44.911666    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcd4c593-5a49-4cdc-bd1a-27c49cbe594c-istio-envoy\") pod \"e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq\" (UID: \"dcd4c593-5a49-4cdc-bd1a-27c49cbe594c\") " pod="openshift-ingress/e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq"
Sep 11 23:18:44.911866 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:44.911705    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcd4c593-5a49-4cdc-bd1a-27c49cbe594c-workload-socket\") pod \"e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq\" (UID: \"dcd4c593-5a49-4cdc-bd1a-27c49cbe594c\") " pod="openshift-ingress/e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq"
Sep 11 23:18:44.911866 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:44.911017    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcd4c593-5a49-4cdc-bd1a-27c49cbe594c-istio-data\") pod \"e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq\" (UID: \"dcd4c593-5a49-4cdc-bd1a-27c49cbe594c\") " pod="openshift-ingress/e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq"
Sep 11 23:18:44.911866 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:44.911799    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcd4c593-5a49-4cdc-bd1a-27c49cbe594c-istiod-ca-cert\") pod \"e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq\" (UID: \"dcd4c593-5a49-4cdc-bd1a-27c49cbe594c\") " pod="openshift-ingress/e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq"
Sep 11 23:18:44.912395 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:44.912369    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcd4c593-5a49-4cdc-bd1a-27c49cbe594c-workload-socket\") pod \"e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq\" (UID: \"dcd4c593-5a49-4cdc-bd1a-27c49cbe594c\") " pod="openshift-ingress/e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq"
Sep 11 23:18:44.912616 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:44.912569    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcd4c593-5a49-4cdc-bd1a-27c49cbe594c-istiod-ca-cert\") pod \"e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq\" (UID: \"dcd4c593-5a49-4cdc-bd1a-27c49cbe594c\") " pod="openshift-ingress/e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq"
Sep 11 23:18:44.915275 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:44.915233    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcd4c593-5a49-4cdc-bd1a-27c49cbe594c-istio-podinfo\") pod \"e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq\" (UID: \"dcd4c593-5a49-4cdc-bd1a-27c49cbe594c\") " pod="openshift-ingress/e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq"
Sep 11 23:18:44.916226 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:44.916110    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcd4c593-5a49-4cdc-bd1a-27c49cbe594c-istio-envoy\") pod \"e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq\" (UID: \"dcd4c593-5a49-4cdc-bd1a-27c49cbe594c\") " pod="openshift-ingress/e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq"
Sep 11 23:18:44.922429 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:44.922387    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcd4c593-5a49-4cdc-bd1a-27c49cbe594c-istio-token\") pod \"e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq\" (UID: \"dcd4c593-5a49-4cdc-bd1a-27c49cbe594c\") " pod="openshift-ingress/e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq"
Sep 11 23:18:44.922931 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:44.922910    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwbk2\" (UniqueName: \"kubernetes.io/projected/dcd4c593-5a49-4cdc-bd1a-27c49cbe594c-kube-api-access-zwbk2\") pod \"e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq\" (UID: \"dcd4c593-5a49-4cdc-bd1a-27c49cbe594c\") " pod="openshift-ingress/e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq"
Sep 11 23:18:45.064937 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:45.064838    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq"
Sep 11 23:18:45.368035 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:45.368007    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq"]
Sep 11 23:18:45.370855 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:18:45.370822    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcd4c593_5a49_4cdc_bd1a_27c49cbe594c.slice/crio-39fadfa9acc6c4867d82dfb77e6408892e5eddb41fe0c5387c7bb4d2f632c07f WatchSource:0}: Error finding container 39fadfa9acc6c4867d82dfb77e6408892e5eddb41fe0c5387c7bb4d2f632c07f: Status 404 returned error can't find the container with id 39fadfa9acc6c4867d82dfb77e6408892e5eddb41fe0c5387c7bb4d2f632c07f
Sep 11 23:18:45.373590 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:45.373555    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:18:45.373695 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:45.373634    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:18:45.373758 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:45.373690    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:18:45.404502 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:45.403714    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"]
Sep 11 23:18:45.493752 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:45.493710    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp" event={"ID":"48592704-31d9-4e6f-bad1-ac862a6dfa72","Type":"ContainerStarted","Data":"554d358cebbba92d7feceaf69fe2551433117579b9ebcb71c2eafd353fb28af4"}
Sep 11 23:18:45.498223 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:45.498189    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq" event={"ID":"dcd4c593-5a49-4cdc-bd1a-27c49cbe594c","Type":"ContainerStarted","Data":"39fadfa9acc6c4867d82dfb77e6408892e5eddb41fe0c5387c7bb4d2f632c07f"}
Sep 11 23:18:45.501642 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:45.501318    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-92d8ce-openshift-default-5ff8db57b8-8xvzw" event={"ID":"2304922d-c8d4-4a95-a81c-4af2ccea36b0","Type":"ContainerStarted","Data":"fbf830e5fb9f4b6ad31034e7f9139fe54af5726c6b7643246f5e6026041dc61e"}
Sep 11 23:18:46.312565 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:46.312531    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp"
Sep 11 23:18:46.312725 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:46.312644    2546 patch_prober.go:28] interesting pod/e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Sep 11 23:18:46.312725 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:46.312708    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp" podUID="48592704-31d9-4e6f-bad1-ac862a6dfa72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Sep 11 23:18:46.481819 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:46.481781    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-92d8ce-openshift-default-5ff8db57b8-8xvzw"
Sep 11 23:18:46.482019 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:46.481915    2546 patch_prober.go:28] interesting pod/e2e-inf-a-92d8ce-openshift-default-5ff8db57b8-8xvzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Sep 11 23:18:46.482019 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:46.481995    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-92d8ce-openshift-default-5ff8db57b8-8xvzw" podUID="2304922d-c8d4-4a95-a81c-4af2ccea36b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Sep 11 23:18:46.513918 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:46.513881    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq" event={"ID":"dcd4c593-5a49-4cdc-bd1a-27c49cbe594c","Type":"ContainerStarted","Data":"b8252117d86680331cf009c4a07735c80a2714ef52eccd148c2cab7e5aae94e7"}
Sep 11 23:18:46.543537 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:46.543473    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq" podStartSLOduration=2.543451562 podStartE2EDuration="2.543451562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 23:18:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:18:46.537147846 +0000 UTC m=+1050.570277065" watchObservedRunningTime="2026-09-11 23:18:46.543451562 +0000 UTC m=+1050.576580780"
Sep 11 23:18:47.066021 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:47.065423    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq"
Sep 11 23:18:47.075541 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:47.075495    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq"
Sep 11 23:18:47.312813 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:47.312786    2546 patch_prober.go:28] interesting pod/e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Sep 11 23:18:47.313060 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:47.313039    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp" podUID="48592704-31d9-4e6f-bad1-ac862a6dfa72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Sep 11 23:18:47.481975 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:47.481923    2546 patch_prober.go:28] interesting pod/e2e-inf-a-92d8ce-openshift-default-5ff8db57b8-8xvzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Sep 11 23:18:47.482310 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:47.481988    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-92d8ce-openshift-default-5ff8db57b8-8xvzw" podUID="2304922d-c8d4-4a95-a81c-4af2ccea36b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Sep 11 23:18:47.518865 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:47.518833    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq"
Sep 11 23:18:47.519924 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:47.519906    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq"
Sep 11 23:18:48.312757 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:48.312720    2546 patch_prober.go:28] interesting pod/e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Sep 11 23:18:48.312978 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:48.312800    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp" podUID="48592704-31d9-4e6f-bad1-ac862a6dfa72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Sep 11 23:18:48.482323 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:48.482280    2546 patch_prober.go:28] interesting pod/e2e-inf-a-92d8ce-openshift-default-5ff8db57b8-8xvzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Sep 11 23:18:48.482612 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:48.482583    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-92d8ce-openshift-default-5ff8db57b8-8xvzw" podUID="2304922d-c8d4-4a95-a81c-4af2ccea36b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Sep 11 23:18:49.312619 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:49.312579    2546 patch_prober.go:28] interesting pod/e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Sep 11 23:18:49.313026 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:49.312653    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp" podUID="48592704-31d9-4e6f-bad1-ac862a6dfa72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Sep 11 23:18:49.482176 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:49.482114    2546 patch_prober.go:28] interesting pod/e2e-inf-a-92d8ce-openshift-default-5ff8db57b8-8xvzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Sep 11 23:18:49.482487 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:49.482458    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-92d8ce-openshift-default-5ff8db57b8-8xvzw" podUID="2304922d-c8d4-4a95-a81c-4af2ccea36b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Sep 11 23:18:50.313142 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:50.313095    2546 patch_prober.go:28] interesting pod/e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Sep 11 23:18:50.313810 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:50.313772    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp" podUID="48592704-31d9-4e6f-bad1-ac862a6dfa72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Sep 11 23:18:50.482975 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:50.482926    2546 patch_prober.go:28] interesting pod/e2e-inf-a-92d8ce-openshift-default-5ff8db57b8-8xvzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Sep 11 23:18:50.483216 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:50.483014    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-92d8ce-openshift-default-5ff8db57b8-8xvzw" podUID="2304922d-c8d4-4a95-a81c-4af2ccea36b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Sep 11 23:18:51.312752 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:51.312589    2546 patch_prober.go:28] interesting pod/e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Sep 11 23:18:51.312752 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:51.312659    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp" podUID="48592704-31d9-4e6f-bad1-ac862a6dfa72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Sep 11 23:18:51.664181 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:51.662946    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-bfa694-7679c8678b-zzpqs"]
Sep 11 23:18:51.672016 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:51.671990    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bfa694-7679c8678b-zzpqs"
Sep 11 23:18:51.678893 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:51.678849    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-bfa694-7679c8678b-zzpqs"]
Sep 11 23:18:51.679936 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:51.679253    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-bfa694\""
Sep 11 23:18:51.680291 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:51.679852    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-bfa694-dockercfg-hpklv\""
Sep 11 23:18:51.691448 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:51.691422    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-92d8ce-openshift-default-5ff8db57b8-8xvzw"
Sep 11 23:18:51.692728 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:51.692709    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-92d8ce-openshift-default-5ff8db57b8-8xvzw"
Sep 11 23:18:51.710834 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:51.710792    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-92d8ce-openshift-default-5ff8db57b8-8xvzw"
Sep 11 23:18:51.731578 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:51.731546    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe27a86b-83cf-4afa-a295-f832ded073a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-bfa694-7679c8678b-zzpqs\" (UID: \"fe27a86b-83cf-4afa-a295-f832ded073a0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bfa694-7679c8678b-zzpqs"
Sep 11 23:18:51.732187 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:51.732140    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq9vk\" (UniqueName: \"kubernetes.io/projected/fe27a86b-83cf-4afa-a295-f832ded073a0-kube-api-access-dq9vk\") pod \"payload-pre-processing-e2e-inf-b-bfa694-7679c8678b-zzpqs\" (UID: \"fe27a86b-83cf-4afa-a295-f832ded073a0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bfa694-7679c8678b-zzpqs"
Sep 11 23:18:51.836137 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:51.836096    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq9vk\" (UniqueName: \"kubernetes.io/projected/fe27a86b-83cf-4afa-a295-f832ded073a0-kube-api-access-dq9vk\") pod \"payload-pre-processing-e2e-inf-b-bfa694-7679c8678b-zzpqs\" (UID: \"fe27a86b-83cf-4afa-a295-f832ded073a0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bfa694-7679c8678b-zzpqs"
Sep 11 23:18:51.836856 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:51.836831    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe27a86b-83cf-4afa-a295-f832ded073a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-bfa694-7679c8678b-zzpqs\" (UID: \"fe27a86b-83cf-4afa-a295-f832ded073a0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bfa694-7679c8678b-zzpqs"
Sep 11 23:18:51.838109 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:51.837916    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe27a86b-83cf-4afa-a295-f832ded073a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-bfa694-7679c8678b-zzpqs\" (UID: \"fe27a86b-83cf-4afa-a295-f832ded073a0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bfa694-7679c8678b-zzpqs"
Sep 11 23:18:51.859421 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:51.859387    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq9vk\" (UniqueName: \"kubernetes.io/projected/fe27a86b-83cf-4afa-a295-f832ded073a0-kube-api-access-dq9vk\") pod \"payload-pre-processing-e2e-inf-b-bfa694-7679c8678b-zzpqs\" (UID: \"fe27a86b-83cf-4afa-a295-f832ded073a0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bfa694-7679c8678b-zzpqs"
Sep 11 23:18:51.993983 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:51.993894    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bfa694-7679c8678b-zzpqs"
Sep 11 23:18:52.082190 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:52.081226    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0c7930e-tvsb4"]
Sep 11 23:18:52.087391 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:52.087361    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0c7930e-tvsb4"
Sep 11 23:18:52.088672 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:52.088642    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0c7930e-tvsb4"]
Sep 11 23:18:52.093547 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:52.093526    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 23:18:52.093982 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:52.093911    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vrx7m\""
Sep 11 23:18:52.141680 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:52.141588    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64a6da1d-282c-45db-9b32-0e6a210a72fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a0c7930e-tvsb4\" (UID: \"64a6da1d-282c-45db-9b32-0e6a210a72fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0c7930e-tvsb4"
Sep 11 23:18:52.248363 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:52.245129    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64a6da1d-282c-45db-9b32-0e6a210a72fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a0c7930e-tvsb4\" (UID: \"64a6da1d-282c-45db-9b32-0e6a210a72fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0c7930e-tvsb4"
Sep 11 23:18:52.248363 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:52.246257    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64a6da1d-282c-45db-9b32-0e6a210a72fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a0c7930e-tvsb4\" (UID: \"64a6da1d-282c-45db-9b32-0e6a210a72fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0c7930e-tvsb4"
Sep 11 23:18:52.317292 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:52.317259    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp"
Sep 11 23:18:52.317752 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:52.317734    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp"
Sep 11 23:18:52.318330 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:52.318314    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp"
Sep 11 23:18:52.387765 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:52.387736    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-bfa694-7679c8678b-zzpqs"]
Sep 11 23:18:52.411741 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:52.411712    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0c7930e-tvsb4"
Sep 11 23:18:52.592178 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:52.591984    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bfa694-7679c8678b-zzpqs" event={"ID":"fe27a86b-83cf-4afa-a295-f832ded073a0","Type":"ContainerStarted","Data":"4d716905040f2fb8536590a7e9f7a09405ef88ad3372085c7b0f58a0cc304a5c"}
Sep 11 23:18:52.592178 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:52.592041    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bfa694-7679c8678b-zzpqs" event={"ID":"fe27a86b-83cf-4afa-a295-f832ded073a0","Type":"ContainerStarted","Data":"b2019883c85f75e719ee0725939439122f4c41879369ba4e712e54109cb856a3"}
Sep 11 23:18:52.595181 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:52.592707    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bfa694-7679c8678b-zzpqs"
Sep 11 23:18:52.612949 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:52.612887    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bfa694-7679c8678b-zzpqs" podStartSLOduration=1.612871806 podStartE2EDuration="1.612871806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 23:18:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:18:52.610999797 +0000 UTC m=+1056.644129015" watchObservedRunningTime="2026-09-11 23:18:52.612871806 +0000 UTC m=+1056.646001022"
Sep 11 23:18:52.793105 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:52.793077    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0c7930e-tvsb4"]
Sep 11 23:18:52.795900 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:18:52.795878    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64a6da1d_282c_45db_9b32_0e6a210a72fd.slice/crio-9e81e78a7d47ced1a428042e7e282edf059858e6ff03b582ff4a7764819012cf WatchSource:0}: Error finding container 9e81e78a7d47ced1a428042e7e282edf059858e6ff03b582ff4a7764819012cf: Status 404 returned error can't find the container with id 9e81e78a7d47ced1a428042e7e282edf059858e6ff03b582ff4a7764819012cf
Sep 11 23:18:53.598162 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:53.598122    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0c7930e-tvsb4" event={"ID":"64a6da1d-282c-45db-9b32-0e6a210a72fd","Type":"ContainerStarted","Data":"9e81e78a7d47ced1a428042e7e282edf059858e6ff03b582ff4a7764819012cf"}
Sep 11 23:18:55.504335 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:55.504302    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-06ef79-77bfb74cfd-5mr6z"
Sep 11 23:18:56.631325 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:56.631230    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0c7930e-tvsb4" event={"ID":"64a6da1d-282c-45db-9b32-0e6a210a72fd","Type":"ContainerStarted","Data":"92772103cb54aece30b906c43c328c7d9fbb1fc9e7c89f135812b0f9dd97b23d"}
Sep 11 23:18:57.643026 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:57.642986    2546 generic.go:356] "Generic (PLEG): container finished" podID="64a6da1d-282c-45db-9b32-0e6a210a72fd" containerID="92772103cb54aece30b906c43c328c7d9fbb1fc9e7c89f135812b0f9dd97b23d" exitCode=0
Sep 11 23:18:57.643836 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:57.643084    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0c7930e-tvsb4" event={"ID":"64a6da1d-282c-45db-9b32-0e6a210a72fd","Type":"ContainerDied","Data":"92772103cb54aece30b906c43c328c7d9fbb1fc9e7c89f135812b0f9dd97b23d"}
Sep 11 23:18:57.645585 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:57.645558    2546 generic.go:356] "Generic (PLEG): container finished" podID="ad22eafb-8cf0-41a7-9286-ee3d645405c4" containerID="a7ee56d8e50965ff0d1d6f5cd7f0856129ced129f7ad9a59d2372bee656adffe" exitCode=0
Sep 11 23:18:57.645887 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:57.645637    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8" event={"ID":"ad22eafb-8cf0-41a7-9286-ee3d645405c4","Type":"ContainerDied","Data":"a7ee56d8e50965ff0d1d6f5cd7f0856129ced129f7ad9a59d2372bee656adffe"}
Sep 11 23:18:57.646128 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:57.646111    2546 scope.go:117] "RemoveContainer" containerID="a7ee56d8e50965ff0d1d6f5cd7f0856129ced129f7ad9a59d2372bee656adffe"
Sep 11 23:18:57.647690 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:57.647650    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:18:57.647814 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:57.647734    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:18:57.647814 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:57.647779    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:18:58.660302 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:58.660261    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8" event={"ID":"ad22eafb-8cf0-41a7-9286-ee3d645405c4","Type":"ContainerStarted","Data":"0ab78a91b61532cdd16422e9e880a364d0bf00f8d94ec0491a327e8f4f17fde8"}
Sep 11 23:18:58.801322 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:58.801230    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8"
Sep 11 23:18:58.802030 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:58.802000    2546 patch_prober.go:28] interesting pod/e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Sep 11 23:18:58.802144 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:58.802046    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8" podUID="ad22eafb-8cf0-41a7-9286-ee3d645405c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Sep 11 23:18:58.836798 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:58.836476    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0c7930e-tvsb4"
Sep 11 23:18:58.962773 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:58.962691    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64a6da1d-282c-45db-9b32-0e6a210a72fd-maas-api-service-ca\") pod \"64a6da1d-282c-45db-9b32-0e6a210a72fd\" (UID: \"64a6da1d-282c-45db-9b32-0e6a210a72fd\") "
Sep 11 23:18:58.963129 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:58.963086    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64a6da1d-282c-45db-9b32-0e6a210a72fd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "64a6da1d-282c-45db-9b32-0e6a210a72fd" (UID: "64a6da1d-282c-45db-9b32-0e6a210a72fd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 23:18:59.063635 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:59.063587    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64a6da1d-282c-45db-9b32-0e6a210a72fd-maas-api-service-ca\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:18:59.668056 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:59.668032    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0c7930e-tvsb4"
Sep 11 23:18:59.668559 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:59.668027    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0c7930e-tvsb4" event={"ID":"64a6da1d-282c-45db-9b32-0e6a210a72fd","Type":"ContainerDied","Data":"9e81e78a7d47ced1a428042e7e282edf059858e6ff03b582ff4a7764819012cf"}
Sep 11 23:18:59.668559 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:59.668192    2546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e81e78a7d47ced1a428042e7e282edf059858e6ff03b582ff4a7764819012cf"
Sep 11 23:18:59.669898 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:59.669875    2546 generic.go:356] "Generic (PLEG): container finished" podID="cd91316e-27f3-4f22-81cd-59e6479cf614" containerID="c1cd33ea46d6fec9c54b50c7d85f7705c401cae46b4ed14d45b76e9ebb960f62" exitCode=0
Sep 11 23:18:59.670020 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:59.669952    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a0c7930e-openshift-default-865d96c648-r2kng" event={"ID":"cd91316e-27f3-4f22-81cd-59e6479cf614","Type":"ContainerDied","Data":"c1cd33ea46d6fec9c54b50c7d85f7705c401cae46b4ed14d45b76e9ebb960f62"}
Sep 11 23:18:59.670511 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:59.670492    2546 scope.go:117] "RemoveContainer" containerID="c1cd33ea46d6fec9c54b50c7d85f7705c401cae46b4ed14d45b76e9ebb960f62"
Sep 11 23:18:59.671949 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:59.671919    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:18:59.672026 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:59.672009    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:18:59.672076 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:59.672062    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:18:59.801342 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:59.801300    2546 patch_prober.go:28] interesting pod/e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Sep 11 23:18:59.801523 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:18:59.801366    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8" podUID="ad22eafb-8cf0-41a7-9286-ee3d645405c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Sep 11 23:19:00.680019 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:00.679971    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a0c7930e-openshift-default-865d96c648-r2kng" event={"ID":"cd91316e-27f3-4f22-81cd-59e6479cf614","Type":"ContainerStarted","Data":"c585325c822bf8d156c8b260d471f8699b35c8841bd94930370b00eac40461ac"}
Sep 11 23:19:00.801861 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:00.801815    2546 patch_prober.go:28] interesting pod/e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Sep 11 23:19:00.802069 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:00.801888    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8" podUID="ad22eafb-8cf0-41a7-9286-ee3d645405c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Sep 11 23:19:01.380578 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:01.380541    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a0c7930e-openshift-default-865d96c648-r2kng"
Sep 11 23:19:01.596497 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:01.596464    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a0c7930e-openshift-default-865d96c648-r2kng"
Sep 11 23:19:01.597762 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:01.597740    2546 patch_prober.go:28] interesting pod/e2e-ait-a0c7930e-openshift-default-865d96c648-r2kng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body=
Sep 11 23:19:01.597881 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:01.597777    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a0c7930e-openshift-default-865d96c648-r2kng" podUID="cd91316e-27f3-4f22-81cd-59e6479cf614" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused"
Sep 11 23:19:01.801041 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:01.801001    2546 patch_prober.go:28] interesting pod/e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Sep 11 23:19:01.801581 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:01.801052    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8" podUID="ad22eafb-8cf0-41a7-9286-ee3d645405c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Sep 11 23:19:02.387133 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:02.387064    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8"
Sep 11 23:19:02.567364 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:02.567327    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0c7930e-tvsb4"]
Sep 11 23:19:02.572418 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:02.570130    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0c7930e-tvsb4"]
Sep 11 23:19:02.596549 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:02.596409    2546 patch_prober.go:28] interesting pod/e2e-ait-a0c7930e-openshift-default-865d96c648-r2kng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body=
Sep 11 23:19:02.596549 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:02.596472    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a0c7930e-openshift-default-865d96c648-r2kng" podUID="cd91316e-27f3-4f22-81cd-59e6479cf614" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused"
Sep 11 23:19:02.710868 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:02.710829    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a0c7930e-openshift-default-865d96c648-r2kng"]
Sep 11 23:19:02.714728 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:02.714700    2546 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-ait-a0c7930e-openshift-default-865d96c648-r2kng" secret="" err="secret \"e2e-ait-a0c7930e-openshift-default-dockercfg-dxsdw\" not found"
Sep 11 23:19:02.805626 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:02.804928    2546 patch_prober.go:28] interesting pod/e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Sep 11 23:19:02.805626 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:02.804992    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8" podUID="ad22eafb-8cf0-41a7-9286-ee3d645405c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Sep 11 23:19:03.598855 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:03.597284    2546 patch_prober.go:28] interesting pod/e2e-ait-a0c7930e-openshift-default-865d96c648-r2kng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body=
Sep 11 23:19:03.598855 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:03.597348    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a0c7930e-openshift-default-865d96c648-r2kng" podUID="cd91316e-27f3-4f22-81cd-59e6479cf614" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused"
Sep 11 23:19:03.686760 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:03.686717    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql"]
Sep 11 23:19:03.687474 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:03.687448    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64a6da1d-282c-45db-9b32-0e6a210a72fd" containerName="revoke-keys"
Sep 11 23:19:03.687474 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:03.687472    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="64a6da1d-282c-45db-9b32-0e6a210a72fd" containerName="revoke-keys"
Sep 11 23:19:03.687698 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:03.687602    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="64a6da1d-282c-45db-9b32-0e6a210a72fd" containerName="revoke-keys"
Sep 11 23:19:03.691403 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:03.691380    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql"
Sep 11 23:19:03.694867 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:03.694746    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-7283a095-gw-openshift-default-dockercfg-wzdzk\""
Sep 11 23:19:03.707728 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:03.707698    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql"]
Sep 11 23:19:03.723955 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:03.723905    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-a0c7930e-openshift-default-865d96c648-r2kng" podUID="cd91316e-27f3-4f22-81cd-59e6479cf614" containerName="istio-proxy" containerID="cri-o://c585325c822bf8d156c8b260d471f8699b35c8841bd94930370b00eac40461ac" gracePeriod=30
Sep 11 23:19:03.801368 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:03.801326    2546 patch_prober.go:28] interesting pod/e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Sep 11 23:19:03.801540 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:03.801402    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8" podUID="ad22eafb-8cf0-41a7-9286-ee3d645405c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Sep 11 23:19:03.822061 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:03.822025    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee160166-a67b-436e-acae-2f1dc159cda9-workload-certs\") pod \"e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql\" (UID: \"ee160166-a67b-436e-acae-2f1dc159cda9\") " pod="openshift-ingress/e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql"
Sep 11 23:19:03.822061 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:03.822062    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee160166-a67b-436e-acae-2f1dc159cda9-istiod-ca-cert\") pod \"e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql\" (UID: \"ee160166-a67b-436e-acae-2f1dc159cda9\") " pod="openshift-ingress/e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql"
Sep 11 23:19:03.822543 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:03.822098    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee160166-a67b-436e-acae-2f1dc159cda9-istio-data\") pod \"e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql\" (UID: \"ee160166-a67b-436e-acae-2f1dc159cda9\") " pod="openshift-ingress/e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql"
Sep 11 23:19:03.822543 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:03.822194    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee160166-a67b-436e-acae-2f1dc159cda9-workload-socket\") pod \"e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql\" (UID: \"ee160166-a67b-436e-acae-2f1dc159cda9\") " pod="openshift-ingress/e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql"
Sep 11 23:19:03.822543 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:03.822237    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tpn7\" (UniqueName: \"kubernetes.io/projected/ee160166-a67b-436e-acae-2f1dc159cda9-kube-api-access-4tpn7\") pod \"e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql\" (UID: \"ee160166-a67b-436e-acae-2f1dc159cda9\") " pod="openshift-ingress/e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql"
Sep 11 23:19:03.822543 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:03.822269    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee160166-a67b-436e-acae-2f1dc159cda9-istio-envoy\") pod \"e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql\" (UID: \"ee160166-a67b-436e-acae-2f1dc159cda9\") " pod="openshift-ingress/e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql"
Sep 11 23:19:03.822543 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:03.822374    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee160166-a67b-436e-acae-2f1dc159cda9-istio-token\") pod \"e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql\" (UID: \"ee160166-a67b-436e-acae-2f1dc159cda9\") " pod="openshift-ingress/e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql"
Sep 11 23:19:03.822543 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:03.822426    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee160166-a67b-436e-acae-2f1dc159cda9-istio-podinfo\") pod \"e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql\" (UID: \"ee160166-a67b-436e-acae-2f1dc159cda9\") " pod="openshift-ingress/e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql"
Sep 11 23:19:03.822543 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:03.822474    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee160166-a67b-436e-acae-2f1dc159cda9-credential-socket\") pod \"e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql\" (UID: \"ee160166-a67b-436e-acae-2f1dc159cda9\") " pod="openshift-ingress/e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql"
Sep 11 23:19:03.924721 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:03.924041    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee160166-a67b-436e-acae-2f1dc159cda9-istio-podinfo\") pod \"e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql\" (UID: \"ee160166-a67b-436e-acae-2f1dc159cda9\") " pod="openshift-ingress/e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql"
Sep 11 23:19:03.924933 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:03.924786    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee160166-a67b-436e-acae-2f1dc159cda9-credential-socket\") pod \"e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql\" (UID: \"ee160166-a67b-436e-acae-2f1dc159cda9\") " pod="openshift-ingress/e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql"
Sep 11 23:19:03.924933 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:03.924869    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee160166-a67b-436e-acae-2f1dc159cda9-workload-certs\") pod \"e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql\" (UID: \"ee160166-a67b-436e-acae-2f1dc159cda9\") " pod="openshift-ingress/e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql"
Sep 11 23:19:03.925064 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:03.924975    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee160166-a67b-436e-acae-2f1dc159cda9-istiod-ca-cert\") pod \"e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql\" (UID: \"ee160166-a67b-436e-acae-2f1dc159cda9\") " pod="openshift-ingress/e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql"
Sep 11 23:19:03.925125 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:03.925064    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee160166-a67b-436e-acae-2f1dc159cda9-istio-data\") pod \"e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql\" (UID: \"ee160166-a67b-436e-acae-2f1dc159cda9\") " pod="openshift-ingress/e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql"
Sep 11 23:19:03.925217 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:03.925147    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee160166-a67b-436e-acae-2f1dc159cda9-workload-socket\") pod \"e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql\" (UID: \"ee160166-a67b-436e-acae-2f1dc159cda9\") " pod="openshift-ingress/e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql"
Sep 11 23:19:03.925309 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:03.925286    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tpn7\" (UniqueName: \"kubernetes.io/projected/ee160166-a67b-436e-acae-2f1dc159cda9-kube-api-access-4tpn7\") pod \"e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql\" (UID: \"ee160166-a67b-436e-acae-2f1dc159cda9\") " pod="openshift-ingress/e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql"
Sep 11 23:19:03.925378 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:03.925356    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee160166-a67b-436e-acae-2f1dc159cda9-istio-envoy\") pod \"e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql\" (UID: \"ee160166-a67b-436e-acae-2f1dc159cda9\") " pod="openshift-ingress/e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql"
Sep 11 23:19:03.925456 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:03.925438    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee160166-a67b-436e-acae-2f1dc159cda9-istio-token\") pod \"e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql\" (UID: \"ee160166-a67b-436e-acae-2f1dc159cda9\") " pod="openshift-ingress/e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql"
Sep 11 23:19:03.925541 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:03.925437    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee160166-a67b-436e-acae-2f1dc159cda9-workload-certs\") pod \"e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql\" (UID: \"ee160166-a67b-436e-acae-2f1dc159cda9\") " pod="openshift-ingress/e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql"
Sep 11 23:19:03.925601 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:03.925585    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee160166-a67b-436e-acae-2f1dc159cda9-credential-socket\") pod \"e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql\" (UID: \"ee160166-a67b-436e-acae-2f1dc159cda9\") " pod="openshift-ingress/e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql"
Sep 11 23:19:03.925722 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:03.925699    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee160166-a67b-436e-acae-2f1dc159cda9-workload-socket\") pod \"e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql\" (UID: \"ee160166-a67b-436e-acae-2f1dc159cda9\") " pod="openshift-ingress/e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql"
Sep 11 23:19:03.926119 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:03.926098    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee160166-a67b-436e-acae-2f1dc159cda9-istio-data\") pod \"e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql\" (UID: \"ee160166-a67b-436e-acae-2f1dc159cda9\") " pod="openshift-ingress/e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql"
Sep 11 23:19:03.926732 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:03.926571    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee160166-a67b-436e-acae-2f1dc159cda9-istiod-ca-cert\") pod \"e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql\" (UID: \"ee160166-a67b-436e-acae-2f1dc159cda9\") " pod="openshift-ingress/e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql"
Sep 11 23:19:03.930389 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:03.930368    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee160166-a67b-436e-acae-2f1dc159cda9-istio-envoy\") pod \"e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql\" (UID: \"ee160166-a67b-436e-acae-2f1dc159cda9\") " pod="openshift-ingress/e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql"
Sep 11 23:19:03.930950 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:03.930902    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee160166-a67b-436e-acae-2f1dc159cda9-istio-podinfo\") pod \"e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql\" (UID: \"ee160166-a67b-436e-acae-2f1dc159cda9\") " pod="openshift-ingress/e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql"
Sep 11 23:19:03.941174 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:03.937134    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tpn7\" (UniqueName: \"kubernetes.io/projected/ee160166-a67b-436e-acae-2f1dc159cda9-kube-api-access-4tpn7\") pod \"e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql\" (UID: \"ee160166-a67b-436e-acae-2f1dc159cda9\") " pod="openshift-ingress/e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql"
Sep 11 23:19:03.941736 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:03.941698    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee160166-a67b-436e-acae-2f1dc159cda9-istio-token\") pod \"e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql\" (UID: \"ee160166-a67b-436e-acae-2f1dc159cda9\") " pod="openshift-ingress/e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql"
Sep 11 23:19:04.005752 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:04.005712    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql"
Sep 11 23:19:04.352296 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:04.351195    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql"]
Sep 11 23:19:04.355239 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:19:04.355200    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee160166_a67b_436e_acae_2f1dc159cda9.slice/crio-0801fe21ff052da6057bcbe373a47a37be9ea53d99229a4edd40fc896b595246 WatchSource:0}: Error finding container 0801fe21ff052da6057bcbe373a47a37be9ea53d99229a4edd40fc896b595246: Status 404 returned error can't find the container with id 0801fe21ff052da6057bcbe373a47a37be9ea53d99229a4edd40fc896b595246
Sep 11 23:19:04.357700 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:04.357664    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:19:04.357789 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:04.357733    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:19:04.357789 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:04.357760    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:19:04.496649 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:04.496612    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64a6da1d-282c-45db-9b32-0e6a210a72fd" path="/var/lib/kubelet/pods/64a6da1d-282c-45db-9b32-0e6a210a72fd/volumes"
Sep 11 23:19:04.612575 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:04.612498    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bfa694-7679c8678b-zzpqs"
Sep 11 23:19:04.744907 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:04.744853    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql" event={"ID":"ee160166-a67b-436e-acae-2f1dc159cda9","Type":"ContainerStarted","Data":"fb80805f2b77d5084b81b23e012c4fa0c7e6d2838ffe4870c6a912499fefe464"}
Sep 11 23:19:04.745202 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:04.745184    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql" event={"ID":"ee160166-a67b-436e-acae-2f1dc159cda9","Type":"ContainerStarted","Data":"0801fe21ff052da6057bcbe373a47a37be9ea53d99229a4edd40fc896b595246"}
Sep 11 23:19:04.772518 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:04.772473    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql" podStartSLOduration=1.772457468 podStartE2EDuration="1.772457468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 23:19:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:19:04.771224901 +0000 UTC m=+1068.804354117" watchObservedRunningTime="2026-09-11 23:19:04.772457468 +0000 UTC m=+1068.805586685"
Sep 11 23:19:04.805204 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:04.805179    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8"
Sep 11 23:19:04.806353 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:04.806334    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8"
Sep 11 23:19:05.006988 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:05.006956    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql"
Sep 11 23:19:05.022828 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:05.022792    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql"
Sep 11 23:19:05.757022 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:05.756986    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql"
Sep 11 23:19:05.764324 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:05.764292    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql"
Sep 11 23:19:11.799380 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:11.799345    2546 generic.go:356] "Generic (PLEG): container finished" podID="cd91316e-27f3-4f22-81cd-59e6479cf614" containerID="c585325c822bf8d156c8b260d471f8699b35c8841bd94930370b00eac40461ac" exitCode=0
Sep 11 23:19:11.799814 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:11.799413    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a0c7930e-openshift-default-865d96c648-r2kng" event={"ID":"cd91316e-27f3-4f22-81cd-59e6479cf614","Type":"ContainerDied","Data":"c585325c822bf8d156c8b260d471f8699b35c8841bd94930370b00eac40461ac"}
Sep 11 23:19:11.799814 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:11.799471    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a0c7930e-openshift-default-865d96c648-r2kng" event={"ID":"cd91316e-27f3-4f22-81cd-59e6479cf614","Type":"ContainerDied","Data":"19751ccd1f1e11ad5430613005cff16df08fe0a52b50b6963c57d9966d3765b0"}
Sep 11 23:19:11.799814 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:11.799487    2546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19751ccd1f1e11ad5430613005cff16df08fe0a52b50b6963c57d9966d3765b0"
Sep 11 23:19:11.799814 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:11.799501    2546 scope.go:117] "RemoveContainer" containerID="c1cd33ea46d6fec9c54b50c7d85f7705c401cae46b4ed14d45b76e9ebb960f62"
Sep 11 23:19:11.811284 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:11.811133    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a0c7930e-openshift-default-865d96c648-r2kng"
Sep 11 23:19:11.829547 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:11.829509    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd91316e-27f3-4f22-81cd-59e6479cf614-istiod-ca-cert\") pod \"cd91316e-27f3-4f22-81cd-59e6479cf614\" (UID: \"cd91316e-27f3-4f22-81cd-59e6479cf614\") "
Sep 11 23:19:11.829711 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:11.829579    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd91316e-27f3-4f22-81cd-59e6479cf614-workload-certs\") pod \"cd91316e-27f3-4f22-81cd-59e6479cf614\" (UID: \"cd91316e-27f3-4f22-81cd-59e6479cf614\") "
Sep 11 23:19:11.829711 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:11.829631    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd91316e-27f3-4f22-81cd-59e6479cf614-istio-envoy\") pod \"cd91316e-27f3-4f22-81cd-59e6479cf614\" (UID: \"cd91316e-27f3-4f22-81cd-59e6479cf614\") "
Sep 11 23:19:11.829711 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:11.829658    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd91316e-27f3-4f22-81cd-59e6479cf614-istio-token\") pod \"cd91316e-27f3-4f22-81cd-59e6479cf614\" (UID: \"cd91316e-27f3-4f22-81cd-59e6479cf614\") "
Sep 11 23:19:11.829711 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:11.829679    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd91316e-27f3-4f22-81cd-59e6479cf614-istio-data\") pod \"cd91316e-27f3-4f22-81cd-59e6479cf614\" (UID: \"cd91316e-27f3-4f22-81cd-59e6479cf614\") "
Sep 11 23:19:11.829940 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:11.829736    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-42jp8\" (UniqueName: \"kubernetes.io/projected/cd91316e-27f3-4f22-81cd-59e6479cf614-kube-api-access-42jp8\") pod \"cd91316e-27f3-4f22-81cd-59e6479cf614\" (UID: \"cd91316e-27f3-4f22-81cd-59e6479cf614\") "
Sep 11 23:19:11.829940 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:11.829760    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd91316e-27f3-4f22-81cd-59e6479cf614-workload-socket\") pod \"cd91316e-27f3-4f22-81cd-59e6479cf614\" (UID: \"cd91316e-27f3-4f22-81cd-59e6479cf614\") "
Sep 11 23:19:11.829940 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:11.829814    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd91316e-27f3-4f22-81cd-59e6479cf614-istio-podinfo\") pod \"cd91316e-27f3-4f22-81cd-59e6479cf614\" (UID: \"cd91316e-27f3-4f22-81cd-59e6479cf614\") "
Sep 11 23:19:11.829940 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:11.829825    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd91316e-27f3-4f22-81cd-59e6479cf614-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cd91316e-27f3-4f22-81cd-59e6479cf614" (UID: "cd91316e-27f3-4f22-81cd-59e6479cf614"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:19:11.829940 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:11.829844    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd91316e-27f3-4f22-81cd-59e6479cf614-credential-socket\") pod \"cd91316e-27f3-4f22-81cd-59e6479cf614\" (UID: \"cd91316e-27f3-4f22-81cd-59e6479cf614\") "
Sep 11 23:19:11.829940 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:11.829904    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd91316e-27f3-4f22-81cd-59e6479cf614-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cd91316e-27f3-4f22-81cd-59e6479cf614" (UID: "cd91316e-27f3-4f22-81cd-59e6479cf614"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 23:19:11.830292 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:11.830043    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd91316e-27f3-4f22-81cd-59e6479cf614-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cd91316e-27f3-4f22-81cd-59e6479cf614" (UID: "cd91316e-27f3-4f22-81cd-59e6479cf614"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:19:11.830292 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:11.830247    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd91316e-27f3-4f22-81cd-59e6479cf614-istio-data" (OuterVolumeSpecName: "istio-data") pod "cd91316e-27f3-4f22-81cd-59e6479cf614" (UID: "cd91316e-27f3-4f22-81cd-59e6479cf614"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:19:11.830400 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:11.830279    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd91316e-27f3-4f22-81cd-59e6479cf614-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cd91316e-27f3-4f22-81cd-59e6479cf614" (UID: "cd91316e-27f3-4f22-81cd-59e6479cf614"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:19:11.830458 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:11.830400    2546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd91316e-27f3-4f22-81cd-59e6479cf614-workload-certs\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:19:11.830458 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:11.830418    2546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd91316e-27f3-4f22-81cd-59e6479cf614-istio-data\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:19:11.830458 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:11.830429    2546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd91316e-27f3-4f22-81cd-59e6479cf614-workload-socket\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:19:11.830458 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:11.830442    2546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd91316e-27f3-4f22-81cd-59e6479cf614-credential-socket\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:19:11.830458 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:11.830456    2546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd91316e-27f3-4f22-81cd-59e6479cf614-istiod-ca-cert\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:19:11.832637 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:11.832604    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd91316e-27f3-4f22-81cd-59e6479cf614-kube-api-access-42jp8" (OuterVolumeSpecName: "kube-api-access-42jp8") pod "cd91316e-27f3-4f22-81cd-59e6479cf614" (UID: "cd91316e-27f3-4f22-81cd-59e6479cf614"). InnerVolumeSpecName "kube-api-access-42jp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 23:19:11.832637 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:11.832610    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd91316e-27f3-4f22-81cd-59e6479cf614-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cd91316e-27f3-4f22-81cd-59e6479cf614" (UID: "cd91316e-27f3-4f22-81cd-59e6479cf614"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:19:11.832835 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:11.832667    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd91316e-27f3-4f22-81cd-59e6479cf614-istio-token" (OuterVolumeSpecName: "istio-token") pod "cd91316e-27f3-4f22-81cd-59e6479cf614" (UID: "cd91316e-27f3-4f22-81cd-59e6479cf614"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 23:19:11.832835 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:11.832806    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cd91316e-27f3-4f22-81cd-59e6479cf614-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cd91316e-27f3-4f22-81cd-59e6479cf614" (UID: "cd91316e-27f3-4f22-81cd-59e6479cf614"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 23:19:11.931517 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:11.931478    2546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd91316e-27f3-4f22-81cd-59e6479cf614-istio-envoy\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:19:11.931517 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:11.931505    2546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd91316e-27f3-4f22-81cd-59e6479cf614-istio-token\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:19:11.931517 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:11.931514    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42jp8\" (UniqueName: \"kubernetes.io/projected/cd91316e-27f3-4f22-81cd-59e6479cf614-kube-api-access-42jp8\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:19:11.931517 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:11.931522    2546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd91316e-27f3-4f22-81cd-59e6479cf614-istio-podinfo\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:19:12.809978 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:12.809895    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a0c7930e-openshift-default-865d96c648-r2kng"
Sep 11 23:19:12.850001 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:12.849963    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a0c7930e-openshift-default-865d96c648-r2kng"]
Sep 11 23:19:12.853354 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:12.853327    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-a0c7930e-openshift-default-865d96c648-r2kng"]
Sep 11 23:19:14.495492 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:14.495463    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd91316e-27f3-4f22-81cd-59e6479cf614" path="/var/lib/kubelet/pods/cd91316e-27f3-4f22-81cd-59e6479cf614/volumes"
Sep 11 23:19:14.825285 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:14.825252    2546 generic.go:356] "Generic (PLEG): container finished" podID="dcd4c593-5a49-4cdc-bd1a-27c49cbe594c" containerID="b8252117d86680331cf009c4a07735c80a2714ef52eccd148c2cab7e5aae94e7" exitCode=0
Sep 11 23:19:14.825507 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:14.825317    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq" event={"ID":"dcd4c593-5a49-4cdc-bd1a-27c49cbe594c","Type":"ContainerDied","Data":"b8252117d86680331cf009c4a07735c80a2714ef52eccd148c2cab7e5aae94e7"}
Sep 11 23:19:14.825752 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:14.825737    2546 scope.go:117] "RemoveContainer" containerID="b8252117d86680331cf009c4a07735c80a2714ef52eccd148c2cab7e5aae94e7"
Sep 11 23:19:14.827042 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:14.827015    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:19:14.827122 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:14.827085    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:19:14.827122 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:14.827116    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:19:15.834834 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:15.834800    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq" event={"ID":"dcd4c593-5a49-4cdc-bd1a-27c49cbe594c","Type":"ContainerStarted","Data":"e3f7a3859610c7b831ae66497695213320dac08ba33e63db7db8a05d0943538e"}
Sep 11 23:19:16.065486 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:16.065447    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq"
Sep 11 23:19:16.067677 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:16.067289    2546 patch_prober.go:28] interesting pod/e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Sep 11 23:19:16.067677 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:16.067368    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq" podUID="dcd4c593-5a49-4cdc-bd1a-27c49cbe594c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Sep 11 23:19:16.502976 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:16.502947    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"]
Sep 11 23:19:17.065544 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:17.065510    2546 patch_prober.go:28] interesting pod/e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Sep 11 23:19:17.065938 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:17.065578    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq" podUID="dcd4c593-5a49-4cdc-bd1a-27c49cbe594c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Sep 11 23:19:17.519690 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:17.519652    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq"
Sep 11 23:19:18.065438 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:18.065399    2546 patch_prober.go:28] interesting pod/e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Sep 11 23:19:18.065607 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:18.065475    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq" podUID="dcd4c593-5a49-4cdc-bd1a-27c49cbe594c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Sep 11 23:19:19.066323 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:19.066278    2546 patch_prober.go:28] interesting pod/e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Sep 11 23:19:19.066847 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:19.066364    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq" podUID="dcd4c593-5a49-4cdc-bd1a-27c49cbe594c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Sep 11 23:19:20.065923 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:20.065876    2546 patch_prober.go:28] interesting pod/e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Sep 11 23:19:20.066145 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:20.065959    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq" podUID="dcd4c593-5a49-4cdc-bd1a-27c49cbe594c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Sep 11 23:19:21.065341 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:21.065298    2546 patch_prober.go:28] interesting pod/e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Sep 11 23:19:21.066403 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:21.065937    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq" podUID="dcd4c593-5a49-4cdc-bd1a-27c49cbe594c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Sep 11 23:19:22.069863 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:22.069828    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq"
Sep 11 23:19:22.070790 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:22.070769    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq"
Sep 11 23:19:27.929793 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:27.929756    2546 generic.go:356] "Generic (PLEG): container finished" podID="ee160166-a67b-436e-acae-2f1dc159cda9" containerID="fb80805f2b77d5084b81b23e012c4fa0c7e6d2838ffe4870c6a912499fefe464" exitCode=0
Sep 11 23:19:27.930292 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:27.929817    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql" event={"ID":"ee160166-a67b-436e-acae-2f1dc159cda9","Type":"ContainerDied","Data":"fb80805f2b77d5084b81b23e012c4fa0c7e6d2838ffe4870c6a912499fefe464"}
Sep 11 23:19:27.930358 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:27.930339    2546 scope.go:117] "RemoveContainer" containerID="fb80805f2b77d5084b81b23e012c4fa0c7e6d2838ffe4870c6a912499fefe464"
Sep 11 23:19:27.931585 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:27.931554    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:19:27.931701 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:27.931645    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:19:27.931701 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:27.931692    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:19:28.943005 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:28.942959    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql" event={"ID":"ee160166-a67b-436e-acae-2f1dc159cda9","Type":"ContainerStarted","Data":"7762a00e6ad6fc41c82c8e8c34d8aab907fbc66ced60c347e14757a0ab0808e3"}
Sep 11 23:19:29.006289 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:29.006256    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql"
Sep 11 23:19:29.006580 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:29.006555    2546 patch_prober.go:28] interesting pod/e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Sep 11 23:19:29.006659 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:29.006610    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql" podUID="ee160166-a67b-436e-acae-2f1dc159cda9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Sep 11 23:19:30.006535 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:30.006502    2546 patch_prober.go:28] interesting pod/e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Sep 11 23:19:30.006936 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:30.006554    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql" podUID="ee160166-a67b-436e-acae-2f1dc159cda9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Sep 11 23:19:30.217389 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:30.217351    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-12fb6b/simulator-w0-12fb6b-kserve-9749df7b4-859hf"]
Sep 11 23:19:30.217810 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:30.217797    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd91316e-27f3-4f22-81cd-59e6479cf614" containerName="istio-proxy"
Sep 11 23:19:30.217855 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:30.217812    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd91316e-27f3-4f22-81cd-59e6479cf614" containerName="istio-proxy"
Sep 11 23:19:30.217855 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:30.217826    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd91316e-27f3-4f22-81cd-59e6479cf614" containerName="istio-proxy"
Sep 11 23:19:30.217855 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:30.217831    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd91316e-27f3-4f22-81cd-59e6479cf614" containerName="istio-proxy"
Sep 11 23:19:30.217953 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:30.217923    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd91316e-27f3-4f22-81cd-59e6479cf614" containerName="istio-proxy"
Sep 11 23:19:30.217953 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:30.217938    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd91316e-27f3-4f22-81cd-59e6479cf614" containerName="istio-proxy"
Sep 11 23:19:30.221243 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:30.221228    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-12fb6b/simulator-w0-12fb6b-kserve-9749df7b4-859hf"
Sep 11 23:19:30.229183 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:30.225805    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-12fb6b\"/\"simulator-w0-12fb6b-kserve-self-signed-certs\""
Sep 11 23:19:30.229183 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:30.226112    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-12fb6b\"/\"kube-root-ca.crt\""
Sep 11 23:19:30.229183 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:30.226345    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-12fb6b\"/\"default-dockercfg-vq6vm\""
Sep 11 23:19:30.229183 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:30.226522    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-12fb6b\"/\"openshift-service-ca.crt\""
Sep 11 23:19:30.234021 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:30.233998    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-12fb6b/simulator-w0-12fb6b-kserve-9749df7b4-859hf"]
Sep 11 23:19:30.339235 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:30.338712    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d4d86af-7f04-4bcf-ad33-fb1fed658519-model-cache\") pod \"simulator-w0-12fb6b-kserve-9749df7b4-859hf\" (UID: \"2d4d86af-7f04-4bcf-ad33-fb1fed658519\") " pod="e2e-models-e2e-worker-w0-12fb6b/simulator-w0-12fb6b-kserve-9749df7b4-859hf"
Sep 11 23:19:30.339235 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:30.338815    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d4d86af-7f04-4bcf-ad33-fb1fed658519-dshm\") pod \"simulator-w0-12fb6b-kserve-9749df7b4-859hf\" (UID: \"2d4d86af-7f04-4bcf-ad33-fb1fed658519\") " pod="e2e-models-e2e-worker-w0-12fb6b/simulator-w0-12fb6b-kserve-9749df7b4-859hf"
Sep 11 23:19:30.339235 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:30.338854    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d4d86af-7f04-4bcf-ad33-fb1fed658519-tmp-dir\") pod \"simulator-w0-12fb6b-kserve-9749df7b4-859hf\" (UID: \"2d4d86af-7f04-4bcf-ad33-fb1fed658519\") " pod="e2e-models-e2e-worker-w0-12fb6b/simulator-w0-12fb6b-kserve-9749df7b4-859hf"
Sep 11 23:19:30.339235 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:30.338901    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbxsk\" (UniqueName: \"kubernetes.io/projected/2d4d86af-7f04-4bcf-ad33-fb1fed658519-kube-api-access-wbxsk\") pod \"simulator-w0-12fb6b-kserve-9749df7b4-859hf\" (UID: \"2d4d86af-7f04-4bcf-ad33-fb1fed658519\") " pod="e2e-models-e2e-worker-w0-12fb6b/simulator-w0-12fb6b-kserve-9749df7b4-859hf"
Sep 11 23:19:30.339235 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:30.338967    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d4d86af-7f04-4bcf-ad33-fb1fed658519-tls-certs\") pod \"simulator-w0-12fb6b-kserve-9749df7b4-859hf\" (UID: \"2d4d86af-7f04-4bcf-ad33-fb1fed658519\") " pod="e2e-models-e2e-worker-w0-12fb6b/simulator-w0-12fb6b-kserve-9749df7b4-859hf"
Sep 11 23:19:30.339235 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:30.339014    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d4d86af-7f04-4bcf-ad33-fb1fed658519-home\") pod \"simulator-w0-12fb6b-kserve-9749df7b4-859hf\" (UID: \"2d4d86af-7f04-4bcf-ad33-fb1fed658519\") " pod="e2e-models-e2e-worker-w0-12fb6b/simulator-w0-12fb6b-kserve-9749df7b4-859hf"
Sep 11 23:19:30.440772 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:30.440258    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d4d86af-7f04-4bcf-ad33-fb1fed658519-dshm\") pod \"simulator-w0-12fb6b-kserve-9749df7b4-859hf\" (UID: \"2d4d86af-7f04-4bcf-ad33-fb1fed658519\") " pod="e2e-models-e2e-worker-w0-12fb6b/simulator-w0-12fb6b-kserve-9749df7b4-859hf"
Sep 11 23:19:30.441328 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:30.441308    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d4d86af-7f04-4bcf-ad33-fb1fed658519-tmp-dir\") pod \"simulator-w0-12fb6b-kserve-9749df7b4-859hf\" (UID: \"2d4d86af-7f04-4bcf-ad33-fb1fed658519\") " pod="e2e-models-e2e-worker-w0-12fb6b/simulator-w0-12fb6b-kserve-9749df7b4-859hf"
Sep 11 23:19:30.441552 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:30.441532    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbxsk\" (UniqueName: \"kubernetes.io/projected/2d4d86af-7f04-4bcf-ad33-fb1fed658519-kube-api-access-wbxsk\") pod \"simulator-w0-12fb6b-kserve-9749df7b4-859hf\" (UID: \"2d4d86af-7f04-4bcf-ad33-fb1fed658519\") " pod="e2e-models-e2e-worker-w0-12fb6b/simulator-w0-12fb6b-kserve-9749df7b4-859hf"
Sep 11 23:19:30.442054 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:30.441893    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d4d86af-7f04-4bcf-ad33-fb1fed658519-tmp-dir\") pod \"simulator-w0-12fb6b-kserve-9749df7b4-859hf\" (UID: \"2d4d86af-7f04-4bcf-ad33-fb1fed658519\") " pod="e2e-models-e2e-worker-w0-12fb6b/simulator-w0-12fb6b-kserve-9749df7b4-859hf"
Sep 11 23:19:30.442262 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:30.442242    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d4d86af-7f04-4bcf-ad33-fb1fed658519-tls-certs\") pod \"simulator-w0-12fb6b-kserve-9749df7b4-859hf\" (UID: \"2d4d86af-7f04-4bcf-ad33-fb1fed658519\") " pod="e2e-models-e2e-worker-w0-12fb6b/simulator-w0-12fb6b-kserve-9749df7b4-859hf"
Sep 11 23:19:30.442847 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:30.442826    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d4d86af-7f04-4bcf-ad33-fb1fed658519-home\") pod \"simulator-w0-12fb6b-kserve-9749df7b4-859hf\" (UID: \"2d4d86af-7f04-4bcf-ad33-fb1fed658519\") " pod="e2e-models-e2e-worker-w0-12fb6b/simulator-w0-12fb6b-kserve-9749df7b4-859hf"
Sep 11 23:19:30.443023 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:30.443006    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d4d86af-7f04-4bcf-ad33-fb1fed658519-model-cache\") pod \"simulator-w0-12fb6b-kserve-9749df7b4-859hf\" (UID: \"2d4d86af-7f04-4bcf-ad33-fb1fed658519\") " pod="e2e-models-e2e-worker-w0-12fb6b/simulator-w0-12fb6b-kserve-9749df7b4-859hf"
Sep 11 23:19:30.443333 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:30.443313    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d4d86af-7f04-4bcf-ad33-fb1fed658519-dshm\") pod \"simulator-w0-12fb6b-kserve-9749df7b4-859hf\" (UID: \"2d4d86af-7f04-4bcf-ad33-fb1fed658519\") " pod="e2e-models-e2e-worker-w0-12fb6b/simulator-w0-12fb6b-kserve-9749df7b4-859hf"
Sep 11 23:19:30.453290 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:30.443365    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d4d86af-7f04-4bcf-ad33-fb1fed658519-model-cache\") pod \"simulator-w0-12fb6b-kserve-9749df7b4-859hf\" (UID: \"2d4d86af-7f04-4bcf-ad33-fb1fed658519\") " pod="e2e-models-e2e-worker-w0-12fb6b/simulator-w0-12fb6b-kserve-9749df7b4-859hf"
Sep 11 23:19:30.453290 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:30.443639    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d4d86af-7f04-4bcf-ad33-fb1fed658519-home\") pod \"simulator-w0-12fb6b-kserve-9749df7b4-859hf\" (UID: \"2d4d86af-7f04-4bcf-ad33-fb1fed658519\") " pod="e2e-models-e2e-worker-w0-12fb6b/simulator-w0-12fb6b-kserve-9749df7b4-859hf"
Sep 11 23:19:30.453290 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:30.452797    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbxsk\" (UniqueName: \"kubernetes.io/projected/2d4d86af-7f04-4bcf-ad33-fb1fed658519-kube-api-access-wbxsk\") pod \"simulator-w0-12fb6b-kserve-9749df7b4-859hf\" (UID: \"2d4d86af-7f04-4bcf-ad33-fb1fed658519\") " pod="e2e-models-e2e-worker-w0-12fb6b/simulator-w0-12fb6b-kserve-9749df7b4-859hf"
Sep 11 23:19:30.454333 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:30.454309    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d4d86af-7f04-4bcf-ad33-fb1fed658519-tls-certs\") pod \"simulator-w0-12fb6b-kserve-9749df7b4-859hf\" (UID: \"2d4d86af-7f04-4bcf-ad33-fb1fed658519\") " pod="e2e-models-e2e-worker-w0-12fb6b/simulator-w0-12fb6b-kserve-9749df7b4-859hf"
Sep 11 23:19:30.545180 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:30.542493    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-12fb6b/simulator-w0-12fb6b-kserve-9749df7b4-859hf"
Sep 11 23:19:30.852329 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:30.852300    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-12fb6b/simulator-w0-12fb6b-kserve-9749df7b4-859hf"]
Sep 11 23:19:30.854759 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:19:30.854729    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2d4d86af_7f04_4bcf_ad33_fb1fed658519.slice/crio-0c7da0ebdc99a47fe9dccd1eca947e860c6e740b24b22c24d62be12a64eae768 WatchSource:0}: Error finding container 0c7da0ebdc99a47fe9dccd1eca947e860c6e740b24b22c24d62be12a64eae768: Status 404 returned error can't find the container with id 0c7da0ebdc99a47fe9dccd1eca947e860c6e740b24b22c24d62be12a64eae768
Sep 11 23:19:30.963586 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:30.963543    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-12fb6b/simulator-w0-12fb6b-kserve-9749df7b4-859hf" event={"ID":"2d4d86af-7f04-4bcf-ad33-fb1fed658519","Type":"ContainerStarted","Data":"d87a308915173a6be103b44404ac88d92021424ebc3386d922e48d83f1985a23"}
Sep 11 23:19:30.963759 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:30.963596    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-12fb6b/simulator-w0-12fb6b-kserve-9749df7b4-859hf" event={"ID":"2d4d86af-7f04-4bcf-ad33-fb1fed658519","Type":"ContainerStarted","Data":"0c7da0ebdc99a47fe9dccd1eca947e860c6e740b24b22c24d62be12a64eae768"}
Sep 11 23:19:30.988174 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:30.988095    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-12fb6b/simulator-w0-12fb6b-kserve-9749df7b4-859hf" podStartSLOduration=0.988078567 podStartE2EDuration="988.078567ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 23:19:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:19:30.986326534 +0000 UTC m=+1095.019455752" watchObservedRunningTime="2026-09-11 23:19:30.988078567 +0000 UTC m=+1095.021207783"
Sep 11 23:19:31.006519 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:31.006488    2546 patch_prober.go:28] interesting pod/e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Sep 11 23:19:31.006666 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:31.006535    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql" podUID="ee160166-a67b-436e-acae-2f1dc159cda9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Sep 11 23:19:32.006397 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:32.006362    2546 patch_prober.go:28] interesting pod/e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Sep 11 23:19:32.006592 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:32.006426    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql" podUID="ee160166-a67b-436e-acae-2f1dc159cda9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Sep 11 23:19:33.006837 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:33.006797    2546 patch_prober.go:28] interesting pod/e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Sep 11 23:19:33.007300 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:33.006878    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql" podUID="ee160166-a67b-436e-acae-2f1dc159cda9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Sep 11 23:19:34.006491 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:34.006456    2546 patch_prober.go:28] interesting pod/e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Sep 11 23:19:34.006673 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:34.006513    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql" podUID="ee160166-a67b-436e-acae-2f1dc159cda9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Sep 11 23:19:35.010859 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:35.010808    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql"
Sep 11 23:19:35.011612 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:35.011586    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql"
Sep 11 23:19:35.014081 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:35.014052    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql"
Sep 11 23:19:35.993434 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:35.993402    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"]
Sep 11 23:19:36.341822 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:36.341727    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7283a095-48tqb"]
Sep 11 23:19:36.353544 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:36.353511    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7283a095-48tqb"
Sep 11 23:19:36.353904 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:36.353879    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7283a095-48tqb"]
Sep 11 23:19:36.358121 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:36.358097    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vrx7m\""
Sep 11 23:19:36.358389 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:36.358099    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 23:19:36.411913 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:36.411877    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c61abbec-aa52-4356-99d1-9e2c4237c743-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-7283a095-48tqb\" (UID: \"c61abbec-aa52-4356-99d1-9e2c4237c743\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7283a095-48tqb"
Sep 11 23:19:36.512598 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:36.512559    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c61abbec-aa52-4356-99d1-9e2c4237c743-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-7283a095-48tqb\" (UID: \"c61abbec-aa52-4356-99d1-9e2c4237c743\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7283a095-48tqb"
Sep 11 23:19:36.513340 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:36.513315    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c61abbec-aa52-4356-99d1-9e2c4237c743-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-7283a095-48tqb\" (UID: \"c61abbec-aa52-4356-99d1-9e2c4237c743\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7283a095-48tqb"
Sep 11 23:19:36.665430 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:36.665381    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7283a095-48tqb"
Sep 11 23:19:36.942820 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:36.942674    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7283a095-48tqb"]
Sep 11 23:19:36.945756 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:19:36.945729    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc61abbec_aa52_4356_99d1_9e2c4237c743.slice/crio-901d910bcd3150bf8558440dd621aa68e5be0be34d854deac0da7ee35b0640c0 WatchSource:0}: Error finding container 901d910bcd3150bf8558440dd621aa68e5be0be34d854deac0da7ee35b0640c0: Status 404 returned error can't find the container with id 901d910bcd3150bf8558440dd621aa68e5be0be34d854deac0da7ee35b0640c0
Sep 11 23:19:37.004675 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:37.004631    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7283a095-48tqb" event={"ID":"c61abbec-aa52-4356-99d1-9e2c4237c743","Type":"ContainerStarted","Data":"901d910bcd3150bf8558440dd621aa68e5be0be34d854deac0da7ee35b0640c0"}
Sep 11 23:19:38.013442 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:38.013347    2546 generic.go:356] "Generic (PLEG): container finished" podID="c61abbec-aa52-4356-99d1-9e2c4237c743" containerID="2b84fc5efe24af4b23f526eafd61dde7c9cdb697789bf077b21b8dbd59eb21b3" exitCode=0
Sep 11 23:19:38.013818 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:38.013431    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7283a095-48tqb" event={"ID":"c61abbec-aa52-4356-99d1-9e2c4237c743","Type":"ContainerDied","Data":"2b84fc5efe24af4b23f526eafd61dde7c9cdb697789bf077b21b8dbd59eb21b3"}
Sep 11 23:19:39.171364 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:39.171340    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7283a095-48tqb"
Sep 11 23:19:39.239145 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:39.239099    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c61abbec-aa52-4356-99d1-9e2c4237c743-maas-api-service-ca\") pod \"c61abbec-aa52-4356-99d1-9e2c4237c743\" (UID: \"c61abbec-aa52-4356-99d1-9e2c4237c743\") "
Sep 11 23:19:39.239473 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:39.239451    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c61abbec-aa52-4356-99d1-9e2c4237c743-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c61abbec-aa52-4356-99d1-9e2c4237c743" (UID: "c61abbec-aa52-4356-99d1-9e2c4237c743"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 23:19:39.340761 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:39.340668    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c61abbec-aa52-4356-99d1-9e2c4237c743-maas-api-service-ca\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:19:40.027453 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:40.027418    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7283a095-48tqb"
Sep 11 23:19:40.027630 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:40.027447    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7283a095-48tqb" event={"ID":"c61abbec-aa52-4356-99d1-9e2c4237c743","Type":"ContainerDied","Data":"901d910bcd3150bf8558440dd621aa68e5be0be34d854deac0da7ee35b0640c0"}
Sep 11 23:19:40.027630 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:40.027500    2546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="901d910bcd3150bf8558440dd621aa68e5be0be34d854deac0da7ee35b0640c0"
Sep 11 23:19:40.544626 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:40.544589    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-12fb6b/simulator-w0-12fb6b-kserve-9749df7b4-859hf"
Sep 11 23:19:40.545033 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:40.544646    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-12fb6b/simulator-w0-12fb6b-kserve-9749df7b4-859hf"
Sep 11 23:19:40.557500 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:40.557475    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-12fb6b/simulator-w0-12fb6b-kserve-9749df7b4-859hf"
Sep 11 23:19:41.045527 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:41.045491    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-12fb6b/simulator-w0-12fb6b-kserve-9749df7b4-859hf"
Sep 11 23:19:47.667703 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:47.667664    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7283a095-48tqb"]
Sep 11 23:19:47.670263 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:47.670239    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7283a095-48tqb"]
Sep 11 23:19:47.811936 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:47.811903    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql"]
Sep 11 23:19:47.812140 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:47.812116    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql" podUID="ee160166-a67b-436e-acae-2f1dc159cda9" containerName="istio-proxy" containerID="cri-o://7762a00e6ad6fc41c82c8e8c34d8aab907fbc66ced60c347e14757a0ab0808e3" gracePeriod=30
Sep 11 23:19:48.495525 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:48.495489    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c61abbec-aa52-4356-99d1-9e2c4237c743" path="/var/lib/kubelet/pods/c61abbec-aa52-4356-99d1-9e2c4237c743/volumes"
Sep 11 23:19:48.528336 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:48.528293    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-c8789319-openshift-default-7bfc489944-spgfv"]
Sep 11 23:19:48.529042 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:48.529010    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c61abbec-aa52-4356-99d1-9e2c4237c743" containerName="revoke-keys"
Sep 11 23:19:48.529042 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:48.529037    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="c61abbec-aa52-4356-99d1-9e2c4237c743" containerName="revoke-keys"
Sep 11 23:19:48.529272 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:48.529188    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="c61abbec-aa52-4356-99d1-9e2c4237c743" containerName="revoke-keys"
Sep 11 23:19:48.532574 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:48.532552    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c8789319-openshift-default-7bfc489944-spgfv"
Sep 11 23:19:48.535733 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:48.535712    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-c8789319-openshift-default-dockercfg-lxg48\""
Sep 11 23:19:48.548113 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:48.548081    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c8789319-openshift-default-7bfc489944-spgfv"]
Sep 11 23:19:48.634328 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:48.633776    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm77j\" (UniqueName: \"kubernetes.io/projected/c9b43763-2b84-4249-bf71-49a9505341bc-kube-api-access-nm77j\") pod \"e2e-ait-c8789319-openshift-default-7bfc489944-spgfv\" (UID: \"c9b43763-2b84-4249-bf71-49a9505341bc\") " pod="openshift-ingress/e2e-ait-c8789319-openshift-default-7bfc489944-spgfv"
Sep 11 23:19:48.634328 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:48.633826    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9b43763-2b84-4249-bf71-49a9505341bc-credential-socket\") pod \"e2e-ait-c8789319-openshift-default-7bfc489944-spgfv\" (UID: \"c9b43763-2b84-4249-bf71-49a9505341bc\") " pod="openshift-ingress/e2e-ait-c8789319-openshift-default-7bfc489944-spgfv"
Sep 11 23:19:48.634328 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:48.633865    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9b43763-2b84-4249-bf71-49a9505341bc-workload-socket\") pod \"e2e-ait-c8789319-openshift-default-7bfc489944-spgfv\" (UID: \"c9b43763-2b84-4249-bf71-49a9505341bc\") " pod="openshift-ingress/e2e-ait-c8789319-openshift-default-7bfc489944-spgfv"
Sep 11 23:19:48.634328 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:48.633893    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9b43763-2b84-4249-bf71-49a9505341bc-istiod-ca-cert\") pod \"e2e-ait-c8789319-openshift-default-7bfc489944-spgfv\" (UID: \"c9b43763-2b84-4249-bf71-49a9505341bc\") " pod="openshift-ingress/e2e-ait-c8789319-openshift-default-7bfc489944-spgfv"
Sep 11 23:19:48.634328 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:48.633930    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9b43763-2b84-4249-bf71-49a9505341bc-istio-envoy\") pod \"e2e-ait-c8789319-openshift-default-7bfc489944-spgfv\" (UID: \"c9b43763-2b84-4249-bf71-49a9505341bc\") " pod="openshift-ingress/e2e-ait-c8789319-openshift-default-7bfc489944-spgfv"
Sep 11 23:19:48.634328 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:48.633971    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9b43763-2b84-4249-bf71-49a9505341bc-istio-token\") pod \"e2e-ait-c8789319-openshift-default-7bfc489944-spgfv\" (UID: \"c9b43763-2b84-4249-bf71-49a9505341bc\") " pod="openshift-ingress/e2e-ait-c8789319-openshift-default-7bfc489944-spgfv"
Sep 11 23:19:48.634328 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:48.634023    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9b43763-2b84-4249-bf71-49a9505341bc-istio-data\") pod \"e2e-ait-c8789319-openshift-default-7bfc489944-spgfv\" (UID: \"c9b43763-2b84-4249-bf71-49a9505341bc\") " pod="openshift-ingress/e2e-ait-c8789319-openshift-default-7bfc489944-spgfv"
Sep 11 23:19:48.634328 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:48.634116    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9b43763-2b84-4249-bf71-49a9505341bc-istio-podinfo\") pod \"e2e-ait-c8789319-openshift-default-7bfc489944-spgfv\" (UID: \"c9b43763-2b84-4249-bf71-49a9505341bc\") " pod="openshift-ingress/e2e-ait-c8789319-openshift-default-7bfc489944-spgfv"
Sep 11 23:19:48.634328 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:48.634201    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9b43763-2b84-4249-bf71-49a9505341bc-workload-certs\") pod \"e2e-ait-c8789319-openshift-default-7bfc489944-spgfv\" (UID: \"c9b43763-2b84-4249-bf71-49a9505341bc\") " pod="openshift-ingress/e2e-ait-c8789319-openshift-default-7bfc489944-spgfv"
Sep 11 23:19:48.735730 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:48.735691    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9b43763-2b84-4249-bf71-49a9505341bc-workload-socket\") pod \"e2e-ait-c8789319-openshift-default-7bfc489944-spgfv\" (UID: \"c9b43763-2b84-4249-bf71-49a9505341bc\") " pod="openshift-ingress/e2e-ait-c8789319-openshift-default-7bfc489944-spgfv"
Sep 11 23:19:48.735730 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:48.735730    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9b43763-2b84-4249-bf71-49a9505341bc-istiod-ca-cert\") pod \"e2e-ait-c8789319-openshift-default-7bfc489944-spgfv\" (UID: \"c9b43763-2b84-4249-bf71-49a9505341bc\") " pod="openshift-ingress/e2e-ait-c8789319-openshift-default-7bfc489944-spgfv"
Sep 11 23:19:48.736271 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:48.735752    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9b43763-2b84-4249-bf71-49a9505341bc-istio-envoy\") pod \"e2e-ait-c8789319-openshift-default-7bfc489944-spgfv\" (UID: \"c9b43763-2b84-4249-bf71-49a9505341bc\") " pod="openshift-ingress/e2e-ait-c8789319-openshift-default-7bfc489944-spgfv"
Sep 11 23:19:48.736271 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:48.735785    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9b43763-2b84-4249-bf71-49a9505341bc-istio-token\") pod \"e2e-ait-c8789319-openshift-default-7bfc489944-spgfv\" (UID: \"c9b43763-2b84-4249-bf71-49a9505341bc\") " pod="openshift-ingress/e2e-ait-c8789319-openshift-default-7bfc489944-spgfv"
Sep 11 23:19:48.736271 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:48.735830    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9b43763-2b84-4249-bf71-49a9505341bc-istio-data\") pod \"e2e-ait-c8789319-openshift-default-7bfc489944-spgfv\" (UID: \"c9b43763-2b84-4249-bf71-49a9505341bc\") " pod="openshift-ingress/e2e-ait-c8789319-openshift-default-7bfc489944-spgfv"
Sep 11 23:19:48.736271 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:48.735893    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9b43763-2b84-4249-bf71-49a9505341bc-istio-podinfo\") pod \"e2e-ait-c8789319-openshift-default-7bfc489944-spgfv\" (UID: \"c9b43763-2b84-4249-bf71-49a9505341bc\") " pod="openshift-ingress/e2e-ait-c8789319-openshift-default-7bfc489944-spgfv"
Sep 11 23:19:48.736271 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:48.735946    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9b43763-2b84-4249-bf71-49a9505341bc-workload-certs\") pod \"e2e-ait-c8789319-openshift-default-7bfc489944-spgfv\" (UID: \"c9b43763-2b84-4249-bf71-49a9505341bc\") " pod="openshift-ingress/e2e-ait-c8789319-openshift-default-7bfc489944-spgfv"
Sep 11 23:19:48.736271 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:48.736061    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm77j\" (UniqueName: \"kubernetes.io/projected/c9b43763-2b84-4249-bf71-49a9505341bc-kube-api-access-nm77j\") pod \"e2e-ait-c8789319-openshift-default-7bfc489944-spgfv\" (UID: \"c9b43763-2b84-4249-bf71-49a9505341bc\") " pod="openshift-ingress/e2e-ait-c8789319-openshift-default-7bfc489944-spgfv"
Sep 11 23:19:48.736271 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:48.736098    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9b43763-2b84-4249-bf71-49a9505341bc-credential-socket\") pod \"e2e-ait-c8789319-openshift-default-7bfc489944-spgfv\" (UID: \"c9b43763-2b84-4249-bf71-49a9505341bc\") " pod="openshift-ingress/e2e-ait-c8789319-openshift-default-7bfc489944-spgfv"
Sep 11 23:19:48.736271 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:48.736138    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9b43763-2b84-4249-bf71-49a9505341bc-workload-socket\") pod \"e2e-ait-c8789319-openshift-default-7bfc489944-spgfv\" (UID: \"c9b43763-2b84-4249-bf71-49a9505341bc\") " pod="openshift-ingress/e2e-ait-c8789319-openshift-default-7bfc489944-spgfv"
Sep 11 23:19:48.736642 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:48.736486    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9b43763-2b84-4249-bf71-49a9505341bc-istio-data\") pod \"e2e-ait-c8789319-openshift-default-7bfc489944-spgfv\" (UID: \"c9b43763-2b84-4249-bf71-49a9505341bc\") " pod="openshift-ingress/e2e-ait-c8789319-openshift-default-7bfc489944-spgfv"
Sep 11 23:19:48.736642 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:48.736504    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9b43763-2b84-4249-bf71-49a9505341bc-workload-certs\") pod \"e2e-ait-c8789319-openshift-default-7bfc489944-spgfv\" (UID: \"c9b43763-2b84-4249-bf71-49a9505341bc\") " pod="openshift-ingress/e2e-ait-c8789319-openshift-default-7bfc489944-spgfv"
Sep 11 23:19:48.736642 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:48.736535    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9b43763-2b84-4249-bf71-49a9505341bc-credential-socket\") pod \"e2e-ait-c8789319-openshift-default-7bfc489944-spgfv\" (UID: \"c9b43763-2b84-4249-bf71-49a9505341bc\") " pod="openshift-ingress/e2e-ait-c8789319-openshift-default-7bfc489944-spgfv"
Sep 11 23:19:48.736642 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:48.736605    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9b43763-2b84-4249-bf71-49a9505341bc-istiod-ca-cert\") pod \"e2e-ait-c8789319-openshift-default-7bfc489944-spgfv\" (UID: \"c9b43763-2b84-4249-bf71-49a9505341bc\") " pod="openshift-ingress/e2e-ait-c8789319-openshift-default-7bfc489944-spgfv"
Sep 11 23:19:48.738975 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:48.738943    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9b43763-2b84-4249-bf71-49a9505341bc-istio-envoy\") pod \"e2e-ait-c8789319-openshift-default-7bfc489944-spgfv\" (UID: \"c9b43763-2b84-4249-bf71-49a9505341bc\") " pod="openshift-ingress/e2e-ait-c8789319-openshift-default-7bfc489944-spgfv"
Sep 11 23:19:48.739081 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:48.739056    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9b43763-2b84-4249-bf71-49a9505341bc-istio-podinfo\") pod \"e2e-ait-c8789319-openshift-default-7bfc489944-spgfv\" (UID: \"c9b43763-2b84-4249-bf71-49a9505341bc\") " pod="openshift-ingress/e2e-ait-c8789319-openshift-default-7bfc489944-spgfv"
Sep 11 23:19:48.744662 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:48.744638    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm77j\" (UniqueName: \"kubernetes.io/projected/c9b43763-2b84-4249-bf71-49a9505341bc-kube-api-access-nm77j\") pod \"e2e-ait-c8789319-openshift-default-7bfc489944-spgfv\" (UID: \"c9b43763-2b84-4249-bf71-49a9505341bc\") " pod="openshift-ingress/e2e-ait-c8789319-openshift-default-7bfc489944-spgfv"
Sep 11 23:19:48.744772 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:48.744755    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9b43763-2b84-4249-bf71-49a9505341bc-istio-token\") pod \"e2e-ait-c8789319-openshift-default-7bfc489944-spgfv\" (UID: \"c9b43763-2b84-4249-bf71-49a9505341bc\") " pod="openshift-ingress/e2e-ait-c8789319-openshift-default-7bfc489944-spgfv"
Sep 11 23:19:48.843487 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:48.843406    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c8789319-openshift-default-7bfc489944-spgfv"
Sep 11 23:19:49.136703 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:49.136677    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c8789319-openshift-default-7bfc489944-spgfv"]
Sep 11 23:19:49.138481 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:19:49.138453    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9b43763_2b84_4249_bf71_49a9505341bc.slice/crio-0f3ef051ea5b26c5bef6751ecc264f111347c7778fd8ea2813f911bcdb1f3f10 WatchSource:0}: Error finding container 0f3ef051ea5b26c5bef6751ecc264f111347c7778fd8ea2813f911bcdb1f3f10: Status 404 returned error can't find the container with id 0f3ef051ea5b26c5bef6751ecc264f111347c7778fd8ea2813f911bcdb1f3f10
Sep 11 23:19:49.140814 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:49.140779    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:19:49.140930 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:49.140847    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:19:49.140930 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:49.140876    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:19:50.013427 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:50.013386    2546 patch_prober.go:28] interesting pod/e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Sep 11 23:19:50.013798 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:50.013452    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql" podUID="ee160166-a67b-436e-acae-2f1dc159cda9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 23:19:50.097672 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:50.097635    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c8789319-openshift-default-7bfc489944-spgfv" event={"ID":"c9b43763-2b84-4249-bf71-49a9505341bc","Type":"ContainerStarted","Data":"c5882d204890b95ae3eba20ade5b4a753d6cb5d34d15f41425b88228990d0909"}
Sep 11 23:19:50.097847 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:50.097685    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c8789319-openshift-default-7bfc489944-spgfv" event={"ID":"c9b43763-2b84-4249-bf71-49a9505341bc","Type":"ContainerStarted","Data":"0f3ef051ea5b26c5bef6751ecc264f111347c7778fd8ea2813f911bcdb1f3f10"}
Sep 11 23:19:50.121559 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:50.121511    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-c8789319-openshift-default-7bfc489944-spgfv" podStartSLOduration=2.1214967160000002 podStartE2EDuration="2.121496716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 23:19:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:19:50.119573514 +0000 UTC m=+1114.152702729" watchObservedRunningTime="2026-09-11 23:19:50.121496716 +0000 UTC m=+1114.154625987"
Sep 11 23:19:50.844609 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:50.844569    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c8789319-openshift-default-7bfc489944-spgfv"
Sep 11 23:19:50.851416 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:50.851384    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c8789319-openshift-default-7bfc489944-spgfv"
Sep 11 23:19:51.104815 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:51.104731    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c8789319-openshift-default-7bfc489944-spgfv"
Sep 11 23:19:51.105758 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:51.105738    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c8789319-openshift-default-7bfc489944-spgfv"
Sep 11 23:19:53.066663 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:53.066639    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql"
Sep 11 23:19:53.120394 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:53.120296    2546 generic.go:356] "Generic (PLEG): container finished" podID="ee160166-a67b-436e-acae-2f1dc159cda9" containerID="7762a00e6ad6fc41c82c8e8c34d8aab907fbc66ced60c347e14757a0ab0808e3" exitCode=0
Sep 11 23:19:53.120394 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:53.120355    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql"
Sep 11 23:19:53.120612 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:53.120384    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql" event={"ID":"ee160166-a67b-436e-acae-2f1dc159cda9","Type":"ContainerDied","Data":"7762a00e6ad6fc41c82c8e8c34d8aab907fbc66ced60c347e14757a0ab0808e3"}
Sep 11 23:19:53.120612 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:53.120442    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql" event={"ID":"ee160166-a67b-436e-acae-2f1dc159cda9","Type":"ContainerDied","Data":"0801fe21ff052da6057bcbe373a47a37be9ea53d99229a4edd40fc896b595246"}
Sep 11 23:19:53.120612 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:53.120467    2546 scope.go:117] "RemoveContainer" containerID="7762a00e6ad6fc41c82c8e8c34d8aab907fbc66ced60c347e14757a0ab0808e3"
Sep 11 23:19:53.130908 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:53.130883    2546 scope.go:117] "RemoveContainer" containerID="fb80805f2b77d5084b81b23e012c4fa0c7e6d2838ffe4870c6a912499fefe464"
Sep 11 23:19:53.140711 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:53.140691    2546 scope.go:117] "RemoveContainer" containerID="7762a00e6ad6fc41c82c8e8c34d8aab907fbc66ced60c347e14757a0ab0808e3"
Sep 11 23:19:53.141005 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:19:53.140985    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7762a00e6ad6fc41c82c8e8c34d8aab907fbc66ced60c347e14757a0ab0808e3\": container with ID starting with 7762a00e6ad6fc41c82c8e8c34d8aab907fbc66ced60c347e14757a0ab0808e3 not found: ID does not exist" containerID="7762a00e6ad6fc41c82c8e8c34d8aab907fbc66ced60c347e14757a0ab0808e3"
Sep 11 23:19:53.141061 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:53.141014    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7762a00e6ad6fc41c82c8e8c34d8aab907fbc66ced60c347e14757a0ab0808e3"} err="failed to get container status \"7762a00e6ad6fc41c82c8e8c34d8aab907fbc66ced60c347e14757a0ab0808e3\": rpc error: code = NotFound desc = could not find container \"7762a00e6ad6fc41c82c8e8c34d8aab907fbc66ced60c347e14757a0ab0808e3\": container with ID starting with 7762a00e6ad6fc41c82c8e8c34d8aab907fbc66ced60c347e14757a0ab0808e3 not found: ID does not exist"
Sep 11 23:19:53.141061 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:53.141032    2546 scope.go:117] "RemoveContainer" containerID="fb80805f2b77d5084b81b23e012c4fa0c7e6d2838ffe4870c6a912499fefe464"
Sep 11 23:19:53.141327 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:19:53.141307    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb80805f2b77d5084b81b23e012c4fa0c7e6d2838ffe4870c6a912499fefe464\": container with ID starting with fb80805f2b77d5084b81b23e012c4fa0c7e6d2838ffe4870c6a912499fefe464 not found: ID does not exist" containerID="fb80805f2b77d5084b81b23e012c4fa0c7e6d2838ffe4870c6a912499fefe464"
Sep 11 23:19:53.141413 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:53.141333    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb80805f2b77d5084b81b23e012c4fa0c7e6d2838ffe4870c6a912499fefe464"} err="failed to get container status \"fb80805f2b77d5084b81b23e012c4fa0c7e6d2838ffe4870c6a912499fefe464\": rpc error: code = NotFound desc = could not find container \"fb80805f2b77d5084b81b23e012c4fa0c7e6d2838ffe4870c6a912499fefe464\": container with ID starting with fb80805f2b77d5084b81b23e012c4fa0c7e6d2838ffe4870c6a912499fefe464 not found: ID does not exist"
Sep 11 23:19:53.186644 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:53.186617    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee160166-a67b-436e-acae-2f1dc159cda9-istio-podinfo\") pod \"ee160166-a67b-436e-acae-2f1dc159cda9\" (UID: \"ee160166-a67b-436e-acae-2f1dc159cda9\") "
Sep 11 23:19:53.186830 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:53.186684    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee160166-a67b-436e-acae-2f1dc159cda9-workload-certs\") pod \"ee160166-a67b-436e-acae-2f1dc159cda9\" (UID: \"ee160166-a67b-436e-acae-2f1dc159cda9\") "
Sep 11 23:19:53.186830 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:53.186722    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee160166-a67b-436e-acae-2f1dc159cda9-istiod-ca-cert\") pod \"ee160166-a67b-436e-acae-2f1dc159cda9\" (UID: \"ee160166-a67b-436e-acae-2f1dc159cda9\") "
Sep 11 23:19:53.186830 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:53.186742    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee160166-a67b-436e-acae-2f1dc159cda9-workload-socket\") pod \"ee160166-a67b-436e-acae-2f1dc159cda9\" (UID: \"ee160166-a67b-436e-acae-2f1dc159cda9\") "
Sep 11 23:19:53.186830 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:53.186802    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee160166-a67b-436e-acae-2f1dc159cda9-istio-envoy\") pod \"ee160166-a67b-436e-acae-2f1dc159cda9\" (UID: \"ee160166-a67b-436e-acae-2f1dc159cda9\") "
Sep 11 23:19:53.186830 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:53.186823    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee160166-a67b-436e-acae-2f1dc159cda9-istio-token\") pod \"ee160166-a67b-436e-acae-2f1dc159cda9\" (UID: \"ee160166-a67b-436e-acae-2f1dc159cda9\") "
Sep 11 23:19:53.187083 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:53.186908    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee160166-a67b-436e-acae-2f1dc159cda9-credential-socket\") pod \"ee160166-a67b-436e-acae-2f1dc159cda9\" (UID: \"ee160166-a67b-436e-acae-2f1dc159cda9\") "
Sep 11 23:19:53.187083 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:53.186940    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee160166-a67b-436e-acae-2f1dc159cda9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ee160166-a67b-436e-acae-2f1dc159cda9" (UID: "ee160166-a67b-436e-acae-2f1dc159cda9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:19:53.187083 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:53.186952    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4tpn7\" (UniqueName: \"kubernetes.io/projected/ee160166-a67b-436e-acae-2f1dc159cda9-kube-api-access-4tpn7\") pod \"ee160166-a67b-436e-acae-2f1dc159cda9\" (UID: \"ee160166-a67b-436e-acae-2f1dc159cda9\") "
Sep 11 23:19:53.187083 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:53.187026    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee160166-a67b-436e-acae-2f1dc159cda9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ee160166-a67b-436e-acae-2f1dc159cda9" (UID: "ee160166-a67b-436e-acae-2f1dc159cda9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:19:53.187083 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:53.187057    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee160166-a67b-436e-acae-2f1dc159cda9-istio-data\") pod \"ee160166-a67b-436e-acae-2f1dc159cda9\" (UID: \"ee160166-a67b-436e-acae-2f1dc159cda9\") "
Sep 11 23:19:53.187385 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:53.187129    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee160166-a67b-436e-acae-2f1dc159cda9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ee160166-a67b-436e-acae-2f1dc159cda9" (UID: "ee160166-a67b-436e-acae-2f1dc159cda9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 23:19:53.187385 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:53.187275    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee160166-a67b-436e-acae-2f1dc159cda9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ee160166-a67b-436e-acae-2f1dc159cda9" (UID: "ee160166-a67b-436e-acae-2f1dc159cda9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:19:53.187496 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:53.187441    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee160166-a67b-436e-acae-2f1dc159cda9-istio-data" (OuterVolumeSpecName: "istio-data") pod "ee160166-a67b-436e-acae-2f1dc159cda9" (UID: "ee160166-a67b-436e-acae-2f1dc159cda9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:19:53.187883 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:53.187856    2546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee160166-a67b-436e-acae-2f1dc159cda9-istio-data\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:19:53.188025 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:53.188004    2546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee160166-a67b-436e-acae-2f1dc159cda9-workload-certs\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:19:53.188103 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:53.188029    2546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee160166-a67b-436e-acae-2f1dc159cda9-istiod-ca-cert\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:19:53.188103 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:53.188042    2546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee160166-a67b-436e-acae-2f1dc159cda9-workload-socket\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:19:53.188103 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:53.188069    2546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee160166-a67b-436e-acae-2f1dc159cda9-credential-socket\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:19:53.189307 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:53.189287    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee160166-a67b-436e-acae-2f1dc159cda9-kube-api-access-4tpn7" (OuterVolumeSpecName: "kube-api-access-4tpn7") pod "ee160166-a67b-436e-acae-2f1dc159cda9" (UID: "ee160166-a67b-436e-acae-2f1dc159cda9"). InnerVolumeSpecName "kube-api-access-4tpn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 23:19:53.189385 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:53.189349    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee160166-a67b-436e-acae-2f1dc159cda9-istio-token" (OuterVolumeSpecName: "istio-token") pod "ee160166-a67b-436e-acae-2f1dc159cda9" (UID: "ee160166-a67b-436e-acae-2f1dc159cda9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 23:19:53.189522 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:53.189507    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee160166-a67b-436e-acae-2f1dc159cda9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ee160166-a67b-436e-acae-2f1dc159cda9" (UID: "ee160166-a67b-436e-acae-2f1dc159cda9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 23:19:53.189562 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:53.189525    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee160166-a67b-436e-acae-2f1dc159cda9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ee160166-a67b-436e-acae-2f1dc159cda9" (UID: "ee160166-a67b-436e-acae-2f1dc159cda9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:19:53.289568 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:53.289529    2546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee160166-a67b-436e-acae-2f1dc159cda9-istio-podinfo\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:19:53.289568 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:53.289563    2546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee160166-a67b-436e-acae-2f1dc159cda9-istio-envoy\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:19:53.289568 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:53.289574    2546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee160166-a67b-436e-acae-2f1dc159cda9-istio-token\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:19:53.289809 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:53.289594    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tpn7\" (UniqueName: \"kubernetes.io/projected/ee160166-a67b-436e-acae-2f1dc159cda9-kube-api-access-4tpn7\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:19:53.444611 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:53.444583    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql"]
Sep 11 23:19:53.448494 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:53.448470    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-7283a095-gw-openshift-default-68b7778dd4-7cqql"]
Sep 11 23:19:54.389438 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:54.389406    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"]
Sep 11 23:19:54.495423 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:19:54.495385    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee160166-a67b-436e-acae-2f1dc159cda9" path="/var/lib/kubelet/pods/ee160166-a67b-436e-acae-2f1dc159cda9/volumes"
Sep 11 23:20:13.486480 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:13.486437    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"]
Sep 11 23:20:14.291030 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:14.290995    2546 generic.go:356] "Generic (PLEG): container finished" podID="c9b43763-2b84-4249-bf71-49a9505341bc" containerID="c5882d204890b95ae3eba20ade5b4a753d6cb5d34d15f41425b88228990d0909" exitCode=0
Sep 11 23:20:14.291229 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:14.291052    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c8789319-openshift-default-7bfc489944-spgfv" event={"ID":"c9b43763-2b84-4249-bf71-49a9505341bc","Type":"ContainerDied","Data":"c5882d204890b95ae3eba20ade5b4a753d6cb5d34d15f41425b88228990d0909"}
Sep 11 23:20:14.291484 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:14.291470    2546 scope.go:117] "RemoveContainer" containerID="c5882d204890b95ae3eba20ade5b4a753d6cb5d34d15f41425b88228990d0909"
Sep 11 23:20:14.292610 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:14.292583    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:20:14.292668 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:14.292653    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:20:14.292704 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:14.292681    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:20:15.299697 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:15.299661    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c8789319-openshift-default-7bfc489944-spgfv" event={"ID":"c9b43763-2b84-4249-bf71-49a9505341bc","Type":"ContainerStarted","Data":"1512c36615eee64859f40789826b99498be5fc26d01285790b2e668c267a9ed7"}
Sep 11 23:20:15.844091 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:15.844045    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c8789319-openshift-default-7bfc489944-spgfv"
Sep 11 23:20:15.844510 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:15.844477    2546 patch_prober.go:28] interesting pod/e2e-ait-c8789319-openshift-default-7bfc489944-spgfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Sep 11 23:20:15.844622 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:15.844552    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c8789319-openshift-default-7bfc489944-spgfv" podUID="c9b43763-2b84-4249-bf71-49a9505341bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Sep 11 23:20:16.845626 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:16.845520    2546 patch_prober.go:28] interesting pod/e2e-ait-c8789319-openshift-default-7bfc489944-spgfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Sep 11 23:20:16.845626 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:16.845587    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c8789319-openshift-default-7bfc489944-spgfv" podUID="c9b43763-2b84-4249-bf71-49a9505341bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Sep 11 23:20:17.109772 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:17.109685    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c8789319-x2kzf"]
Sep 11 23:20:17.110304 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:17.110284    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee160166-a67b-436e-acae-2f1dc159cda9" containerName="istio-proxy"
Sep 11 23:20:17.110304 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:17.110305    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee160166-a67b-436e-acae-2f1dc159cda9" containerName="istio-proxy"
Sep 11 23:20:17.110467 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:17.110414    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee160166-a67b-436e-acae-2f1dc159cda9" containerName="istio-proxy"
Sep 11 23:20:17.110467 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:17.110433    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee160166-a67b-436e-acae-2f1dc159cda9" containerName="istio-proxy"
Sep 11 23:20:17.114974 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:17.114953    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c8789319-x2kzf"
Sep 11 23:20:17.118556 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:17.118535    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vrx7m\""
Sep 11 23:20:17.118768 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:17.118602    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 23:20:17.123684 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:17.123626    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c8789319-x2kzf"]
Sep 11 23:20:17.254809 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:17.254778    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/182d05c2-b55c-4257-a3e0-4bf120668165-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-c8789319-x2kzf\" (UID: \"182d05c2-b55c-4257-a3e0-4bf120668165\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c8789319-x2kzf"
Sep 11 23:20:17.356102 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:17.356063    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/182d05c2-b55c-4257-a3e0-4bf120668165-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-c8789319-x2kzf\" (UID: \"182d05c2-b55c-4257-a3e0-4bf120668165\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c8789319-x2kzf"
Sep 11 23:20:17.356708 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:17.356687    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/182d05c2-b55c-4257-a3e0-4bf120668165-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-c8789319-x2kzf\" (UID: \"182d05c2-b55c-4257-a3e0-4bf120668165\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c8789319-x2kzf"
Sep 11 23:20:17.427461 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:17.427424    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c8789319-x2kzf"
Sep 11 23:20:17.714379 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:17.714346    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c8789319-x2kzf"]
Sep 11 23:20:17.715133 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:20:17.715094    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod182d05c2_b55c_4257_a3e0_4bf120668165.slice/crio-ff420bed7caa429dea80bb781d6506f575849d70fc752bb1ffd392bf249fa9e3 WatchSource:0}: Error finding container ff420bed7caa429dea80bb781d6506f575849d70fc752bb1ffd392bf249fa9e3: Status 404 returned error can't find the container with id ff420bed7caa429dea80bb781d6506f575849d70fc752bb1ffd392bf249fa9e3
Sep 11 23:20:17.844142 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:17.844096    2546 patch_prober.go:28] interesting pod/e2e-ait-c8789319-openshift-default-7bfc489944-spgfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Sep 11 23:20:17.844356 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:17.844185    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c8789319-openshift-default-7bfc489944-spgfv" podUID="c9b43763-2b84-4249-bf71-49a9505341bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Sep 11 23:20:18.328865 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:18.328668    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c8789319-x2kzf" event={"ID":"182d05c2-b55c-4257-a3e0-4bf120668165","Type":"ContainerStarted","Data":"c66c36a1680cab71d0b006ee4a574a011b2f00b51689f6733d52bdbf84b0f965"}
Sep 11 23:20:18.328865 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:18.328739    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c8789319-x2kzf" event={"ID":"182d05c2-b55c-4257-a3e0-4bf120668165","Type":"ContainerStarted","Data":"ff420bed7caa429dea80bb781d6506f575849d70fc752bb1ffd392bf249fa9e3"}
Sep 11 23:20:18.844708 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:18.844666    2546 patch_prober.go:28] interesting pod/e2e-ait-c8789319-openshift-default-7bfc489944-spgfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Sep 11 23:20:18.844979 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:18.844740    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c8789319-openshift-default-7bfc489944-spgfv" podUID="c9b43763-2b84-4249-bf71-49a9505341bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Sep 11 23:20:19.342091 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:19.342045    2546 generic.go:356] "Generic (PLEG): container finished" podID="182d05c2-b55c-4257-a3e0-4bf120668165" containerID="c66c36a1680cab71d0b006ee4a574a011b2f00b51689f6733d52bdbf84b0f965" exitCode=0
Sep 11 23:20:19.342621 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:19.342184    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c8789319-x2kzf" event={"ID":"182d05c2-b55c-4257-a3e0-4bf120668165","Type":"ContainerDied","Data":"c66c36a1680cab71d0b006ee4a574a011b2f00b51689f6733d52bdbf84b0f965"}
Sep 11 23:20:19.844709 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:19.844663    2546 patch_prober.go:28] interesting pod/e2e-ait-c8789319-openshift-default-7bfc489944-spgfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Sep 11 23:20:19.844934 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:19.844752    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c8789319-openshift-default-7bfc489944-spgfv" podUID="c9b43763-2b84-4249-bf71-49a9505341bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Sep 11 23:20:20.493310 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:20.493284    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c8789319-x2kzf"
Sep 11 23:20:20.589688 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:20.589650    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/182d05c2-b55c-4257-a3e0-4bf120668165-maas-api-service-ca\") pod \"182d05c2-b55c-4257-a3e0-4bf120668165\" (UID: \"182d05c2-b55c-4257-a3e0-4bf120668165\") "
Sep 11 23:20:20.590088 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:20.590042    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/182d05c2-b55c-4257-a3e0-4bf120668165-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "182d05c2-b55c-4257-a3e0-4bf120668165" (UID: "182d05c2-b55c-4257-a3e0-4bf120668165"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 23:20:20.691303 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:20.691269    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/182d05c2-b55c-4257-a3e0-4bf120668165-maas-api-service-ca\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:20:20.848135 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:20.848100    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c8789319-openshift-default-7bfc489944-spgfv"
Sep 11 23:20:20.848479 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:20.848460    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c8789319-openshift-default-7bfc489944-spgfv"
Sep 11 23:20:20.849240 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:20.849223    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c8789319-openshift-default-7bfc489944-spgfv"
Sep 11 23:20:21.359086 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:21.359046    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c8789319-x2kzf"
Sep 11 23:20:21.359294 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:21.359081    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c8789319-x2kzf" event={"ID":"182d05c2-b55c-4257-a3e0-4bf120668165","Type":"ContainerDied","Data":"ff420bed7caa429dea80bb781d6506f575849d70fc752bb1ffd392bf249fa9e3"}
Sep 11 23:20:21.359294 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:21.359140    2546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff420bed7caa429dea80bb781d6506f575849d70fc752bb1ffd392bf249fa9e3"
Sep 11 23:20:23.368723 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:23.368686    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c8789319-x2kzf"]
Sep 11 23:20:23.373512 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:23.373482    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c8789319-x2kzf"]
Sep 11 23:20:24.429875 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:24.429839    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c8789319-openshift-default-7bfc489944-spgfv"]
Sep 11 23:20:24.430339 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:24.430099    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-c8789319-openshift-default-7bfc489944-spgfv" podUID="c9b43763-2b84-4249-bf71-49a9505341bc" containerName="istio-proxy" containerID="cri-o://1512c36615eee64859f40789826b99498be5fc26d01285790b2e668c267a9ed7" gracePeriod=30
Sep 11 23:20:24.495239 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:24.495209    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="182d05c2-b55c-4257-a3e0-4bf120668165" path="/var/lib/kubelet/pods/182d05c2-b55c-4257-a3e0-4bf120668165/volumes"
Sep 11 23:20:25.065259 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:25.065220    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0a7a66-5ltxw"]
Sep 11 23:20:25.065751 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:25.065732    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="182d05c2-b55c-4257-a3e0-4bf120668165" containerName="revoke-keys"
Sep 11 23:20:25.065847 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:25.065755    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="182d05c2-b55c-4257-a3e0-4bf120668165" containerName="revoke-keys"
Sep 11 23:20:25.065847 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:25.065794    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee160166-a67b-436e-acae-2f1dc159cda9" containerName="istio-proxy"
Sep 11 23:20:25.065847 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:25.065802    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee160166-a67b-436e-acae-2f1dc159cda9" containerName="istio-proxy"
Sep 11 23:20:25.065997 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:25.065914    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="182d05c2-b55c-4257-a3e0-4bf120668165" containerName="revoke-keys"
Sep 11 23:20:25.069232 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:25.069212    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0a7a66-5ltxw"
Sep 11 23:20:25.072092 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:25.072064    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 23:20:25.072223 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:25.072096    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vrx7m\""
Sep 11 23:20:25.077625 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:25.077600    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0a7a66-5ltxw"]
Sep 11 23:20:25.136626 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:25.136589    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a6a8cdf-f22f-4c18-881d-3084b3ef1c22-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-0a7a66-5ltxw\" (UID: \"0a6a8cdf-f22f-4c18-881d-3084b3ef1c22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0a7a66-5ltxw"
Sep 11 23:20:25.193847 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:25.193817    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-7e6fa6af-openshift-default-5587bc9748-srqxx"]
Sep 11 23:20:25.197430 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:25.197412    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-7e6fa6af-openshift-default-5587bc9748-srqxx"
Sep 11 23:20:25.204682 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:25.204663    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-7e6fa6af-openshift-default-dockercfg-lc9c7\""
Sep 11 23:20:25.215901 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:25.215871    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-7e6fa6af-openshift-default-5587bc9748-srqxx"]
Sep 11 23:20:25.238297 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:25.238265    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a6a8cdf-f22f-4c18-881d-3084b3ef1c22-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-0a7a66-5ltxw\" (UID: \"0a6a8cdf-f22f-4c18-881d-3084b3ef1c22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0a7a66-5ltxw"
Sep 11 23:20:25.239089 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:25.239065    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a6a8cdf-f22f-4c18-881d-3084b3ef1c22-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-0a7a66-5ltxw\" (UID: \"0a6a8cdf-f22f-4c18-881d-3084b3ef1c22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0a7a66-5ltxw"
Sep 11 23:20:25.339875 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:25.339775    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71dd368e-c415-465c-838a-ecd215bc4069-istiod-ca-cert\") pod \"e2e-derive-7e6fa6af-openshift-default-5587bc9748-srqxx\" (UID: \"71dd368e-c415-465c-838a-ecd215bc4069\") " pod="openshift-ingress/e2e-derive-7e6fa6af-openshift-default-5587bc9748-srqxx"
Sep 11 23:20:25.339875 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:25.339821    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71dd368e-c415-465c-838a-ecd215bc4069-credential-socket\") pod \"e2e-derive-7e6fa6af-openshift-default-5587bc9748-srqxx\" (UID: \"71dd368e-c415-465c-838a-ecd215bc4069\") " pod="openshift-ingress/e2e-derive-7e6fa6af-openshift-default-5587bc9748-srqxx"
Sep 11 23:20:25.339875 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:25.339840    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71dd368e-c415-465c-838a-ecd215bc4069-istio-data\") pod \"e2e-derive-7e6fa6af-openshift-default-5587bc9748-srqxx\" (UID: \"71dd368e-c415-465c-838a-ecd215bc4069\") " pod="openshift-ingress/e2e-derive-7e6fa6af-openshift-default-5587bc9748-srqxx"
Sep 11 23:20:25.339875 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:25.339862    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71dd368e-c415-465c-838a-ecd215bc4069-istio-token\") pod \"e2e-derive-7e6fa6af-openshift-default-5587bc9748-srqxx\" (UID: \"71dd368e-c415-465c-838a-ecd215bc4069\") " pod="openshift-ingress/e2e-derive-7e6fa6af-openshift-default-5587bc9748-srqxx"
Sep 11 23:20:25.339875 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:25.339881    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71dd368e-c415-465c-838a-ecd215bc4069-istio-envoy\") pod \"e2e-derive-7e6fa6af-openshift-default-5587bc9748-srqxx\" (UID: \"71dd368e-c415-465c-838a-ecd215bc4069\") " pod="openshift-ingress/e2e-derive-7e6fa6af-openshift-default-5587bc9748-srqxx"
Sep 11 23:20:25.340250 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:25.339897    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71dd368e-c415-465c-838a-ecd215bc4069-istio-podinfo\") pod \"e2e-derive-7e6fa6af-openshift-default-5587bc9748-srqxx\" (UID: \"71dd368e-c415-465c-838a-ecd215bc4069\") " pod="openshift-ingress/e2e-derive-7e6fa6af-openshift-default-5587bc9748-srqxx"
Sep 11 23:20:25.340250 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:25.339932    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l9mk\" (UniqueName: \"kubernetes.io/projected/71dd368e-c415-465c-838a-ecd215bc4069-kube-api-access-7l9mk\") pod \"e2e-derive-7e6fa6af-openshift-default-5587bc9748-srqxx\" (UID: \"71dd368e-c415-465c-838a-ecd215bc4069\") " pod="openshift-ingress/e2e-derive-7e6fa6af-openshift-default-5587bc9748-srqxx"
Sep 11 23:20:25.340250 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:25.340057    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71dd368e-c415-465c-838a-ecd215bc4069-workload-socket\") pod \"e2e-derive-7e6fa6af-openshift-default-5587bc9748-srqxx\" (UID: \"71dd368e-c415-465c-838a-ecd215bc4069\") " pod="openshift-ingress/e2e-derive-7e6fa6af-openshift-default-5587bc9748-srqxx"
Sep 11 23:20:25.340250 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:25.340099    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71dd368e-c415-465c-838a-ecd215bc4069-workload-certs\") pod \"e2e-derive-7e6fa6af-openshift-default-5587bc9748-srqxx\" (UID: \"71dd368e-c415-465c-838a-ecd215bc4069\") " pod="openshift-ingress/e2e-derive-7e6fa6af-openshift-default-5587bc9748-srqxx"
Sep 11 23:20:25.381109 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:25.381078    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0a7a66-5ltxw"
Sep 11 23:20:25.440856 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:25.440827    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l9mk\" (UniqueName: \"kubernetes.io/projected/71dd368e-c415-465c-838a-ecd215bc4069-kube-api-access-7l9mk\") pod \"e2e-derive-7e6fa6af-openshift-default-5587bc9748-srqxx\" (UID: \"71dd368e-c415-465c-838a-ecd215bc4069\") " pod="openshift-ingress/e2e-derive-7e6fa6af-openshift-default-5587bc9748-srqxx"
Sep 11 23:20:25.441279 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:25.440897    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71dd368e-c415-465c-838a-ecd215bc4069-workload-socket\") pod \"e2e-derive-7e6fa6af-openshift-default-5587bc9748-srqxx\" (UID: \"71dd368e-c415-465c-838a-ecd215bc4069\") " pod="openshift-ingress/e2e-derive-7e6fa6af-openshift-default-5587bc9748-srqxx"
Sep 11 23:20:25.441279 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:25.440953    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71dd368e-c415-465c-838a-ecd215bc4069-workload-certs\") pod \"e2e-derive-7e6fa6af-openshift-default-5587bc9748-srqxx\" (UID: \"71dd368e-c415-465c-838a-ecd215bc4069\") " pod="openshift-ingress/e2e-derive-7e6fa6af-openshift-default-5587bc9748-srqxx"
Sep 11 23:20:25.441279 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:25.441035    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71dd368e-c415-465c-838a-ecd215bc4069-istiod-ca-cert\") pod \"e2e-derive-7e6fa6af-openshift-default-5587bc9748-srqxx\" (UID: \"71dd368e-c415-465c-838a-ecd215bc4069\") " pod="openshift-ingress/e2e-derive-7e6fa6af-openshift-default-5587bc9748-srqxx"
Sep 11 23:20:25.441279 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:25.441074    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71dd368e-c415-465c-838a-ecd215bc4069-credential-socket\") pod \"e2e-derive-7e6fa6af-openshift-default-5587bc9748-srqxx\" (UID: \"71dd368e-c415-465c-838a-ecd215bc4069\") " pod="openshift-ingress/e2e-derive-7e6fa6af-openshift-default-5587bc9748-srqxx"
Sep 11 23:20:25.441279 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:25.441102    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71dd368e-c415-465c-838a-ecd215bc4069-istio-data\") pod \"e2e-derive-7e6fa6af-openshift-default-5587bc9748-srqxx\" (UID: \"71dd368e-c415-465c-838a-ecd215bc4069\") " pod="openshift-ingress/e2e-derive-7e6fa6af-openshift-default-5587bc9748-srqxx"
Sep 11 23:20:25.441279 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:25.441139    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71dd368e-c415-465c-838a-ecd215bc4069-istio-token\") pod \"e2e-derive-7e6fa6af-openshift-default-5587bc9748-srqxx\" (UID: \"71dd368e-c415-465c-838a-ecd215bc4069\") " pod="openshift-ingress/e2e-derive-7e6fa6af-openshift-default-5587bc9748-srqxx"
Sep 11 23:20:25.441279 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:25.441260    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71dd368e-c415-465c-838a-ecd215bc4069-istio-envoy\") pod \"e2e-derive-7e6fa6af-openshift-default-5587bc9748-srqxx\" (UID: \"71dd368e-c415-465c-838a-ecd215bc4069\") " pod="openshift-ingress/e2e-derive-7e6fa6af-openshift-default-5587bc9748-srqxx"
Sep 11 23:20:25.441728 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:25.441286    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71dd368e-c415-465c-838a-ecd215bc4069-istio-podinfo\") pod \"e2e-derive-7e6fa6af-openshift-default-5587bc9748-srqxx\" (UID: \"71dd368e-c415-465c-838a-ecd215bc4069\") " pod="openshift-ingress/e2e-derive-7e6fa6af-openshift-default-5587bc9748-srqxx"
Sep 11 23:20:25.442337 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:25.441999    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71dd368e-c415-465c-838a-ecd215bc4069-istiod-ca-cert\") pod \"e2e-derive-7e6fa6af-openshift-default-5587bc9748-srqxx\" (UID: \"71dd368e-c415-465c-838a-ecd215bc4069\") " pod="openshift-ingress/e2e-derive-7e6fa6af-openshift-default-5587bc9748-srqxx"
Sep 11 23:20:25.442686 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:25.442431    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71dd368e-c415-465c-838a-ecd215bc4069-workload-certs\") pod \"e2e-derive-7e6fa6af-openshift-default-5587bc9748-srqxx\" (UID: \"71dd368e-c415-465c-838a-ecd215bc4069\") " pod="openshift-ingress/e2e-derive-7e6fa6af-openshift-default-5587bc9748-srqxx"
Sep 11 23:20:25.442686 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:25.442646    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71dd368e-c415-465c-838a-ecd215bc4069-credential-socket\") pod \"e2e-derive-7e6fa6af-openshift-default-5587bc9748-srqxx\" (UID: \"71dd368e-c415-465c-838a-ecd215bc4069\") " pod="openshift-ingress/e2e-derive-7e6fa6af-openshift-default-5587bc9748-srqxx"
Sep 11 23:20:25.442967 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:25.442948    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71dd368e-c415-465c-838a-ecd215bc4069-istio-data\") pod \"e2e-derive-7e6fa6af-openshift-default-5587bc9748-srqxx\" (UID: \"71dd368e-c415-465c-838a-ecd215bc4069\") " pod="openshift-ingress/e2e-derive-7e6fa6af-openshift-default-5587bc9748-srqxx"
Sep 11 23:20:25.443311 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:25.443272    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71dd368e-c415-465c-838a-ecd215bc4069-workload-socket\") pod \"e2e-derive-7e6fa6af-openshift-default-5587bc9748-srqxx\" (UID: \"71dd368e-c415-465c-838a-ecd215bc4069\") " pod="openshift-ingress/e2e-derive-7e6fa6af-openshift-default-5587bc9748-srqxx"
Sep 11 23:20:25.456993 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:25.445436    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71dd368e-c415-465c-838a-ecd215bc4069-istio-envoy\") pod \"e2e-derive-7e6fa6af-openshift-default-5587bc9748-srqxx\" (UID: \"71dd368e-c415-465c-838a-ecd215bc4069\") " pod="openshift-ingress/e2e-derive-7e6fa6af-openshift-default-5587bc9748-srqxx"
Sep 11 23:20:25.459896 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:25.457782    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71dd368e-c415-465c-838a-ecd215bc4069-istio-podinfo\") pod \"e2e-derive-7e6fa6af-openshift-default-5587bc9748-srqxx\" (UID: \"71dd368e-c415-465c-838a-ecd215bc4069\") " pod="openshift-ingress/e2e-derive-7e6fa6af-openshift-default-5587bc9748-srqxx"
Sep 11 23:20:25.462693 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:25.462643    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71dd368e-c415-465c-838a-ecd215bc4069-istio-token\") pod \"e2e-derive-7e6fa6af-openshift-default-5587bc9748-srqxx\" (UID: \"71dd368e-c415-465c-838a-ecd215bc4069\") " pod="openshift-ingress/e2e-derive-7e6fa6af-openshift-default-5587bc9748-srqxx"
Sep 11 23:20:25.466711 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:25.466661    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l9mk\" (UniqueName: \"kubernetes.io/projected/71dd368e-c415-465c-838a-ecd215bc4069-kube-api-access-7l9mk\") pod \"e2e-derive-7e6fa6af-openshift-default-5587bc9748-srqxx\" (UID: \"71dd368e-c415-465c-838a-ecd215bc4069\") " pod="openshift-ingress/e2e-derive-7e6fa6af-openshift-default-5587bc9748-srqxx"
Sep 11 23:20:25.507405 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:25.507375    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-7e6fa6af-openshift-default-5587bc9748-srqxx"
Sep 11 23:20:25.659607 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:25.659583    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0a7a66-5ltxw"]
Sep 11 23:20:25.660610 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:20:25.660587    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a6a8cdf_f22f_4c18_881d_3084b3ef1c22.slice/crio-d90b0d0be397cbf67b0eafc9848fb6ffe983562bf743d3d823bc8bba8b9933c5 WatchSource:0}: Error finding container d90b0d0be397cbf67b0eafc9848fb6ffe983562bf743d3d823bc8bba8b9933c5: Status 404 returned error can't find the container with id d90b0d0be397cbf67b0eafc9848fb6ffe983562bf743d3d823bc8bba8b9933c5
Sep 11 23:20:25.791251 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:25.791222    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-7e6fa6af-openshift-default-5587bc9748-srqxx"]
Sep 11 23:20:25.791682 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:20:25.791654    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71dd368e_c415_465c_838a_ecd215bc4069.slice/crio-105cffbed8f53acb9932e7c7a22611383406cb169b16fc1aef6d4d1135a6705d WatchSource:0}: Error finding container 105cffbed8f53acb9932e7c7a22611383406cb169b16fc1aef6d4d1135a6705d: Status 404 returned error can't find the container with id 105cffbed8f53acb9932e7c7a22611383406cb169b16fc1aef6d4d1135a6705d
Sep 11 23:20:25.794123 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:25.794084    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:20:25.794214 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:25.794186    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:20:25.794255 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:25.794219    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:20:26.405575 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:26.405530    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-7e6fa6af-openshift-default-5587bc9748-srqxx" event={"ID":"71dd368e-c415-465c-838a-ecd215bc4069","Type":"ContainerStarted","Data":"898407d595f1cd3a0c4da621a045a87dc30052d86ec0fd9986cb2ea38e232a9e"}
Sep 11 23:20:26.405758 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:26.405644    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-7e6fa6af-openshift-default-5587bc9748-srqxx" event={"ID":"71dd368e-c415-465c-838a-ecd215bc4069","Type":"ContainerStarted","Data":"105cffbed8f53acb9932e7c7a22611383406cb169b16fc1aef6d4d1135a6705d"}
Sep 11 23:20:26.410434 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:26.410405    2546 generic.go:356] "Generic (PLEG): container finished" podID="0a6a8cdf-f22f-4c18-881d-3084b3ef1c22" containerID="a6ae1bc7fd52882563b22bf5def45f50e527dd5c4f2e4d6be6afafcaa1c77302" exitCode=0
Sep 11 23:20:26.410570 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:26.410483    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0a7a66-5ltxw" event={"ID":"0a6a8cdf-f22f-4c18-881d-3084b3ef1c22","Type":"ContainerDied","Data":"a6ae1bc7fd52882563b22bf5def45f50e527dd5c4f2e4d6be6afafcaa1c77302"}
Sep 11 23:20:26.410570 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:26.410530    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0a7a66-5ltxw" event={"ID":"0a6a8cdf-f22f-4c18-881d-3084b3ef1c22","Type":"ContainerStarted","Data":"d90b0d0be397cbf67b0eafc9848fb6ffe983562bf743d3d823bc8bba8b9933c5"}
Sep 11 23:20:26.428878 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:26.428832    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-7e6fa6af-openshift-default-5587bc9748-srqxx" podStartSLOduration=1.428817543 podStartE2EDuration="1.428817543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 23:20:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:20:26.426733644 +0000 UTC m=+1150.459862862" watchObservedRunningTime="2026-09-11 23:20:26.428817543 +0000 UTC m=+1150.461946806"
Sep 11 23:20:26.508140 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:26.508109    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-7e6fa6af-openshift-default-5587bc9748-srqxx"
Sep 11 23:20:26.513132 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:26.513111    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-7e6fa6af-openshift-default-5587bc9748-srqxx"
Sep 11 23:20:27.417969 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:27.417928    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-7e6fa6af-openshift-default-5587bc9748-srqxx"
Sep 11 23:20:27.418949 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:27.418930    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-7e6fa6af-openshift-default-5587bc9748-srqxx"
Sep 11 23:20:27.571354 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:27.571326    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0a7a66-5ltxw"
Sep 11 23:20:27.666957 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:27.666927    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a6a8cdf-f22f-4c18-881d-3084b3ef1c22-maas-api-service-ca\") pod \"0a6a8cdf-f22f-4c18-881d-3084b3ef1c22\" (UID: \"0a6a8cdf-f22f-4c18-881d-3084b3ef1c22\") "
Sep 11 23:20:27.667332 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:27.667308    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a6a8cdf-f22f-4c18-881d-3084b3ef1c22-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0a6a8cdf-f22f-4c18-881d-3084b3ef1c22" (UID: "0a6a8cdf-f22f-4c18-881d-3084b3ef1c22"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 23:20:27.768882 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:27.768792    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a6a8cdf-f22f-4c18-881d-3084b3ef1c22-maas-api-service-ca\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:20:28.427099 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:28.427066    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0a7a66-5ltxw"
Sep 11 23:20:28.427282 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:28.427095    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0a7a66-5ltxw" event={"ID":"0a6a8cdf-f22f-4c18-881d-3084b3ef1c22","Type":"ContainerDied","Data":"d90b0d0be397cbf67b0eafc9848fb6ffe983562bf743d3d823bc8bba8b9933c5"}
Sep 11 23:20:28.427282 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:28.427148    2546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d90b0d0be397cbf67b0eafc9848fb6ffe983562bf743d3d823bc8bba8b9933c5"
Sep 11 23:20:29.690353 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:29.690289    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c8789319-openshift-default-7bfc489944-spgfv"
Sep 11 23:20:29.793256 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:29.793222    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9b43763-2b84-4249-bf71-49a9505341bc-istio-podinfo\") pod \"c9b43763-2b84-4249-bf71-49a9505341bc\" (UID: \"c9b43763-2b84-4249-bf71-49a9505341bc\") "
Sep 11 23:20:29.793432 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:29.793269    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9b43763-2b84-4249-bf71-49a9505341bc-workload-certs\") pod \"c9b43763-2b84-4249-bf71-49a9505341bc\" (UID: \"c9b43763-2b84-4249-bf71-49a9505341bc\") "
Sep 11 23:20:29.793432 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:29.793318    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nm77j\" (UniqueName: \"kubernetes.io/projected/c9b43763-2b84-4249-bf71-49a9505341bc-kube-api-access-nm77j\") pod \"c9b43763-2b84-4249-bf71-49a9505341bc\" (UID: \"c9b43763-2b84-4249-bf71-49a9505341bc\") "
Sep 11 23:20:29.793432 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:29.793342    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9b43763-2b84-4249-bf71-49a9505341bc-workload-socket\") pod \"c9b43763-2b84-4249-bf71-49a9505341bc\" (UID: \"c9b43763-2b84-4249-bf71-49a9505341bc\") "
Sep 11 23:20:29.793432 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:29.793399    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9b43763-2b84-4249-bf71-49a9505341bc-credential-socket\") pod \"c9b43763-2b84-4249-bf71-49a9505341bc\" (UID: \"c9b43763-2b84-4249-bf71-49a9505341bc\") "
Sep 11 23:20:29.793657 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:29.793439    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9b43763-2b84-4249-bf71-49a9505341bc-istio-token\") pod \"c9b43763-2b84-4249-bf71-49a9505341bc\" (UID: \"c9b43763-2b84-4249-bf71-49a9505341bc\") "
Sep 11 23:20:29.793657 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:29.793492    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9b43763-2b84-4249-bf71-49a9505341bc-istio-data\") pod \"c9b43763-2b84-4249-bf71-49a9505341bc\" (UID: \"c9b43763-2b84-4249-bf71-49a9505341bc\") "
Sep 11 23:20:29.793657 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:29.793524    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9b43763-2b84-4249-bf71-49a9505341bc-istiod-ca-cert\") pod \"c9b43763-2b84-4249-bf71-49a9505341bc\" (UID: \"c9b43763-2b84-4249-bf71-49a9505341bc\") "
Sep 11 23:20:29.793657 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:29.793615    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9b43763-2b84-4249-bf71-49a9505341bc-istio-envoy\") pod \"c9b43763-2b84-4249-bf71-49a9505341bc\" (UID: \"c9b43763-2b84-4249-bf71-49a9505341bc\") "
Sep 11 23:20:29.793657 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:29.793636    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9b43763-2b84-4249-bf71-49a9505341bc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c9b43763-2b84-4249-bf71-49a9505341bc" (UID: "c9b43763-2b84-4249-bf71-49a9505341bc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:20:29.793897 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:29.793667    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9b43763-2b84-4249-bf71-49a9505341bc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c9b43763-2b84-4249-bf71-49a9505341bc" (UID: "c9b43763-2b84-4249-bf71-49a9505341bc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:20:29.793897 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:29.793685    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9b43763-2b84-4249-bf71-49a9505341bc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c9b43763-2b84-4249-bf71-49a9505341bc" (UID: "c9b43763-2b84-4249-bf71-49a9505341bc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:20:29.793897 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:29.793806    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9b43763-2b84-4249-bf71-49a9505341bc-istio-data" (OuterVolumeSpecName: "istio-data") pod "c9b43763-2b84-4249-bf71-49a9505341bc" (UID: "c9b43763-2b84-4249-bf71-49a9505341bc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:20:29.794074 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:29.793922    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9b43763-2b84-4249-bf71-49a9505341bc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c9b43763-2b84-4249-bf71-49a9505341bc" (UID: "c9b43763-2b84-4249-bf71-49a9505341bc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 23:20:29.794074 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:29.794017    2546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9b43763-2b84-4249-bf71-49a9505341bc-istio-data\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:20:29.794074 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:29.794034    2546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9b43763-2b84-4249-bf71-49a9505341bc-istiod-ca-cert\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:20:29.794074 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:29.794058    2546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9b43763-2b84-4249-bf71-49a9505341bc-workload-certs\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:20:29.794074 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:29.794068    2546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9b43763-2b84-4249-bf71-49a9505341bc-workload-socket\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:20:29.794074 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:29.794076    2546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9b43763-2b84-4249-bf71-49a9505341bc-credential-socket\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:20:29.795609 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:29.795582    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c9b43763-2b84-4249-bf71-49a9505341bc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c9b43763-2b84-4249-bf71-49a9505341bc" (UID: "c9b43763-2b84-4249-bf71-49a9505341bc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 23:20:29.795788 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:29.795768    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9b43763-2b84-4249-bf71-49a9505341bc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c9b43763-2b84-4249-bf71-49a9505341bc" (UID: "c9b43763-2b84-4249-bf71-49a9505341bc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:20:29.795857 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:29.795799    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9b43763-2b84-4249-bf71-49a9505341bc-istio-token" (OuterVolumeSpecName: "istio-token") pod "c9b43763-2b84-4249-bf71-49a9505341bc" (UID: "c9b43763-2b84-4249-bf71-49a9505341bc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 23:20:29.795898 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:29.795878    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9b43763-2b84-4249-bf71-49a9505341bc-kube-api-access-nm77j" (OuterVolumeSpecName: "kube-api-access-nm77j") pod "c9b43763-2b84-4249-bf71-49a9505341bc" (UID: "c9b43763-2b84-4249-bf71-49a9505341bc"). InnerVolumeSpecName "kube-api-access-nm77j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 23:20:29.895361 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:29.895332    2546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9b43763-2b84-4249-bf71-49a9505341bc-istio-envoy\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:20:29.895361 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:29.895354    2546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9b43763-2b84-4249-bf71-49a9505341bc-istio-podinfo\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:20:29.895541 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:29.895363    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nm77j\" (UniqueName: \"kubernetes.io/projected/c9b43763-2b84-4249-bf71-49a9505341bc-kube-api-access-nm77j\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:20:29.895541 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:29.895381    2546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9b43763-2b84-4249-bf71-49a9505341bc-istio-token\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:20:30.089766 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:30.089683    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"]
Sep 11 23:20:30.441606 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:30.441569    2546 generic.go:356] "Generic (PLEG): container finished" podID="c9b43763-2b84-4249-bf71-49a9505341bc" containerID="1512c36615eee64859f40789826b99498be5fc26d01285790b2e668c267a9ed7" exitCode=0
Sep 11 23:20:30.441773 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:30.441623    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c8789319-openshift-default-7bfc489944-spgfv"
Sep 11 23:20:30.441773 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:30.441656    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c8789319-openshift-default-7bfc489944-spgfv" event={"ID":"c9b43763-2b84-4249-bf71-49a9505341bc","Type":"ContainerDied","Data":"1512c36615eee64859f40789826b99498be5fc26d01285790b2e668c267a9ed7"}
Sep 11 23:20:30.441773 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:30.441709    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c8789319-openshift-default-7bfc489944-spgfv" event={"ID":"c9b43763-2b84-4249-bf71-49a9505341bc","Type":"ContainerDied","Data":"0f3ef051ea5b26c5bef6751ecc264f111347c7778fd8ea2813f911bcdb1f3f10"}
Sep 11 23:20:30.441773 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:30.441731    2546 scope.go:117] "RemoveContainer" containerID="1512c36615eee64859f40789826b99498be5fc26d01285790b2e668c267a9ed7"
Sep 11 23:20:30.452388 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:30.452368    2546 scope.go:117] "RemoveContainer" containerID="c5882d204890b95ae3eba20ade5b4a753d6cb5d34d15f41425b88228990d0909"
Sep 11 23:20:30.461371 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:30.461353    2546 scope.go:117] "RemoveContainer" containerID="1512c36615eee64859f40789826b99498be5fc26d01285790b2e668c267a9ed7"
Sep 11 23:20:30.461788 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:20:30.461746    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1512c36615eee64859f40789826b99498be5fc26d01285790b2e668c267a9ed7\": container with ID starting with 1512c36615eee64859f40789826b99498be5fc26d01285790b2e668c267a9ed7 not found: ID does not exist" containerID="1512c36615eee64859f40789826b99498be5fc26d01285790b2e668c267a9ed7"
Sep 11 23:20:30.461996 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:30.461789    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1512c36615eee64859f40789826b99498be5fc26d01285790b2e668c267a9ed7"} err="failed to get container status \"1512c36615eee64859f40789826b99498be5fc26d01285790b2e668c267a9ed7\": rpc error: code = NotFound desc = could not find container \"1512c36615eee64859f40789826b99498be5fc26d01285790b2e668c267a9ed7\": container with ID starting with 1512c36615eee64859f40789826b99498be5fc26d01285790b2e668c267a9ed7 not found: ID does not exist"
Sep 11 23:20:30.461996 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:30.461820    2546 scope.go:117] "RemoveContainer" containerID="c5882d204890b95ae3eba20ade5b4a753d6cb5d34d15f41425b88228990d0909"
Sep 11 23:20:30.465238 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:20:30.463065    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5882d204890b95ae3eba20ade5b4a753d6cb5d34d15f41425b88228990d0909\": container with ID starting with c5882d204890b95ae3eba20ade5b4a753d6cb5d34d15f41425b88228990d0909 not found: ID does not exist" containerID="c5882d204890b95ae3eba20ade5b4a753d6cb5d34d15f41425b88228990d0909"
Sep 11 23:20:30.465238 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:30.463107    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5882d204890b95ae3eba20ade5b4a753d6cb5d34d15f41425b88228990d0909"} err="failed to get container status \"c5882d204890b95ae3eba20ade5b4a753d6cb5d34d15f41425b88228990d0909\": rpc error: code = NotFound desc = could not find container \"c5882d204890b95ae3eba20ade5b4a753d6cb5d34d15f41425b88228990d0909\": container with ID starting with c5882d204890b95ae3eba20ade5b4a753d6cb5d34d15f41425b88228990d0909 not found: ID does not exist"
Sep 11 23:20:30.470682 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:30.470658    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c8789319-openshift-default-7bfc489944-spgfv"]
Sep 11 23:20:30.474082 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:30.474053    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-c8789319-openshift-default-7bfc489944-spgfv"]
Sep 11 23:20:30.495056 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:30.495029    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9b43763-2b84-4249-bf71-49a9505341bc" path="/var/lib/kubelet/pods/c9b43763-2b84-4249-bf71-49a9505341bc/volumes"
Sep 11 23:20:36.081281 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:36.081247    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0a7a66-5ltxw"]
Sep 11 23:20:36.083780 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:36.083755    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0a7a66-5ltxw"]
Sep 11 23:20:36.494022 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:36.493988    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a6a8cdf-f22f-4c18-881d-3084b3ef1c22" path="/var/lib/kubelet/pods/0a6a8cdf-f22f-4c18-881d-3084b3ef1c22/volumes"
Sep 11 23:20:42.360863 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:42.360622    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7e6fa6af-r4xlc"]
Sep 11 23:20:42.361911 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:42.361787    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9b43763-2b84-4249-bf71-49a9505341bc" containerName="istio-proxy"
Sep 11 23:20:42.361911 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:42.361819    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9b43763-2b84-4249-bf71-49a9505341bc" containerName="istio-proxy"
Sep 11 23:20:42.361911 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:42.361860    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9b43763-2b84-4249-bf71-49a9505341bc" containerName="istio-proxy"
Sep 11 23:20:42.361911 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:42.361873    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9b43763-2b84-4249-bf71-49a9505341bc" containerName="istio-proxy"
Sep 11 23:20:42.361911 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:42.361888    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a6a8cdf-f22f-4c18-881d-3084b3ef1c22" containerName="revoke-keys"
Sep 11 23:20:42.361911 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:42.361897    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a6a8cdf-f22f-4c18-881d-3084b3ef1c22" containerName="revoke-keys"
Sep 11 23:20:42.362315 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:42.362098    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a6a8cdf-f22f-4c18-881d-3084b3ef1c22" containerName="revoke-keys"
Sep 11 23:20:42.362315 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:42.362117    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9b43763-2b84-4249-bf71-49a9505341bc" containerName="istio-proxy"
Sep 11 23:20:42.362315 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:42.362131    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9b43763-2b84-4249-bf71-49a9505341bc" containerName="istio-proxy"
Sep 11 23:20:42.368420 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:42.368391    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7e6fa6af-r4xlc"
Sep 11 23:20:42.369032 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:42.368290    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7e6fa6af-r4xlc"]
Sep 11 23:20:42.371650 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:42.371629    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vrx7m\""
Sep 11 23:20:42.371958 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:42.371690    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 23:20:42.437179 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:42.430999    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13bcc4eb-bc44-48e1-bf93-99f38943b020-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-7e6fa6af-r4xlc\" (UID: \"13bcc4eb-bc44-48e1-bf93-99f38943b020\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7e6fa6af-r4xlc"
Sep 11 23:20:42.531790 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:42.531739    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13bcc4eb-bc44-48e1-bf93-99f38943b020-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-7e6fa6af-r4xlc\" (UID: \"13bcc4eb-bc44-48e1-bf93-99f38943b020\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7e6fa6af-r4xlc"
Sep 11 23:20:42.532676 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:42.532642    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13bcc4eb-bc44-48e1-bf93-99f38943b020-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-7e6fa6af-r4xlc\" (UID: \"13bcc4eb-bc44-48e1-bf93-99f38943b020\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7e6fa6af-r4xlc"
Sep 11 23:20:42.679952 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:42.679907    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7e6fa6af-r4xlc"
Sep 11 23:20:42.981849 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:42.981819    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7e6fa6af-r4xlc"]
Sep 11 23:20:42.982705 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:20:42.982679    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13bcc4eb_bc44_48e1_bf93_99f38943b020.slice/crio-47d242634cc84b0029d5ca12590a980d43784f6ffa9412b6f0404725bf05126c WatchSource:0}: Error finding container 47d242634cc84b0029d5ca12590a980d43784f6ffa9412b6f0404725bf05126c: Status 404 returned error can't find the container with id 47d242634cc84b0029d5ca12590a980d43784f6ffa9412b6f0404725bf05126c
Sep 11 23:20:43.532575 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:43.532527    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7e6fa6af-r4xlc" event={"ID":"13bcc4eb-bc44-48e1-bf93-99f38943b020","Type":"ContainerStarted","Data":"fff138816c6423bd5b24402d47336908f723eadeaed54805ea5432701347152f"}
Sep 11 23:20:43.533038 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:43.532601    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7e6fa6af-r4xlc" event={"ID":"13bcc4eb-bc44-48e1-bf93-99f38943b020","Type":"ContainerStarted","Data":"47d242634cc84b0029d5ca12590a980d43784f6ffa9412b6f0404725bf05126c"}
Sep 11 23:20:43.549697 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:43.549639    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7e6fa6af-r4xlc" podStartSLOduration=1.549604304 podStartE2EDuration="1.549604304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 23:20:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:20:43.547714786 +0000 UTC m=+1167.580844004" watchObservedRunningTime="2026-09-11 23:20:43.549604304 +0000 UTC m=+1167.582733522"
Sep 11 23:20:44.549862 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:44.549825    2546 generic.go:356] "Generic (PLEG): container finished" podID="13bcc4eb-bc44-48e1-bf93-99f38943b020" containerID="fff138816c6423bd5b24402d47336908f723eadeaed54805ea5432701347152f" exitCode=0
Sep 11 23:20:44.550779 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:44.550547    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7e6fa6af-r4xlc" event={"ID":"13bcc4eb-bc44-48e1-bf93-99f38943b020","Type":"ContainerDied","Data":"fff138816c6423bd5b24402d47336908f723eadeaed54805ea5432701347152f"}
Sep 11 23:20:45.709066 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:45.709041    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7e6fa6af-r4xlc"
Sep 11 23:20:45.777241 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:45.777203    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13bcc4eb-bc44-48e1-bf93-99f38943b020-maas-api-service-ca\") pod \"13bcc4eb-bc44-48e1-bf93-99f38943b020\" (UID: \"13bcc4eb-bc44-48e1-bf93-99f38943b020\") "
Sep 11 23:20:45.777555 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:45.777532    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13bcc4eb-bc44-48e1-bf93-99f38943b020-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "13bcc4eb-bc44-48e1-bf93-99f38943b020" (UID: "13bcc4eb-bc44-48e1-bf93-99f38943b020"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 23:20:45.777677 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:45.777662    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13bcc4eb-bc44-48e1-bf93-99f38943b020-maas-api-service-ca\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:20:46.565677 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:46.565639    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7e6fa6af-r4xlc"
Sep 11 23:20:46.565873 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:46.565656    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7e6fa6af-r4xlc" event={"ID":"13bcc4eb-bc44-48e1-bf93-99f38943b020","Type":"ContainerDied","Data":"47d242634cc84b0029d5ca12590a980d43784f6ffa9412b6f0404725bf05126c"}
Sep 11 23:20:46.566086 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:46.566060    2546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="47d242634cc84b0029d5ca12590a980d43784f6ffa9412b6f0404725bf05126c"
Sep 11 23:20:50.595119 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:50.595019    2546 generic.go:356] "Generic (PLEG): container finished" podID="71dd368e-c415-465c-838a-ecd215bc4069" containerID="898407d595f1cd3a0c4da621a045a87dc30052d86ec0fd9986cb2ea38e232a9e" exitCode=0
Sep 11 23:20:50.595119 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:50.595074    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-7e6fa6af-openshift-default-5587bc9748-srqxx" event={"ID":"71dd368e-c415-465c-838a-ecd215bc4069","Type":"ContainerDied","Data":"898407d595f1cd3a0c4da621a045a87dc30052d86ec0fd9986cb2ea38e232a9e"}
Sep 11 23:20:50.595581 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:50.595498    2546 scope.go:117] "RemoveContainer" containerID="898407d595f1cd3a0c4da621a045a87dc30052d86ec0fd9986cb2ea38e232a9e"
Sep 11 23:20:50.596603 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:50.596572    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:20:50.596706 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:50.596660    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:20:50.596769 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:50.596705    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:20:51.603058 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:51.603020    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-7e6fa6af-openshift-default-5587bc9748-srqxx" event={"ID":"71dd368e-c415-465c-838a-ecd215bc4069","Type":"ContainerStarted","Data":"98c69ea3b3050ecf186617029ed8c586c7461b83d8dbcdf6d21e44be4104d318"}
Sep 11 23:20:52.005865 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:52.005837    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"]
Sep 11 23:20:52.507889 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:52.507858    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-7e6fa6af-openshift-default-5587bc9748-srqxx"
Sep 11 23:20:52.508086 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:52.508065    2546 patch_prober.go:28] interesting pod/e2e-derive-7e6fa6af-openshift-default-5587bc9748-srqxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Sep 11 23:20:52.508132 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:52.508113    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-7e6fa6af-openshift-default-5587bc9748-srqxx" podUID="71dd368e-c415-465c-838a-ecd215bc4069" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Sep 11 23:20:53.121173 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:53.121118    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7e6fa6af-r4xlc"]
Sep 11 23:20:53.124743 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:53.124709    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7e6fa6af-r4xlc"]
Sep 11 23:20:53.281557 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:53.281449    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-7e6fa6af-openshift-default-5587bc9748-srqxx"]
Sep 11 23:20:53.508777 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:53.508742    2546 patch_prober.go:28] interesting pod/e2e-derive-7e6fa6af-openshift-default-5587bc9748-srqxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Sep 11 23:20:53.508967 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:53.508813    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-7e6fa6af-openshift-default-5587bc9748-srqxx" podUID="71dd368e-c415-465c-838a-ecd215bc4069" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Sep 11 23:20:53.616352 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:53.616289    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-7e6fa6af-openshift-default-5587bc9748-srqxx" podUID="71dd368e-c415-465c-838a-ecd215bc4069" containerName="istio-proxy" containerID="cri-o://98c69ea3b3050ecf186617029ed8c586c7461b83d8dbcdf6d21e44be4104d318" gracePeriod=30
Sep 11 23:20:54.496387 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:54.496354    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13bcc4eb-bc44-48e1-bf93-99f38943b020" path="/var/lib/kubelet/pods/13bcc4eb-bc44-48e1-bf93-99f38943b020/volumes"
Sep 11 23:20:57.417866 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:20:57.417827    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-7e6fa6af-openshift-default-5587bc9748-srqxx"
Sep 11 23:21:02.382971 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:02.382909    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-7e6fa6af-openshift-default-5587bc9748-srqxx"
Sep 11 23:21:02.469845 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:02.469814    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71dd368e-c415-465c-838a-ecd215bc4069-istio-envoy\") pod \"71dd368e-c415-465c-838a-ecd215bc4069\" (UID: \"71dd368e-c415-465c-838a-ecd215bc4069\") "
Sep 11 23:21:02.470041 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:02.469872    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71dd368e-c415-465c-838a-ecd215bc4069-istio-podinfo\") pod \"71dd368e-c415-465c-838a-ecd215bc4069\" (UID: \"71dd368e-c415-465c-838a-ecd215bc4069\") "
Sep 11 23:21:02.470041 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:02.469894    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71dd368e-c415-465c-838a-ecd215bc4069-istio-data\") pod \"71dd368e-c415-465c-838a-ecd215bc4069\" (UID: \"71dd368e-c415-465c-838a-ecd215bc4069\") "
Sep 11 23:21:02.470041 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:02.469913    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71dd368e-c415-465c-838a-ecd215bc4069-istio-token\") pod \"71dd368e-c415-465c-838a-ecd215bc4069\" (UID: \"71dd368e-c415-465c-838a-ecd215bc4069\") "
Sep 11 23:21:02.470041 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:02.469931    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71dd368e-c415-465c-838a-ecd215bc4069-istiod-ca-cert\") pod \"71dd368e-c415-465c-838a-ecd215bc4069\" (UID: \"71dd368e-c415-465c-838a-ecd215bc4069\") "
Sep 11 23:21:02.470041 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:02.469975    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7l9mk\" (UniqueName: \"kubernetes.io/projected/71dd368e-c415-465c-838a-ecd215bc4069-kube-api-access-7l9mk\") pod \"71dd368e-c415-465c-838a-ecd215bc4069\" (UID: \"71dd368e-c415-465c-838a-ecd215bc4069\") "
Sep 11 23:21:02.470357 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:02.470071    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71dd368e-c415-465c-838a-ecd215bc4069-workload-socket\") pod \"71dd368e-c415-465c-838a-ecd215bc4069\" (UID: \"71dd368e-c415-465c-838a-ecd215bc4069\") "
Sep 11 23:21:02.470357 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:02.470144    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71dd368e-c415-465c-838a-ecd215bc4069-credential-socket\") pod \"71dd368e-c415-465c-838a-ecd215bc4069\" (UID: \"71dd368e-c415-465c-838a-ecd215bc4069\") "
Sep 11 23:21:02.470357 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:02.470208    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71dd368e-c415-465c-838a-ecd215bc4069-workload-certs\") pod \"71dd368e-c415-465c-838a-ecd215bc4069\" (UID: \"71dd368e-c415-465c-838a-ecd215bc4069\") "
Sep 11 23:21:02.470357 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:02.470314    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71dd368e-c415-465c-838a-ecd215bc4069-istio-data" (OuterVolumeSpecName: "istio-data") pod "71dd368e-c415-465c-838a-ecd215bc4069" (UID: "71dd368e-c415-465c-838a-ecd215bc4069"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:21:02.470552 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:02.470406    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71dd368e-c415-465c-838a-ecd215bc4069-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "71dd368e-c415-465c-838a-ecd215bc4069" (UID: "71dd368e-c415-465c-838a-ecd215bc4069"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 23:21:02.470552 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:02.470526    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71dd368e-c415-465c-838a-ecd215bc4069-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "71dd368e-c415-465c-838a-ecd215bc4069" (UID: "71dd368e-c415-465c-838a-ecd215bc4069"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:21:02.470655 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:02.470574    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71dd368e-c415-465c-838a-ecd215bc4069-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "71dd368e-c415-465c-838a-ecd215bc4069" (UID: "71dd368e-c415-465c-838a-ecd215bc4069"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:21:02.470710 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:02.470670    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71dd368e-c415-465c-838a-ecd215bc4069-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "71dd368e-c415-465c-838a-ecd215bc4069" (UID: "71dd368e-c415-465c-838a-ecd215bc4069"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:21:02.470814 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:02.470794    2546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71dd368e-c415-465c-838a-ecd215bc4069-workload-socket\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:21:02.470871 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:02.470820    2546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71dd368e-c415-465c-838a-ecd215bc4069-credential-socket\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:21:02.470871 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:02.470835    2546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71dd368e-c415-465c-838a-ecd215bc4069-workload-certs\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:21:02.470871 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:02.470847    2546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71dd368e-c415-465c-838a-ecd215bc4069-istio-data\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:21:02.470871 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:02.470859    2546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71dd368e-c415-465c-838a-ecd215bc4069-istiod-ca-cert\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:21:02.472374 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:02.472327    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/71dd368e-c415-465c-838a-ecd215bc4069-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "71dd368e-c415-465c-838a-ecd215bc4069" (UID: "71dd368e-c415-465c-838a-ecd215bc4069"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 23:21:02.472503 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:02.472422    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71dd368e-c415-465c-838a-ecd215bc4069-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "71dd368e-c415-465c-838a-ecd215bc4069" (UID: "71dd368e-c415-465c-838a-ecd215bc4069"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:21:02.472503 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:02.472458    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71dd368e-c415-465c-838a-ecd215bc4069-kube-api-access-7l9mk" (OuterVolumeSpecName: "kube-api-access-7l9mk") pod "71dd368e-c415-465c-838a-ecd215bc4069" (UID: "71dd368e-c415-465c-838a-ecd215bc4069"). InnerVolumeSpecName "kube-api-access-7l9mk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 23:21:02.472582 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:02.472544    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71dd368e-c415-465c-838a-ecd215bc4069-istio-token" (OuterVolumeSpecName: "istio-token") pod "71dd368e-c415-465c-838a-ecd215bc4069" (UID: "71dd368e-c415-465c-838a-ecd215bc4069"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 23:21:02.572081 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:02.572053    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7l9mk\" (UniqueName: \"kubernetes.io/projected/71dd368e-c415-465c-838a-ecd215bc4069-kube-api-access-7l9mk\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:21:02.572081 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:02.572078    2546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71dd368e-c415-465c-838a-ecd215bc4069-istio-envoy\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:21:02.572081 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:02.572087    2546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71dd368e-c415-465c-838a-ecd215bc4069-istio-podinfo\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:21:02.572302 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:02.572094    2546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71dd368e-c415-465c-838a-ecd215bc4069-istio-token\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:21:02.699094 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:02.699061    2546 generic.go:356] "Generic (PLEG): container finished" podID="71dd368e-c415-465c-838a-ecd215bc4069" containerID="98c69ea3b3050ecf186617029ed8c586c7461b83d8dbcdf6d21e44be4104d318" exitCode=0
Sep 11 23:21:02.699268 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:02.699114    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-7e6fa6af-openshift-default-5587bc9748-srqxx"
Sep 11 23:21:02.699268 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:02.699127    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-7e6fa6af-openshift-default-5587bc9748-srqxx" event={"ID":"71dd368e-c415-465c-838a-ecd215bc4069","Type":"ContainerDied","Data":"98c69ea3b3050ecf186617029ed8c586c7461b83d8dbcdf6d21e44be4104d318"}
Sep 11 23:21:02.699268 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:02.699192    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-7e6fa6af-openshift-default-5587bc9748-srqxx" event={"ID":"71dd368e-c415-465c-838a-ecd215bc4069","Type":"ContainerDied","Data":"105cffbed8f53acb9932e7c7a22611383406cb169b16fc1aef6d4d1135a6705d"}
Sep 11 23:21:02.699268 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:02.699220    2546 scope.go:117] "RemoveContainer" containerID="98c69ea3b3050ecf186617029ed8c586c7461b83d8dbcdf6d21e44be4104d318"
Sep 11 23:21:02.708666 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:02.708646    2546 scope.go:117] "RemoveContainer" containerID="898407d595f1cd3a0c4da621a045a87dc30052d86ec0fd9986cb2ea38e232a9e"
Sep 11 23:21:02.717571 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:02.717553    2546 scope.go:117] "RemoveContainer" containerID="98c69ea3b3050ecf186617029ed8c586c7461b83d8dbcdf6d21e44be4104d318"
Sep 11 23:21:02.717821 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:21:02.717801    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98c69ea3b3050ecf186617029ed8c586c7461b83d8dbcdf6d21e44be4104d318\": container with ID starting with 98c69ea3b3050ecf186617029ed8c586c7461b83d8dbcdf6d21e44be4104d318 not found: ID does not exist" containerID="98c69ea3b3050ecf186617029ed8c586c7461b83d8dbcdf6d21e44be4104d318"
Sep 11 23:21:02.717870 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:02.717830    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98c69ea3b3050ecf186617029ed8c586c7461b83d8dbcdf6d21e44be4104d318"} err="failed to get container status \"98c69ea3b3050ecf186617029ed8c586c7461b83d8dbcdf6d21e44be4104d318\": rpc error: code = NotFound desc = could not find container \"98c69ea3b3050ecf186617029ed8c586c7461b83d8dbcdf6d21e44be4104d318\": container with ID starting with 98c69ea3b3050ecf186617029ed8c586c7461b83d8dbcdf6d21e44be4104d318 not found: ID does not exist"
Sep 11 23:21:02.717870 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:02.717845    2546 scope.go:117] "RemoveContainer" containerID="898407d595f1cd3a0c4da621a045a87dc30052d86ec0fd9986cb2ea38e232a9e"
Sep 11 23:21:02.718100 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:21:02.718082    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"898407d595f1cd3a0c4da621a045a87dc30052d86ec0fd9986cb2ea38e232a9e\": container with ID starting with 898407d595f1cd3a0c4da621a045a87dc30052d86ec0fd9986cb2ea38e232a9e not found: ID does not exist" containerID="898407d595f1cd3a0c4da621a045a87dc30052d86ec0fd9986cb2ea38e232a9e"
Sep 11 23:21:02.718142 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:02.718107    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"898407d595f1cd3a0c4da621a045a87dc30052d86ec0fd9986cb2ea38e232a9e"} err="failed to get container status \"898407d595f1cd3a0c4da621a045a87dc30052d86ec0fd9986cb2ea38e232a9e\": rpc error: code = NotFound desc = could not find container \"898407d595f1cd3a0c4da621a045a87dc30052d86ec0fd9986cb2ea38e232a9e\": container with ID starting with 898407d595f1cd3a0c4da621a045a87dc30052d86ec0fd9986cb2ea38e232a9e not found: ID does not exist"
Sep 11 23:21:02.728967 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:02.728935    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-7e6fa6af-openshift-default-5587bc9748-srqxx"]
Sep 11 23:21:02.737060 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:02.737034    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-7e6fa6af-openshift-default-5587bc9748-srqxx"]
Sep 11 23:21:04.494536 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:04.494502    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71dd368e-c415-465c-838a-ecd215bc4069" path="/var/lib/kubelet/pods/71dd368e-c415-465c-838a-ecd215bc4069/volumes"
Sep 11 23:21:11.292862 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:11.292826    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"]
Sep 11 23:21:13.535321 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:13.535285    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8"]
Sep 11 23:21:13.535756 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:13.535521    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8" podUID="ad22eafb-8cf0-41a7-9286-ee3d645405c4" containerName="istio-proxy" containerID="cri-o://0ab78a91b61532cdd16422e9e880a364d0bf00f8d94ec0491a327e8f4f17fde8" gracePeriod=30
Sep 11 23:21:14.244381 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:14.244342    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-7719ed-openshift-default-7cf684688d-bjzms"]
Sep 11 23:21:14.244861 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:14.244848    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71dd368e-c415-465c-838a-ecd215bc4069" containerName="istio-proxy"
Sep 11 23:21:14.244906 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:14.244864    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="71dd368e-c415-465c-838a-ecd215bc4069" containerName="istio-proxy"
Sep 11 23:21:14.244906 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:14.244875    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71dd368e-c415-465c-838a-ecd215bc4069" containerName="istio-proxy"
Sep 11 23:21:14.244906 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:14.244881    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="71dd368e-c415-465c-838a-ecd215bc4069" containerName="istio-proxy"
Sep 11 23:21:14.245014 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:14.244913    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13bcc4eb-bc44-48e1-bf93-99f38943b020" containerName="revoke-keys"
Sep 11 23:21:14.245014 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:14.244919    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="13bcc4eb-bc44-48e1-bf93-99f38943b020" containerName="revoke-keys"
Sep 11 23:21:14.245014 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:14.245001    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="71dd368e-c415-465c-838a-ecd215bc4069" containerName="istio-proxy"
Sep 11 23:21:14.245014 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:14.245014    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="13bcc4eb-bc44-48e1-bf93-99f38943b020" containerName="revoke-keys"
Sep 11 23:21:14.248249 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:14.248229    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-7719ed-openshift-default-7cf684688d-bjzms"
Sep 11 23:21:14.251998 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:14.251975    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-7719ed-openshift-default-dockercfg-hrmds\""
Sep 11 23:21:14.260435 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:14.260411    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-7719ed-openshift-default-7cf684688d-bjzms"]
Sep 11 23:21:14.294927 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:14.294897    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0511d07e-cb77-4534-bc61-48f06dfd3bf3-workload-socket\") pod \"e2e-explicit-7719ed-openshift-default-7cf684688d-bjzms\" (UID: \"0511d07e-cb77-4534-bc61-48f06dfd3bf3\") " pod="openshift-ingress/e2e-explicit-7719ed-openshift-default-7cf684688d-bjzms"
Sep 11 23:21:14.295073 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:14.294934    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0511d07e-cb77-4534-bc61-48f06dfd3bf3-istio-data\") pod \"e2e-explicit-7719ed-openshift-default-7cf684688d-bjzms\" (UID: \"0511d07e-cb77-4534-bc61-48f06dfd3bf3\") " pod="openshift-ingress/e2e-explicit-7719ed-openshift-default-7cf684688d-bjzms"
Sep 11 23:21:14.295073 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:14.294961    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7sp7\" (UniqueName: \"kubernetes.io/projected/0511d07e-cb77-4534-bc61-48f06dfd3bf3-kube-api-access-k7sp7\") pod \"e2e-explicit-7719ed-openshift-default-7cf684688d-bjzms\" (UID: \"0511d07e-cb77-4534-bc61-48f06dfd3bf3\") " pod="openshift-ingress/e2e-explicit-7719ed-openshift-default-7cf684688d-bjzms"
Sep 11 23:21:14.295073 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:14.295037    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0511d07e-cb77-4534-bc61-48f06dfd3bf3-credential-socket\") pod \"e2e-explicit-7719ed-openshift-default-7cf684688d-bjzms\" (UID: \"0511d07e-cb77-4534-bc61-48f06dfd3bf3\") " pod="openshift-ingress/e2e-explicit-7719ed-openshift-default-7cf684688d-bjzms"
Sep 11 23:21:14.295073 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:14.295063    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0511d07e-cb77-4534-bc61-48f06dfd3bf3-istio-envoy\") pod \"e2e-explicit-7719ed-openshift-default-7cf684688d-bjzms\" (UID: \"0511d07e-cb77-4534-bc61-48f06dfd3bf3\") " pod="openshift-ingress/e2e-explicit-7719ed-openshift-default-7cf684688d-bjzms"
Sep 11 23:21:14.295267 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:14.295098    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0511d07e-cb77-4534-bc61-48f06dfd3bf3-istio-podinfo\") pod \"e2e-explicit-7719ed-openshift-default-7cf684688d-bjzms\" (UID: \"0511d07e-cb77-4534-bc61-48f06dfd3bf3\") " pod="openshift-ingress/e2e-explicit-7719ed-openshift-default-7cf684688d-bjzms"
Sep 11 23:21:14.295267 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:14.295122    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0511d07e-cb77-4534-bc61-48f06dfd3bf3-workload-certs\") pod \"e2e-explicit-7719ed-openshift-default-7cf684688d-bjzms\" (UID: \"0511d07e-cb77-4534-bc61-48f06dfd3bf3\") " pod="openshift-ingress/e2e-explicit-7719ed-openshift-default-7cf684688d-bjzms"
Sep 11 23:21:14.295267 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:14.295225    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0511d07e-cb77-4534-bc61-48f06dfd3bf3-istiod-ca-cert\") pod \"e2e-explicit-7719ed-openshift-default-7cf684688d-bjzms\" (UID: \"0511d07e-cb77-4534-bc61-48f06dfd3bf3\") " pod="openshift-ingress/e2e-explicit-7719ed-openshift-default-7cf684688d-bjzms"
Sep 11 23:21:14.295267 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:14.295260    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0511d07e-cb77-4534-bc61-48f06dfd3bf3-istio-token\") pod \"e2e-explicit-7719ed-openshift-default-7cf684688d-bjzms\" (UID: \"0511d07e-cb77-4534-bc61-48f06dfd3bf3\") " pod="openshift-ingress/e2e-explicit-7719ed-openshift-default-7cf684688d-bjzms"
Sep 11 23:21:14.396603 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:14.396555    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0511d07e-cb77-4534-bc61-48f06dfd3bf3-istiod-ca-cert\") pod \"e2e-explicit-7719ed-openshift-default-7cf684688d-bjzms\" (UID: \"0511d07e-cb77-4534-bc61-48f06dfd3bf3\") " pod="openshift-ingress/e2e-explicit-7719ed-openshift-default-7cf684688d-bjzms"
Sep 11 23:21:14.396795 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:14.396625    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0511d07e-cb77-4534-bc61-48f06dfd3bf3-istio-token\") pod \"e2e-explicit-7719ed-openshift-default-7cf684688d-bjzms\" (UID: \"0511d07e-cb77-4534-bc61-48f06dfd3bf3\") " pod="openshift-ingress/e2e-explicit-7719ed-openshift-default-7cf684688d-bjzms"
Sep 11 23:21:14.396795 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:14.396672    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0511d07e-cb77-4534-bc61-48f06dfd3bf3-workload-socket\") pod \"e2e-explicit-7719ed-openshift-default-7cf684688d-bjzms\" (UID: \"0511d07e-cb77-4534-bc61-48f06dfd3bf3\") " pod="openshift-ingress/e2e-explicit-7719ed-openshift-default-7cf684688d-bjzms"
Sep 11 23:21:14.396795 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:14.396704    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0511d07e-cb77-4534-bc61-48f06dfd3bf3-istio-data\") pod \"e2e-explicit-7719ed-openshift-default-7cf684688d-bjzms\" (UID: \"0511d07e-cb77-4534-bc61-48f06dfd3bf3\") " pod="openshift-ingress/e2e-explicit-7719ed-openshift-default-7cf684688d-bjzms"
Sep 11 23:21:14.396795 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:14.396742    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7sp7\" (UniqueName: \"kubernetes.io/projected/0511d07e-cb77-4534-bc61-48f06dfd3bf3-kube-api-access-k7sp7\") pod \"e2e-explicit-7719ed-openshift-default-7cf684688d-bjzms\" (UID: \"0511d07e-cb77-4534-bc61-48f06dfd3bf3\") " pod="openshift-ingress/e2e-explicit-7719ed-openshift-default-7cf684688d-bjzms"
Sep 11 23:21:14.397029 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:14.396800    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0511d07e-cb77-4534-bc61-48f06dfd3bf3-credential-socket\") pod \"e2e-explicit-7719ed-openshift-default-7cf684688d-bjzms\" (UID: \"0511d07e-cb77-4534-bc61-48f06dfd3bf3\") " pod="openshift-ingress/e2e-explicit-7719ed-openshift-default-7cf684688d-bjzms"
Sep 11 23:21:14.397029 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:14.396828    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0511d07e-cb77-4534-bc61-48f06dfd3bf3-istio-envoy\") pod \"e2e-explicit-7719ed-openshift-default-7cf684688d-bjzms\" (UID: \"0511d07e-cb77-4534-bc61-48f06dfd3bf3\") " pod="openshift-ingress/e2e-explicit-7719ed-openshift-default-7cf684688d-bjzms"
Sep 11 23:21:14.397029 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:14.396887    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0511d07e-cb77-4534-bc61-48f06dfd3bf3-istio-podinfo\") pod \"e2e-explicit-7719ed-openshift-default-7cf684688d-bjzms\" (UID: \"0511d07e-cb77-4534-bc61-48f06dfd3bf3\") " pod="openshift-ingress/e2e-explicit-7719ed-openshift-default-7cf684688d-bjzms"
Sep 11 23:21:14.397029 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:14.396918    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0511d07e-cb77-4534-bc61-48f06dfd3bf3-workload-certs\") pod \"e2e-explicit-7719ed-openshift-default-7cf684688d-bjzms\" (UID: \"0511d07e-cb77-4534-bc61-48f06dfd3bf3\") " pod="openshift-ingress/e2e-explicit-7719ed-openshift-default-7cf684688d-bjzms"
Sep 11 23:21:14.397253 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:14.397085    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0511d07e-cb77-4534-bc61-48f06dfd3bf3-workload-socket\") pod \"e2e-explicit-7719ed-openshift-default-7cf684688d-bjzms\" (UID: \"0511d07e-cb77-4534-bc61-48f06dfd3bf3\") " pod="openshift-ingress/e2e-explicit-7719ed-openshift-default-7cf684688d-bjzms"
Sep 11 23:21:14.397253 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:14.397178    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0511d07e-cb77-4534-bc61-48f06dfd3bf3-istio-data\") pod \"e2e-explicit-7719ed-openshift-default-7cf684688d-bjzms\" (UID: \"0511d07e-cb77-4534-bc61-48f06dfd3bf3\") " pod="openshift-ingress/e2e-explicit-7719ed-openshift-default-7cf684688d-bjzms"
Sep 11 23:21:14.397253 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:14.397233    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0511d07e-cb77-4534-bc61-48f06dfd3bf3-credential-socket\") pod \"e2e-explicit-7719ed-openshift-default-7cf684688d-bjzms\" (UID: \"0511d07e-cb77-4534-bc61-48f06dfd3bf3\") " pod="openshift-ingress/e2e-explicit-7719ed-openshift-default-7cf684688d-bjzms"
Sep 11 23:21:14.397375 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:14.397354    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0511d07e-cb77-4534-bc61-48f06dfd3bf3-workload-certs\") pod \"e2e-explicit-7719ed-openshift-default-7cf684688d-bjzms\" (UID: \"0511d07e-cb77-4534-bc61-48f06dfd3bf3\") " pod="openshift-ingress/e2e-explicit-7719ed-openshift-default-7cf684688d-bjzms"
Sep 11 23:21:14.397410 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:14.397385    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0511d07e-cb77-4534-bc61-48f06dfd3bf3-istiod-ca-cert\") pod \"e2e-explicit-7719ed-openshift-default-7cf684688d-bjzms\" (UID: \"0511d07e-cb77-4534-bc61-48f06dfd3bf3\") " pod="openshift-ingress/e2e-explicit-7719ed-openshift-default-7cf684688d-bjzms"
Sep 11 23:21:14.399321 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:14.399289    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0511d07e-cb77-4534-bc61-48f06dfd3bf3-istio-envoy\") pod \"e2e-explicit-7719ed-openshift-default-7cf684688d-bjzms\" (UID: \"0511d07e-cb77-4534-bc61-48f06dfd3bf3\") " pod="openshift-ingress/e2e-explicit-7719ed-openshift-default-7cf684688d-bjzms"
Sep 11 23:21:14.399551 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:14.399530    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0511d07e-cb77-4534-bc61-48f06dfd3bf3-istio-podinfo\") pod \"e2e-explicit-7719ed-openshift-default-7cf684688d-bjzms\" (UID: \"0511d07e-cb77-4534-bc61-48f06dfd3bf3\") " pod="openshift-ingress/e2e-explicit-7719ed-openshift-default-7cf684688d-bjzms"
Sep 11 23:21:14.405265 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:14.405245    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0511d07e-cb77-4534-bc61-48f06dfd3bf3-istio-token\") pod \"e2e-explicit-7719ed-openshift-default-7cf684688d-bjzms\" (UID: \"0511d07e-cb77-4534-bc61-48f06dfd3bf3\") " pod="openshift-ingress/e2e-explicit-7719ed-openshift-default-7cf684688d-bjzms"
Sep 11 23:21:14.405436 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:14.405405    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7sp7\" (UniqueName: \"kubernetes.io/projected/0511d07e-cb77-4534-bc61-48f06dfd3bf3-kube-api-access-k7sp7\") pod \"e2e-explicit-7719ed-openshift-default-7cf684688d-bjzms\" (UID: \"0511d07e-cb77-4534-bc61-48f06dfd3bf3\") " pod="openshift-ingress/e2e-explicit-7719ed-openshift-default-7cf684688d-bjzms"
Sep 11 23:21:14.560045 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:14.559962    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-7719ed-openshift-default-7cf684688d-bjzms"
Sep 11 23:21:14.844501 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:14.844477    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-7719ed-openshift-default-7cf684688d-bjzms"]
Sep 11 23:21:14.846450 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:21:14.846422    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0511d07e_cb77_4534_bc61_48f06dfd3bf3.slice/crio-bca92236df4b5fe25d2729801d505d2643a00bc404fae1ec003313ea405cc206 WatchSource:0}: Error finding container bca92236df4b5fe25d2729801d505d2643a00bc404fae1ec003313ea405cc206: Status 404 returned error can't find the container with id bca92236df4b5fe25d2729801d505d2643a00bc404fae1ec003313ea405cc206
Sep 11 23:21:14.848906 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:14.848870    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:21:14.848989 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:14.848967    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:21:14.849030 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:14.849016    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:21:15.784686 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:15.784652    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-7719ed-openshift-default-7cf684688d-bjzms" event={"ID":"0511d07e-cb77-4534-bc61-48f06dfd3bf3","Type":"ContainerStarted","Data":"61783ba24aeb6addc8bddbcdeca4e8e7766b1e530d4ba7b43a29ce538d685630"}
Sep 11 23:21:15.785080 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:15.784707    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-7719ed-openshift-default-7cf684688d-bjzms" event={"ID":"0511d07e-cb77-4534-bc61-48f06dfd3bf3","Type":"ContainerStarted","Data":"bca92236df4b5fe25d2729801d505d2643a00bc404fae1ec003313ea405cc206"}
Sep 11 23:21:15.806639 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:15.806595    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-7719ed-openshift-default-7cf684688d-bjzms" podStartSLOduration=1.8065806100000001 podStartE2EDuration="1.80658061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 23:21:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:21:15.804957398 +0000 UTC m=+1199.838086592" watchObservedRunningTime="2026-09-11 23:21:15.80658061 +0000 UTC m=+1199.839709824"
Sep 11 23:21:16.527737 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:16.527708    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7244_6b0a8af3-932c-4091-b7a0-180fe1751a3a/ovn-acl-logging/0.log"
Sep 11 23:21:16.530507 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:16.530483    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7244_6b0a8af3-932c-4091-b7a0-180fe1751a3a/ovn-acl-logging/0.log"
Sep 11 23:21:16.560849 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:16.560824    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-7719ed-openshift-default-7cf684688d-bjzms"
Sep 11 23:21:16.566830 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:16.566807    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-7719ed-openshift-default-7cf684688d-bjzms"
Sep 11 23:21:16.792232 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:16.792126    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-7719ed-openshift-default-7cf684688d-bjzms"
Sep 11 23:21:16.793317 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:16.793296    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-7719ed-openshift-default-7cf684688d-bjzms"
Sep 11 23:21:18.803027 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:18.802969    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8"
Sep 11 23:21:18.805972 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:18.805949    2546 generic.go:356] "Generic (PLEG): container finished" podID="ad22eafb-8cf0-41a7-9286-ee3d645405c4" containerID="0ab78a91b61532cdd16422e9e880a364d0bf00f8d94ec0491a327e8f4f17fde8" exitCode=0
Sep 11 23:21:18.806073 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:18.805988    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8"
Sep 11 23:21:18.806073 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:18.806030    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8" event={"ID":"ad22eafb-8cf0-41a7-9286-ee3d645405c4","Type":"ContainerDied","Data":"0ab78a91b61532cdd16422e9e880a364d0bf00f8d94ec0491a327e8f4f17fde8"}
Sep 11 23:21:18.806146 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:18.806084    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8" event={"ID":"ad22eafb-8cf0-41a7-9286-ee3d645405c4","Type":"ContainerDied","Data":"7f5c0c3f0404ecbcb01dab5db7f7a40e9c7498a764e6ae512c6e4b15c122569d"}
Sep 11 23:21:18.806146 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:18.806116    2546 scope.go:117] "RemoveContainer" containerID="0ab78a91b61532cdd16422e9e880a364d0bf00f8d94ec0491a327e8f4f17fde8"
Sep 11 23:21:18.815396 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:18.815376    2546 scope.go:117] "RemoveContainer" containerID="a7ee56d8e50965ff0d1d6f5cd7f0856129ced129f7ad9a59d2372bee656adffe"
Sep 11 23:21:18.823723 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:18.823703    2546 scope.go:117] "RemoveContainer" containerID="0ab78a91b61532cdd16422e9e880a364d0bf00f8d94ec0491a327e8f4f17fde8"
Sep 11 23:21:18.824028 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:21:18.824008    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ab78a91b61532cdd16422e9e880a364d0bf00f8d94ec0491a327e8f4f17fde8\": container with ID starting with 0ab78a91b61532cdd16422e9e880a364d0bf00f8d94ec0491a327e8f4f17fde8 not found: ID does not exist" containerID="0ab78a91b61532cdd16422e9e880a364d0bf00f8d94ec0491a327e8f4f17fde8"
Sep 11 23:21:18.824083 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:18.824037    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ab78a91b61532cdd16422e9e880a364d0bf00f8d94ec0491a327e8f4f17fde8"} err="failed to get container status \"0ab78a91b61532cdd16422e9e880a364d0bf00f8d94ec0491a327e8f4f17fde8\": rpc error: code = NotFound desc = could not find container \"0ab78a91b61532cdd16422e9e880a364d0bf00f8d94ec0491a327e8f4f17fde8\": container with ID starting with 0ab78a91b61532cdd16422e9e880a364d0bf00f8d94ec0491a327e8f4f17fde8 not found: ID does not exist"
Sep 11 23:21:18.824083 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:18.824054    2546 scope.go:117] "RemoveContainer" containerID="a7ee56d8e50965ff0d1d6f5cd7f0856129ced129f7ad9a59d2372bee656adffe"
Sep 11 23:21:18.824343 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:21:18.824320    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7ee56d8e50965ff0d1d6f5cd7f0856129ced129f7ad9a59d2372bee656adffe\": container with ID starting with a7ee56d8e50965ff0d1d6f5cd7f0856129ced129f7ad9a59d2372bee656adffe not found: ID does not exist" containerID="a7ee56d8e50965ff0d1d6f5cd7f0856129ced129f7ad9a59d2372bee656adffe"
Sep 11 23:21:18.824392 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:18.824353    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7ee56d8e50965ff0d1d6f5cd7f0856129ced129f7ad9a59d2372bee656adffe"} err="failed to get container status \"a7ee56d8e50965ff0d1d6f5cd7f0856129ced129f7ad9a59d2372bee656adffe\": rpc error: code = NotFound desc = could not find container \"a7ee56d8e50965ff0d1d6f5cd7f0856129ced129f7ad9a59d2372bee656adffe\": container with ID starting with a7ee56d8e50965ff0d1d6f5cd7f0856129ced129f7ad9a59d2372bee656adffe not found: ID does not exist"
Sep 11 23:21:18.843777 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:18.843746    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad22eafb-8cf0-41a7-9286-ee3d645405c4-istio-token\") pod \"ad22eafb-8cf0-41a7-9286-ee3d645405c4\" (UID: \"ad22eafb-8cf0-41a7-9286-ee3d645405c4\") "
Sep 11 23:21:18.843930 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:18.843818    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad22eafb-8cf0-41a7-9286-ee3d645405c4-istiod-ca-cert\") pod \"ad22eafb-8cf0-41a7-9286-ee3d645405c4\" (UID: \"ad22eafb-8cf0-41a7-9286-ee3d645405c4\") "
Sep 11 23:21:18.843930 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:18.843852    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad22eafb-8cf0-41a7-9286-ee3d645405c4-workload-socket\") pod \"ad22eafb-8cf0-41a7-9286-ee3d645405c4\" (UID: \"ad22eafb-8cf0-41a7-9286-ee3d645405c4\") "
Sep 11 23:21:18.843930 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:18.843876    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-54bwf\" (UniqueName: \"kubernetes.io/projected/ad22eafb-8cf0-41a7-9286-ee3d645405c4-kube-api-access-54bwf\") pod \"ad22eafb-8cf0-41a7-9286-ee3d645405c4\" (UID: \"ad22eafb-8cf0-41a7-9286-ee3d645405c4\") "
Sep 11 23:21:18.844108 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:18.843929    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad22eafb-8cf0-41a7-9286-ee3d645405c4-workload-certs\") pod \"ad22eafb-8cf0-41a7-9286-ee3d645405c4\" (UID: \"ad22eafb-8cf0-41a7-9286-ee3d645405c4\") "
Sep 11 23:21:18.844108 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:18.843981    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad22eafb-8cf0-41a7-9286-ee3d645405c4-istio-envoy\") pod \"ad22eafb-8cf0-41a7-9286-ee3d645405c4\" (UID: \"ad22eafb-8cf0-41a7-9286-ee3d645405c4\") "
Sep 11 23:21:18.844108 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:18.844020    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad22eafb-8cf0-41a7-9286-ee3d645405c4-istio-data\") pod \"ad22eafb-8cf0-41a7-9286-ee3d645405c4\" (UID: \"ad22eafb-8cf0-41a7-9286-ee3d645405c4\") "
Sep 11 23:21:18.844279 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:18.844124    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad22eafb-8cf0-41a7-9286-ee3d645405c4-istio-podinfo\") pod \"ad22eafb-8cf0-41a7-9286-ee3d645405c4\" (UID: \"ad22eafb-8cf0-41a7-9286-ee3d645405c4\") "
Sep 11 23:21:18.844279 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:18.844181    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad22eafb-8cf0-41a7-9286-ee3d645405c4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ad22eafb-8cf0-41a7-9286-ee3d645405c4" (UID: "ad22eafb-8cf0-41a7-9286-ee3d645405c4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:21:18.844279 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:18.844197    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad22eafb-8cf0-41a7-9286-ee3d645405c4-credential-socket\") pod \"ad22eafb-8cf0-41a7-9286-ee3d645405c4\" (UID: \"ad22eafb-8cf0-41a7-9286-ee3d645405c4\") "
Sep 11 23:21:18.844279 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:18.844270    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad22eafb-8cf0-41a7-9286-ee3d645405c4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ad22eafb-8cf0-41a7-9286-ee3d645405c4" (UID: "ad22eafb-8cf0-41a7-9286-ee3d645405c4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:21:18.844489 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:18.844394    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad22eafb-8cf0-41a7-9286-ee3d645405c4-istio-data" (OuterVolumeSpecName: "istio-data") pod "ad22eafb-8cf0-41a7-9286-ee3d645405c4" (UID: "ad22eafb-8cf0-41a7-9286-ee3d645405c4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:21:18.844553 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:18.844531    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad22eafb-8cf0-41a7-9286-ee3d645405c4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ad22eafb-8cf0-41a7-9286-ee3d645405c4" (UID: "ad22eafb-8cf0-41a7-9286-ee3d645405c4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:21:18.844602 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:18.844558    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad22eafb-8cf0-41a7-9286-ee3d645405c4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ad22eafb-8cf0-41a7-9286-ee3d645405c4" (UID: "ad22eafb-8cf0-41a7-9286-ee3d645405c4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 23:21:18.845092 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:18.844847    2546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad22eafb-8cf0-41a7-9286-ee3d645405c4-credential-socket\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:21:18.845092 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:18.844995    2546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad22eafb-8cf0-41a7-9286-ee3d645405c4-istiod-ca-cert\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:21:18.845092 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:18.845017    2546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad22eafb-8cf0-41a7-9286-ee3d645405c4-workload-socket\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:21:18.845092 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:18.845033    2546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad22eafb-8cf0-41a7-9286-ee3d645405c4-workload-certs\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:21:18.845092 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:18.845048    2546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad22eafb-8cf0-41a7-9286-ee3d645405c4-istio-data\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:21:18.846514 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:18.846492    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad22eafb-8cf0-41a7-9286-ee3d645405c4-istio-token" (OuterVolumeSpecName: "istio-token") pod "ad22eafb-8cf0-41a7-9286-ee3d645405c4" (UID: "ad22eafb-8cf0-41a7-9286-ee3d645405c4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 23:21:18.846650 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:18.846624    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad22eafb-8cf0-41a7-9286-ee3d645405c4-kube-api-access-54bwf" (OuterVolumeSpecName: "kube-api-access-54bwf") pod "ad22eafb-8cf0-41a7-9286-ee3d645405c4" (UID: "ad22eafb-8cf0-41a7-9286-ee3d645405c4"). InnerVolumeSpecName "kube-api-access-54bwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 23:21:18.846936 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:18.846916    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ad22eafb-8cf0-41a7-9286-ee3d645405c4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ad22eafb-8cf0-41a7-9286-ee3d645405c4" (UID: "ad22eafb-8cf0-41a7-9286-ee3d645405c4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 23:21:18.847233 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:18.847201    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad22eafb-8cf0-41a7-9286-ee3d645405c4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ad22eafb-8cf0-41a7-9286-ee3d645405c4" (UID: "ad22eafb-8cf0-41a7-9286-ee3d645405c4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:21:18.945873 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:18.945843    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54bwf\" (UniqueName: \"kubernetes.io/projected/ad22eafb-8cf0-41a7-9286-ee3d645405c4-kube-api-access-54bwf\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:21:18.945873 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:18.945872    2546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad22eafb-8cf0-41a7-9286-ee3d645405c4-istio-envoy\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:21:18.945873 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:18.945881    2546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad22eafb-8cf0-41a7-9286-ee3d645405c4-istio-podinfo\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:21:18.946097 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:18.945890    2546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad22eafb-8cf0-41a7-9286-ee3d645405c4-istio-token\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:21:19.144988 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:19.144951    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8"]
Sep 11 23:21:19.156444 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:19.156413    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-0a7a66-openshift-default-57c46f5964-j4wm8"]
Sep 11 23:21:20.499255 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:20.499217    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad22eafb-8cf0-41a7-9286-ee3d645405c4" path="/var/lib/kubelet/pods/ad22eafb-8cf0-41a7-9286-ee3d645405c4/volumes"
Sep 11 23:21:29.991879 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:29.991848    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"]
Sep 11 23:21:35.532005 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:35.531969    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-12fb6b/premium-w0-12fb6b-kserve-557db4c777-5zbbn"]
Sep 11 23:21:35.533168 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:35.533125    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad22eafb-8cf0-41a7-9286-ee3d645405c4" containerName="istio-proxy"
Sep 11 23:21:35.533317 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:35.533305    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad22eafb-8cf0-41a7-9286-ee3d645405c4" containerName="istio-proxy"
Sep 11 23:21:35.533560 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:35.533547    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad22eafb-8cf0-41a7-9286-ee3d645405c4" containerName="istio-proxy"
Sep 11 23:21:35.533648 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:35.533639    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad22eafb-8cf0-41a7-9286-ee3d645405c4" containerName="istio-proxy"
Sep 11 23:21:35.533721 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:35.533713    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="71dd368e-c415-465c-838a-ecd215bc4069" containerName="istio-proxy"
Sep 11 23:21:35.553020 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:35.552996    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-12fb6b/premium-w0-12fb6b-kserve-557db4c777-5zbbn"
Sep 11 23:21:35.557898 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:35.557569    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-12fb6b\"/\"premium-w0-12fb6b-kserve-self-signed-certs\""
Sep 11 23:21:35.558037 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:35.557896    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-12fb6b/premium-w0-12fb6b-kserve-557db4c777-5zbbn"]
Sep 11 23:21:35.719749 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:35.719707    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74942f9b-139d-4bee-9380-c251ac6dc246-model-cache\") pod \"premium-w0-12fb6b-kserve-557db4c777-5zbbn\" (UID: \"74942f9b-139d-4bee-9380-c251ac6dc246\") " pod="e2e-models-e2e-worker-w0-12fb6b/premium-w0-12fb6b-kserve-557db4c777-5zbbn"
Sep 11 23:21:35.720031 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:35.720010    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jflrm\" (UniqueName: \"kubernetes.io/projected/74942f9b-139d-4bee-9380-c251ac6dc246-kube-api-access-jflrm\") pod \"premium-w0-12fb6b-kserve-557db4c777-5zbbn\" (UID: \"74942f9b-139d-4bee-9380-c251ac6dc246\") " pod="e2e-models-e2e-worker-w0-12fb6b/premium-w0-12fb6b-kserve-557db4c777-5zbbn"
Sep 11 23:21:35.720219 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:35.720203    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74942f9b-139d-4bee-9380-c251ac6dc246-tls-certs\") pod \"premium-w0-12fb6b-kserve-557db4c777-5zbbn\" (UID: \"74942f9b-139d-4bee-9380-c251ac6dc246\") " pod="e2e-models-e2e-worker-w0-12fb6b/premium-w0-12fb6b-kserve-557db4c777-5zbbn"
Sep 11 23:21:35.720418 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:35.720404    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74942f9b-139d-4bee-9380-c251ac6dc246-tmp-dir\") pod \"premium-w0-12fb6b-kserve-557db4c777-5zbbn\" (UID: \"74942f9b-139d-4bee-9380-c251ac6dc246\") " pod="e2e-models-e2e-worker-w0-12fb6b/premium-w0-12fb6b-kserve-557db4c777-5zbbn"
Sep 11 23:21:35.720566 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:35.720551    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74942f9b-139d-4bee-9380-c251ac6dc246-dshm\") pod \"premium-w0-12fb6b-kserve-557db4c777-5zbbn\" (UID: \"74942f9b-139d-4bee-9380-c251ac6dc246\") " pod="e2e-models-e2e-worker-w0-12fb6b/premium-w0-12fb6b-kserve-557db4c777-5zbbn"
Sep 11 23:21:35.720715 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:35.720703    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74942f9b-139d-4bee-9380-c251ac6dc246-home\") pod \"premium-w0-12fb6b-kserve-557db4c777-5zbbn\" (UID: \"74942f9b-139d-4bee-9380-c251ac6dc246\") " pod="e2e-models-e2e-worker-w0-12fb6b/premium-w0-12fb6b-kserve-557db4c777-5zbbn"
Sep 11 23:21:35.821924 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:35.821838    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74942f9b-139d-4bee-9380-c251ac6dc246-model-cache\") pod \"premium-w0-12fb6b-kserve-557db4c777-5zbbn\" (UID: \"74942f9b-139d-4bee-9380-c251ac6dc246\") " pod="e2e-models-e2e-worker-w0-12fb6b/premium-w0-12fb6b-kserve-557db4c777-5zbbn"
Sep 11 23:21:35.821924 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:35.821888    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jflrm\" (UniqueName: \"kubernetes.io/projected/74942f9b-139d-4bee-9380-c251ac6dc246-kube-api-access-jflrm\") pod \"premium-w0-12fb6b-kserve-557db4c777-5zbbn\" (UID: \"74942f9b-139d-4bee-9380-c251ac6dc246\") " pod="e2e-models-e2e-worker-w0-12fb6b/premium-w0-12fb6b-kserve-557db4c777-5zbbn"
Sep 11 23:21:35.821924 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:35.821917    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74942f9b-139d-4bee-9380-c251ac6dc246-tls-certs\") pod \"premium-w0-12fb6b-kserve-557db4c777-5zbbn\" (UID: \"74942f9b-139d-4bee-9380-c251ac6dc246\") " pod="e2e-models-e2e-worker-w0-12fb6b/premium-w0-12fb6b-kserve-557db4c777-5zbbn"
Sep 11 23:21:35.822252 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:35.821958    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74942f9b-139d-4bee-9380-c251ac6dc246-tmp-dir\") pod \"premium-w0-12fb6b-kserve-557db4c777-5zbbn\" (UID: \"74942f9b-139d-4bee-9380-c251ac6dc246\") " pod="e2e-models-e2e-worker-w0-12fb6b/premium-w0-12fb6b-kserve-557db4c777-5zbbn"
Sep 11 23:21:35.822252 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:35.821980    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74942f9b-139d-4bee-9380-c251ac6dc246-dshm\") pod \"premium-w0-12fb6b-kserve-557db4c777-5zbbn\" (UID: \"74942f9b-139d-4bee-9380-c251ac6dc246\") " pod="e2e-models-e2e-worker-w0-12fb6b/premium-w0-12fb6b-kserve-557db4c777-5zbbn"
Sep 11 23:21:35.822252 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:35.822128    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74942f9b-139d-4bee-9380-c251ac6dc246-home\") pod \"premium-w0-12fb6b-kserve-557db4c777-5zbbn\" (UID: \"74942f9b-139d-4bee-9380-c251ac6dc246\") " pod="e2e-models-e2e-worker-w0-12fb6b/premium-w0-12fb6b-kserve-557db4c777-5zbbn"
Sep 11 23:21:35.822376 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:35.822316    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74942f9b-139d-4bee-9380-c251ac6dc246-model-cache\") pod \"premium-w0-12fb6b-kserve-557db4c777-5zbbn\" (UID: \"74942f9b-139d-4bee-9380-c251ac6dc246\") " pod="e2e-models-e2e-worker-w0-12fb6b/premium-w0-12fb6b-kserve-557db4c777-5zbbn"
Sep 11 23:21:35.822376 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:35.822365    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74942f9b-139d-4bee-9380-c251ac6dc246-tmp-dir\") pod \"premium-w0-12fb6b-kserve-557db4c777-5zbbn\" (UID: \"74942f9b-139d-4bee-9380-c251ac6dc246\") " pod="e2e-models-e2e-worker-w0-12fb6b/premium-w0-12fb6b-kserve-557db4c777-5zbbn"
Sep 11 23:21:35.822484 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:35.822453    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74942f9b-139d-4bee-9380-c251ac6dc246-home\") pod \"premium-w0-12fb6b-kserve-557db4c777-5zbbn\" (UID: \"74942f9b-139d-4bee-9380-c251ac6dc246\") " pod="e2e-models-e2e-worker-w0-12fb6b/premium-w0-12fb6b-kserve-557db4c777-5zbbn"
Sep 11 23:21:35.827409 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:35.827382    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74942f9b-139d-4bee-9380-c251ac6dc246-dshm\") pod \"premium-w0-12fb6b-kserve-557db4c777-5zbbn\" (UID: \"74942f9b-139d-4bee-9380-c251ac6dc246\") " pod="e2e-models-e2e-worker-w0-12fb6b/premium-w0-12fb6b-kserve-557db4c777-5zbbn"
Sep 11 23:21:35.832172 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:35.831115    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jflrm\" (UniqueName: \"kubernetes.io/projected/74942f9b-139d-4bee-9380-c251ac6dc246-kube-api-access-jflrm\") pod \"premium-w0-12fb6b-kserve-557db4c777-5zbbn\" (UID: \"74942f9b-139d-4bee-9380-c251ac6dc246\") " pod="e2e-models-e2e-worker-w0-12fb6b/premium-w0-12fb6b-kserve-557db4c777-5zbbn"
Sep 11 23:21:35.833683 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:35.833656    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74942f9b-139d-4bee-9380-c251ac6dc246-tls-certs\") pod \"premium-w0-12fb6b-kserve-557db4c777-5zbbn\" (UID: \"74942f9b-139d-4bee-9380-c251ac6dc246\") " pod="e2e-models-e2e-worker-w0-12fb6b/premium-w0-12fb6b-kserve-557db4c777-5zbbn"
Sep 11 23:21:35.865343 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:35.865315    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-12fb6b/premium-w0-12fb6b-kserve-557db4c777-5zbbn"
Sep 11 23:21:36.152344 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:36.152118    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-12fb6b/premium-w0-12fb6b-kserve-557db4c777-5zbbn"]
Sep 11 23:21:36.154989 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:21:36.154963    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod74942f9b_139d_4bee_9380_c251ac6dc246.slice/crio-a7843d3ae9110528901299eb03474f7309982c4fff978f1b60fa7375264de9f8 WatchSource:0}: Error finding container a7843d3ae9110528901299eb03474f7309982c4fff978f1b60fa7375264de9f8: Status 404 returned error can't find the container with id a7843d3ae9110528901299eb03474f7309982c4fff978f1b60fa7375264de9f8
Sep 11 23:21:36.921954 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:36.921914    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-12fb6b/premium-w0-12fb6b-kserve-557db4c777-5zbbn" event={"ID":"74942f9b-139d-4bee-9380-c251ac6dc246","Type":"ContainerStarted","Data":"f842f454d6c900755eaf2f8137c9d8810ad3a6e983d8a1c9e7171fbfc8c7f708"}
Sep 11 23:21:36.922358 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:36.921965    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-12fb6b/premium-w0-12fb6b-kserve-557db4c777-5zbbn" event={"ID":"74942f9b-139d-4bee-9380-c251ac6dc246","Type":"ContainerStarted","Data":"a7843d3ae9110528901299eb03474f7309982c4fff978f1b60fa7375264de9f8"}
Sep 11 23:21:36.946534 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:36.946455    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-12fb6b/premium-w0-12fb6b-kserve-557db4c777-5zbbn" podStartSLOduration=1.946433124 podStartE2EDuration="1.946433124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 23:21:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:21:36.939837726 +0000 UTC m=+1220.972966942" watchObservedRunningTime="2026-09-11 23:21:36.946433124 +0000 UTC m=+1220.979562341"
Sep 11 23:21:40.951425 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:40.951388    2546 generic.go:356] "Generic (PLEG): container finished" podID="0511d07e-cb77-4534-bc61-48f06dfd3bf3" containerID="61783ba24aeb6addc8bddbcdeca4e8e7766b1e530d4ba7b43a29ce538d685630" exitCode=0
Sep 11 23:21:40.951873 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:40.951465    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-7719ed-openshift-default-7cf684688d-bjzms" event={"ID":"0511d07e-cb77-4534-bc61-48f06dfd3bf3","Type":"ContainerDied","Data":"61783ba24aeb6addc8bddbcdeca4e8e7766b1e530d4ba7b43a29ce538d685630"}
Sep 11 23:21:40.951919 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:40.951899    2546 scope.go:117] "RemoveContainer" containerID="61783ba24aeb6addc8bddbcdeca4e8e7766b1e530d4ba7b43a29ce538d685630"
Sep 11 23:21:40.953061 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:40.953036    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:21:40.953135 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:40.953100    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:21:40.953135 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:40.953128    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:21:41.962365 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:41.962332    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-7719ed-openshift-default-7cf684688d-bjzms" event={"ID":"0511d07e-cb77-4534-bc61-48f06dfd3bf3","Type":"ContainerStarted","Data":"5011aeb3872e42b718d85a3664e15fda79e9576ae31647d6431ea43ae00b97b2"}
Sep 11 23:21:42.561118 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:42.561069    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-7719ed-openshift-default-7cf684688d-bjzms"
Sep 11 23:21:42.562661 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:42.562628    2546 patch_prober.go:28] interesting pod/e2e-explicit-7719ed-openshift-default-7cf684688d-bjzms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Sep 11 23:21:42.562813 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:42.562689    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-7719ed-openshift-default-7cf684688d-bjzms" podUID="0511d07e-cb77-4534-bc61-48f06dfd3bf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Sep 11 23:21:43.560550 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:43.560515    2546 patch_prober.go:28] interesting pod/e2e-explicit-7719ed-openshift-default-7cf684688d-bjzms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Sep 11 23:21:43.561025 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:43.560999    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-7719ed-openshift-default-7cf684688d-bjzms" podUID="0511d07e-cb77-4534-bc61-48f06dfd3bf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Sep 11 23:21:44.561426 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:44.561391    2546 patch_prober.go:28] interesting pod/e2e-explicit-7719ed-openshift-default-7cf684688d-bjzms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Sep 11 23:21:44.561912 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:44.561526    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-7719ed-openshift-default-7cf684688d-bjzms" podUID="0511d07e-cb77-4534-bc61-48f06dfd3bf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Sep 11 23:21:45.560408 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:45.560372    2546 patch_prober.go:28] interesting pod/e2e-explicit-7719ed-openshift-default-7cf684688d-bjzms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Sep 11 23:21:45.560584 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:45.560425    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-7719ed-openshift-default-7cf684688d-bjzms" podUID="0511d07e-cb77-4534-bc61-48f06dfd3bf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Sep 11 23:21:45.865876 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:45.865797    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-12fb6b/premium-w0-12fb6b-kserve-557db4c777-5zbbn"
Sep 11 23:21:45.866341 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:45.866327    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-12fb6b/premium-w0-12fb6b-kserve-557db4c777-5zbbn"
Sep 11 23:21:45.878289 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:45.878266    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-12fb6b/premium-w0-12fb6b-kserve-557db4c777-5zbbn"
Sep 11 23:21:46.005494 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:46.005469    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-12fb6b/premium-w0-12fb6b-kserve-557db4c777-5zbbn"
Sep 11 23:21:46.561202 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:46.561171    2546 patch_prober.go:28] interesting pod/e2e-explicit-7719ed-openshift-default-7cf684688d-bjzms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Sep 11 23:21:46.561471 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:46.561446    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-7719ed-openshift-default-7cf684688d-bjzms" podUID="0511d07e-cb77-4534-bc61-48f06dfd3bf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Sep 11 23:21:46.792607 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:46.792576    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-7719ed-openshift-default-7cf684688d-bjzms"
Sep 11 23:21:47.564212 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:47.564148    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-7719ed-openshift-default-7cf684688d-bjzms"
Sep 11 23:21:47.565064 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:47.565045    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-7719ed-openshift-default-7cf684688d-bjzms"
Sep 11 23:21:47.733780 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:47.733738    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-bfa694/test-model-bfa694-kserve-6949bf5cdf-xjwpl"]
Sep 11 23:21:47.734309 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:47.734290    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad22eafb-8cf0-41a7-9286-ee3d645405c4" containerName="istio-proxy"
Sep 11 23:21:47.734309 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:47.734310    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad22eafb-8cf0-41a7-9286-ee3d645405c4" containerName="istio-proxy"
Sep 11 23:21:47.737876 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:47.737852    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-bfa694/test-model-bfa694-kserve-6949bf5cdf-xjwpl"
Sep 11 23:21:47.742510 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:47.741754    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-bfa694\"/\"kube-root-ca.crt\""
Sep 11 23:21:47.742510 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:47.741933    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-bfa694\"/\"default-dockercfg-d5th7\""
Sep 11 23:21:47.742510 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:47.742038    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-bfa694\"/\"openshift-service-ca.crt\""
Sep 11 23:21:47.742510 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:47.742358    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-bfa694\"/\"test-model-bfa694-kserve-self-signed-certs\""
Sep 11 23:21:47.745939 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:47.745841    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-bfa694/test-model-bfa694-kserve-6949bf5cdf-xjwpl"]
Sep 11 23:21:47.766447 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:47.766420    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2abb209f-6719-4bbb-acc5-3e9a1d9b2f94-home\") pod \"test-model-bfa694-kserve-6949bf5cdf-xjwpl\" (UID: \"2abb209f-6719-4bbb-acc5-3e9a1d9b2f94\") " pod="ai-tenant-e2e-inf-b-bfa694/test-model-bfa694-kserve-6949bf5cdf-xjwpl"
Sep 11 23:21:47.766602 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:47.766459    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2abb209f-6719-4bbb-acc5-3e9a1d9b2f94-tmp-dir\") pod \"test-model-bfa694-kserve-6949bf5cdf-xjwpl\" (UID: \"2abb209f-6719-4bbb-acc5-3e9a1d9b2f94\") " pod="ai-tenant-e2e-inf-b-bfa694/test-model-bfa694-kserve-6949bf5cdf-xjwpl"
Sep 11 23:21:47.766602 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:47.766501    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2abb209f-6719-4bbb-acc5-3e9a1d9b2f94-model-cache\") pod \"test-model-bfa694-kserve-6949bf5cdf-xjwpl\" (UID: \"2abb209f-6719-4bbb-acc5-3e9a1d9b2f94\") " pod="ai-tenant-e2e-inf-b-bfa694/test-model-bfa694-kserve-6949bf5cdf-xjwpl"
Sep 11 23:21:47.766602 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:47.766572    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2abb209f-6719-4bbb-acc5-3e9a1d9b2f94-tls-certs\") pod \"test-model-bfa694-kserve-6949bf5cdf-xjwpl\" (UID: \"2abb209f-6719-4bbb-acc5-3e9a1d9b2f94\") " pod="ai-tenant-e2e-inf-b-bfa694/test-model-bfa694-kserve-6949bf5cdf-xjwpl"
Sep 11 23:21:47.766742 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:47.766615    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2abb209f-6719-4bbb-acc5-3e9a1d9b2f94-dshm\") pod \"test-model-bfa694-kserve-6949bf5cdf-xjwpl\" (UID: \"2abb209f-6719-4bbb-acc5-3e9a1d9b2f94\") " pod="ai-tenant-e2e-inf-b-bfa694/test-model-bfa694-kserve-6949bf5cdf-xjwpl"
Sep 11 23:21:47.766789 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:47.766756    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwvz8\" (UniqueName: \"kubernetes.io/projected/2abb209f-6719-4bbb-acc5-3e9a1d9b2f94-kube-api-access-lwvz8\") pod \"test-model-bfa694-kserve-6949bf5cdf-xjwpl\" (UID: \"2abb209f-6719-4bbb-acc5-3e9a1d9b2f94\") " pod="ai-tenant-e2e-inf-b-bfa694/test-model-bfa694-kserve-6949bf5cdf-xjwpl"
Sep 11 23:21:47.868412 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:47.868267    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwvz8\" (UniqueName: \"kubernetes.io/projected/2abb209f-6719-4bbb-acc5-3e9a1d9b2f94-kube-api-access-lwvz8\") pod \"test-model-bfa694-kserve-6949bf5cdf-xjwpl\" (UID: \"2abb209f-6719-4bbb-acc5-3e9a1d9b2f94\") " pod="ai-tenant-e2e-inf-b-bfa694/test-model-bfa694-kserve-6949bf5cdf-xjwpl"
Sep 11 23:21:47.868412 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:47.868377    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2abb209f-6719-4bbb-acc5-3e9a1d9b2f94-home\") pod \"test-model-bfa694-kserve-6949bf5cdf-xjwpl\" (UID: \"2abb209f-6719-4bbb-acc5-3e9a1d9b2f94\") " pod="ai-tenant-e2e-inf-b-bfa694/test-model-bfa694-kserve-6949bf5cdf-xjwpl"
Sep 11 23:21:47.868412 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:47.868402    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2abb209f-6719-4bbb-acc5-3e9a1d9b2f94-tmp-dir\") pod \"test-model-bfa694-kserve-6949bf5cdf-xjwpl\" (UID: \"2abb209f-6719-4bbb-acc5-3e9a1d9b2f94\") " pod="ai-tenant-e2e-inf-b-bfa694/test-model-bfa694-kserve-6949bf5cdf-xjwpl"
Sep 11 23:21:47.868642 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:47.868432    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2abb209f-6719-4bbb-acc5-3e9a1d9b2f94-model-cache\") pod \"test-model-bfa694-kserve-6949bf5cdf-xjwpl\" (UID: \"2abb209f-6719-4bbb-acc5-3e9a1d9b2f94\") " pod="ai-tenant-e2e-inf-b-bfa694/test-model-bfa694-kserve-6949bf5cdf-xjwpl"
Sep 11 23:21:47.868642 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:47.868465    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2abb209f-6719-4bbb-acc5-3e9a1d9b2f94-tls-certs\") pod \"test-model-bfa694-kserve-6949bf5cdf-xjwpl\" (UID: \"2abb209f-6719-4bbb-acc5-3e9a1d9b2f94\") " pod="ai-tenant-e2e-inf-b-bfa694/test-model-bfa694-kserve-6949bf5cdf-xjwpl"
Sep 11 23:21:47.868642 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:47.868495    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2abb209f-6719-4bbb-acc5-3e9a1d9b2f94-dshm\") pod \"test-model-bfa694-kserve-6949bf5cdf-xjwpl\" (UID: \"2abb209f-6719-4bbb-acc5-3e9a1d9b2f94\") " pod="ai-tenant-e2e-inf-b-bfa694/test-model-bfa694-kserve-6949bf5cdf-xjwpl"
Sep 11 23:21:47.868829 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:47.868804    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2abb209f-6719-4bbb-acc5-3e9a1d9b2f94-home\") pod \"test-model-bfa694-kserve-6949bf5cdf-xjwpl\" (UID: \"2abb209f-6719-4bbb-acc5-3e9a1d9b2f94\") " pod="ai-tenant-e2e-inf-b-bfa694/test-model-bfa694-kserve-6949bf5cdf-xjwpl"
Sep 11 23:21:47.868874 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:47.868839    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2abb209f-6719-4bbb-acc5-3e9a1d9b2f94-tmp-dir\") pod \"test-model-bfa694-kserve-6949bf5cdf-xjwpl\" (UID: \"2abb209f-6719-4bbb-acc5-3e9a1d9b2f94\") " pod="ai-tenant-e2e-inf-b-bfa694/test-model-bfa694-kserve-6949bf5cdf-xjwpl"
Sep 11 23:21:47.868874 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:47.868854    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2abb209f-6719-4bbb-acc5-3e9a1d9b2f94-model-cache\") pod \"test-model-bfa694-kserve-6949bf5cdf-xjwpl\" (UID: \"2abb209f-6719-4bbb-acc5-3e9a1d9b2f94\") " pod="ai-tenant-e2e-inf-b-bfa694/test-model-bfa694-kserve-6949bf5cdf-xjwpl"
Sep 11 23:21:47.870862 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:47.870841    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2abb209f-6719-4bbb-acc5-3e9a1d9b2f94-dshm\") pod \"test-model-bfa694-kserve-6949bf5cdf-xjwpl\" (UID: \"2abb209f-6719-4bbb-acc5-3e9a1d9b2f94\") " pod="ai-tenant-e2e-inf-b-bfa694/test-model-bfa694-kserve-6949bf5cdf-xjwpl"
Sep 11 23:21:47.871094 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:47.871079    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2abb209f-6719-4bbb-acc5-3e9a1d9b2f94-tls-certs\") pod \"test-model-bfa694-kserve-6949bf5cdf-xjwpl\" (UID: \"2abb209f-6719-4bbb-acc5-3e9a1d9b2f94\") " pod="ai-tenant-e2e-inf-b-bfa694/test-model-bfa694-kserve-6949bf5cdf-xjwpl"
Sep 11 23:21:47.876526 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:47.876504    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwvz8\" (UniqueName: \"kubernetes.io/projected/2abb209f-6719-4bbb-acc5-3e9a1d9b2f94-kube-api-access-lwvz8\") pod \"test-model-bfa694-kserve-6949bf5cdf-xjwpl\" (UID: \"2abb209f-6719-4bbb-acc5-3e9a1d9b2f94\") " pod="ai-tenant-e2e-inf-b-bfa694/test-model-bfa694-kserve-6949bf5cdf-xjwpl"
Sep 11 23:21:48.052314 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:48.052272    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-bfa694/test-model-bfa694-kserve-6949bf5cdf-xjwpl"
Sep 11 23:21:48.330237 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:48.330211    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-bfa694/test-model-bfa694-kserve-6949bf5cdf-xjwpl"]
Sep 11 23:21:48.332273 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:21:48.332244    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2abb209f_6719_4bbb_acc5_3e9a1d9b2f94.slice/crio-4a248cb6e2f1792507d9236e7515f5105a0fa5f72109797f2781f35c4f0cd707 WatchSource:0}: Error finding container 4a248cb6e2f1792507d9236e7515f5105a0fa5f72109797f2781f35c4f0cd707: Status 404 returned error can't find the container with id 4a248cb6e2f1792507d9236e7515f5105a0fa5f72109797f2781f35c4f0cd707
Sep 11 23:21:49.015857 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:49.015825    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-bfa694/test-model-bfa694-kserve-6949bf5cdf-xjwpl" event={"ID":"2abb209f-6719-4bbb-acc5-3e9a1d9b2f94","Type":"ContainerStarted","Data":"3db1d9c21696faccf6528f7153118955d82c36574446ee3040c199fc55c7851d"}
Sep 11 23:21:49.015857 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:49.015864    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-bfa694/test-model-bfa694-kserve-6949bf5cdf-xjwpl" event={"ID":"2abb209f-6719-4bbb-acc5-3e9a1d9b2f94","Type":"ContainerStarted","Data":"4a248cb6e2f1792507d9236e7515f5105a0fa5f72109797f2781f35c4f0cd707"}
Sep 11 23:21:49.037143 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:49.037089    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-bfa694/test-model-bfa694-kserve-6949bf5cdf-xjwpl" podStartSLOduration=2.037074683 podStartE2EDuration="2.037074683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 23:21:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:21:49.036238658 +0000 UTC m=+1233.069367876" watchObservedRunningTime="2026-09-11 23:21:49.037074683 +0000 UTC m=+1233.070203899"
Sep 11 23:21:49.390940 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:49.390859    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"]
Sep 11 23:21:58.052633 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:58.052595    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-bfa694/test-model-bfa694-kserve-6949bf5cdf-xjwpl"
Sep 11 23:21:58.053125 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:58.052648    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-bfa694/test-model-bfa694-kserve-6949bf5cdf-xjwpl"
Sep 11 23:21:58.065578 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:58.065546    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-bfa694/test-model-bfa694-kserve-6949bf5cdf-xjwpl"
Sep 11 23:21:58.084004 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:21:58.083962    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-bfa694/test-model-bfa694-kserve-6949bf5cdf-xjwpl"
Sep 11 23:22:12.200890 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:22:12.200807    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"]
Sep 11 23:22:28.936811 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:22:28.936780    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-1ebfb31f-openshift-default-5fb66f6676-wpbr6"]
Sep 11 23:22:28.941811 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:22:28.941794    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1ebfb31f-openshift-default-5fb66f6676-wpbr6"
Sep 11 23:22:28.945141 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:22:28.945117    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-1ebfb31f-openshift-default-dockercfg-58zvt\""
Sep 11 23:22:28.956858 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:22:28.956827    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1ebfb31f-openshift-default-5fb66f6676-wpbr6"]
Sep 11 23:22:28.983880 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:22:28.983852    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4575ea1-784b-4c29-9eff-a2d23d6772a8-credential-socket\") pod \"e2e-mt-1ebfb31f-openshift-default-5fb66f6676-wpbr6\" (UID: \"f4575ea1-784b-4c29-9eff-a2d23d6772a8\") " pod="openshift-ingress/e2e-mt-1ebfb31f-openshift-default-5fb66f6676-wpbr6"
Sep 11 23:22:28.984032 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:22:28.983894    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4575ea1-784b-4c29-9eff-a2d23d6772a8-istio-podinfo\") pod \"e2e-mt-1ebfb31f-openshift-default-5fb66f6676-wpbr6\" (UID: \"f4575ea1-784b-4c29-9eff-a2d23d6772a8\") " pod="openshift-ingress/e2e-mt-1ebfb31f-openshift-default-5fb66f6676-wpbr6"
Sep 11 23:22:28.984032 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:22:28.983918    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh2ks\" (UniqueName: \"kubernetes.io/projected/f4575ea1-784b-4c29-9eff-a2d23d6772a8-kube-api-access-hh2ks\") pod \"e2e-mt-1ebfb31f-openshift-default-5fb66f6676-wpbr6\" (UID: \"f4575ea1-784b-4c29-9eff-a2d23d6772a8\") " pod="openshift-ingress/e2e-mt-1ebfb31f-openshift-default-5fb66f6676-wpbr6"
Sep 11 23:22:28.984032 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:22:28.984012    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4575ea1-784b-4c29-9eff-a2d23d6772a8-istio-token\") pod \"e2e-mt-1ebfb31f-openshift-default-5fb66f6676-wpbr6\" (UID: \"f4575ea1-784b-4c29-9eff-a2d23d6772a8\") " pod="openshift-ingress/e2e-mt-1ebfb31f-openshift-default-5fb66f6676-wpbr6"
Sep 11 23:22:28.984195 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:22:28.984087    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4575ea1-784b-4c29-9eff-a2d23d6772a8-istio-envoy\") pod \"e2e-mt-1ebfb31f-openshift-default-5fb66f6676-wpbr6\" (UID: \"f4575ea1-784b-4c29-9eff-a2d23d6772a8\") " pod="openshift-ingress/e2e-mt-1ebfb31f-openshift-default-5fb66f6676-wpbr6"
Sep 11 23:22:28.984195 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:22:28.984122    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4575ea1-784b-4c29-9eff-a2d23d6772a8-workload-certs\") pod \"e2e-mt-1ebfb31f-openshift-default-5fb66f6676-wpbr6\" (UID: \"f4575ea1-784b-4c29-9eff-a2d23d6772a8\") " pod="openshift-ingress/e2e-mt-1ebfb31f-openshift-default-5fb66f6676-wpbr6"
Sep 11 23:22:28.984195 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:22:28.984176    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4575ea1-784b-4c29-9eff-a2d23d6772a8-istio-data\") pod \"e2e-mt-1ebfb31f-openshift-default-5fb66f6676-wpbr6\" (UID: \"f4575ea1-784b-4c29-9eff-a2d23d6772a8\") " pod="openshift-ingress/e2e-mt-1ebfb31f-openshift-default-5fb66f6676-wpbr6"
Sep 11 23:22:28.984315 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:22:28.984232    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4575ea1-784b-4c29-9eff-a2d23d6772a8-workload-socket\") pod \"e2e-mt-1ebfb31f-openshift-default-5fb66f6676-wpbr6\" (UID: \"f4575ea1-784b-4c29-9eff-a2d23d6772a8\") " pod="openshift-ingress/e2e-mt-1ebfb31f-openshift-default-5fb66f6676-wpbr6"
Sep 11 23:22:28.984315 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:22:28.984273    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4575ea1-784b-4c29-9eff-a2d23d6772a8-istiod-ca-cert\") pod \"e2e-mt-1ebfb31f-openshift-default-5fb66f6676-wpbr6\" (UID: \"f4575ea1-784b-4c29-9eff-a2d23d6772a8\") " pod="openshift-ingress/e2e-mt-1ebfb31f-openshift-default-5fb66f6676-wpbr6"
Sep 11 23:22:29.085318 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:22:29.085278    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4575ea1-784b-4c29-9eff-a2d23d6772a8-istio-podinfo\") pod \"e2e-mt-1ebfb31f-openshift-default-5fb66f6676-wpbr6\" (UID: \"f4575ea1-784b-4c29-9eff-a2d23d6772a8\") " pod="openshift-ingress/e2e-mt-1ebfb31f-openshift-default-5fb66f6676-wpbr6"
Sep 11 23:22:29.085318 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:22:29.085323    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh2ks\" (UniqueName: \"kubernetes.io/projected/f4575ea1-784b-4c29-9eff-a2d23d6772a8-kube-api-access-hh2ks\") pod \"e2e-mt-1ebfb31f-openshift-default-5fb66f6676-wpbr6\" (UID: \"f4575ea1-784b-4c29-9eff-a2d23d6772a8\") " pod="openshift-ingress/e2e-mt-1ebfb31f-openshift-default-5fb66f6676-wpbr6"
Sep 11 23:22:29.085526 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:22:29.085370    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4575ea1-784b-4c29-9eff-a2d23d6772a8-istio-token\") pod \"e2e-mt-1ebfb31f-openshift-default-5fb66f6676-wpbr6\" (UID: \"f4575ea1-784b-4c29-9eff-a2d23d6772a8\") " pod="openshift-ingress/e2e-mt-1ebfb31f-openshift-default-5fb66f6676-wpbr6"
Sep 11 23:22:29.085526 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:22:29.085410    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4575ea1-784b-4c29-9eff-a2d23d6772a8-istio-envoy\") pod \"e2e-mt-1ebfb31f-openshift-default-5fb66f6676-wpbr6\" (UID: \"f4575ea1-784b-4c29-9eff-a2d23d6772a8\") " pod="openshift-ingress/e2e-mt-1ebfb31f-openshift-default-5fb66f6676-wpbr6"
Sep 11 23:22:29.085526 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:22:29.085434    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4575ea1-784b-4c29-9eff-a2d23d6772a8-workload-certs\") pod \"e2e-mt-1ebfb31f-openshift-default-5fb66f6676-wpbr6\" (UID: \"f4575ea1-784b-4c29-9eff-a2d23d6772a8\") " pod="openshift-ingress/e2e-mt-1ebfb31f-openshift-default-5fb66f6676-wpbr6"
Sep 11 23:22:29.085526 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:22:29.085471    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4575ea1-784b-4c29-9eff-a2d23d6772a8-istio-data\") pod \"e2e-mt-1ebfb31f-openshift-default-5fb66f6676-wpbr6\" (UID: \"f4575ea1-784b-4c29-9eff-a2d23d6772a8\") " pod="openshift-ingress/e2e-mt-1ebfb31f-openshift-default-5fb66f6676-wpbr6"
Sep 11 23:22:29.085526 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:22:29.085510    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4575ea1-784b-4c29-9eff-a2d23d6772a8-workload-socket\") pod \"e2e-mt-1ebfb31f-openshift-default-5fb66f6676-wpbr6\" (UID: \"f4575ea1-784b-4c29-9eff-a2d23d6772a8\") " pod="openshift-ingress/e2e-mt-1ebfb31f-openshift-default-5fb66f6676-wpbr6"
Sep 11 23:22:29.085754 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:22:29.085541    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4575ea1-784b-4c29-9eff-a2d23d6772a8-istiod-ca-cert\") pod \"e2e-mt-1ebfb31f-openshift-default-5fb66f6676-wpbr6\" (UID: \"f4575ea1-784b-4c29-9eff-a2d23d6772a8\") " pod="openshift-ingress/e2e-mt-1ebfb31f-openshift-default-5fb66f6676-wpbr6"
Sep 11 23:22:29.085754 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:22:29.085606    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4575ea1-784b-4c29-9eff-a2d23d6772a8-credential-socket\") pod \"e2e-mt-1ebfb31f-openshift-default-5fb66f6676-wpbr6\" (UID: \"f4575ea1-784b-4c29-9eff-a2d23d6772a8\") " pod="openshift-ingress/e2e-mt-1ebfb31f-openshift-default-5fb66f6676-wpbr6"
Sep 11 23:22:29.085947 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:22:29.085903    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4575ea1-784b-4c29-9eff-a2d23d6772a8-workload-socket\") pod \"e2e-mt-1ebfb31f-openshift-default-5fb66f6676-wpbr6\" (UID: \"f4575ea1-784b-4c29-9eff-a2d23d6772a8\") " pod="openshift-ingress/e2e-mt-1ebfb31f-openshift-default-5fb66f6676-wpbr6"
Sep 11 23:22:29.086034 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:22:29.085951    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4575ea1-784b-4c29-9eff-a2d23d6772a8-workload-certs\") pod \"e2e-mt-1ebfb31f-openshift-default-5fb66f6676-wpbr6\" (UID: \"f4575ea1-784b-4c29-9eff-a2d23d6772a8\") " pod="openshift-ingress/e2e-mt-1ebfb31f-openshift-default-5fb66f6676-wpbr6"
Sep 11 23:22:29.086128 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:22:29.086106    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4575ea1-784b-4c29-9eff-a2d23d6772a8-credential-socket\") pod \"e2e-mt-1ebfb31f-openshift-default-5fb66f6676-wpbr6\" (UID: \"f4575ea1-784b-4c29-9eff-a2d23d6772a8\") " pod="openshift-ingress/e2e-mt-1ebfb31f-openshift-default-5fb66f6676-wpbr6"
Sep 11 23:22:29.086269 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:22:29.086110    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4575ea1-784b-4c29-9eff-a2d23d6772a8-istio-data\") pod \"e2e-mt-1ebfb31f-openshift-default-5fb66f6676-wpbr6\" (UID: \"f4575ea1-784b-4c29-9eff-a2d23d6772a8\") " pod="openshift-ingress/e2e-mt-1ebfb31f-openshift-default-5fb66f6676-wpbr6"
Sep 11 23:22:29.086737 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:22:29.086719    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4575ea1-784b-4c29-9eff-a2d23d6772a8-istiod-ca-cert\") pod \"e2e-mt-1ebfb31f-openshift-default-5fb66f6676-wpbr6\" (UID: \"f4575ea1-784b-4c29-9eff-a2d23d6772a8\") " pod="openshift-ingress/e2e-mt-1ebfb31f-openshift-default-5fb66f6676-wpbr6"
Sep 11 23:22:29.087802 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:22:29.087778    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4575ea1-784b-4c29-9eff-a2d23d6772a8-istio-envoy\") pod \"e2e-mt-1ebfb31f-openshift-default-5fb66f6676-wpbr6\" (UID: \"f4575ea1-784b-4c29-9eff-a2d23d6772a8\") " pod="openshift-ingress/e2e-mt-1ebfb31f-openshift-default-5fb66f6676-wpbr6"
Sep 11 23:22:29.088133 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:22:29.088113    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4575ea1-784b-4c29-9eff-a2d23d6772a8-istio-podinfo\") pod \"e2e-mt-1ebfb31f-openshift-default-5fb66f6676-wpbr6\" (UID: \"f4575ea1-784b-4c29-9eff-a2d23d6772a8\") " pod="openshift-ingress/e2e-mt-1ebfb31f-openshift-default-5fb66f6676-wpbr6"
Sep 11 23:22:29.094641 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:22:29.094624    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4575ea1-784b-4c29-9eff-a2d23d6772a8-istio-token\") pod \"e2e-mt-1ebfb31f-openshift-default-5fb66f6676-wpbr6\" (UID: \"f4575ea1-784b-4c29-9eff-a2d23d6772a8\") " pod="openshift-ingress/e2e-mt-1ebfb31f-openshift-default-5fb66f6676-wpbr6"
Sep 11 23:22:29.095018 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:22:29.094997    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh2ks\" (UniqueName: \"kubernetes.io/projected/f4575ea1-784b-4c29-9eff-a2d23d6772a8-kube-api-access-hh2ks\") pod \"e2e-mt-1ebfb31f-openshift-default-5fb66f6676-wpbr6\" (UID: \"f4575ea1-784b-4c29-9eff-a2d23d6772a8\") " pod="openshift-ingress/e2e-mt-1ebfb31f-openshift-default-5fb66f6676-wpbr6"
Sep 11 23:22:29.252269 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:22:29.252186    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1ebfb31f-openshift-default-5fb66f6676-wpbr6"
Sep 11 23:22:29.622870 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:22:29.622843    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1ebfb31f-openshift-default-5fb66f6676-wpbr6"]
Sep 11 23:22:29.625466 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:22:29.625437    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4575ea1_784b_4c29_9eff_a2d23d6772a8.slice/crio-0cec7bc653df8b71260852aa7dee6113acfb034e259d0971798959f75039716e WatchSource:0}: Error finding container 0cec7bc653df8b71260852aa7dee6113acfb034e259d0971798959f75039716e: Status 404 returned error can't find the container with id 0cec7bc653df8b71260852aa7dee6113acfb034e259d0971798959f75039716e
Sep 11 23:22:29.628053 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:22:29.628022    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:22:29.628185 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:22:29.628087    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:22:29.628185 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:22:29.628115    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:22:30.289798 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:22:30.289755    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1ebfb31f-openshift-default-5fb66f6676-wpbr6" event={"ID":"f4575ea1-784b-4c29-9eff-a2d23d6772a8","Type":"ContainerStarted","Data":"40769837750fd21fe8bc21eb8f05b933462d6005d9c642d805920c80c87493cb"}
Sep 11 23:22:30.290380 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:22:30.290354    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1ebfb31f-openshift-default-5fb66f6676-wpbr6" event={"ID":"f4575ea1-784b-4c29-9eff-a2d23d6772a8","Type":"ContainerStarted","Data":"0cec7bc653df8b71260852aa7dee6113acfb034e259d0971798959f75039716e"}
Sep 11 23:22:30.342166 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:22:30.342108    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-1ebfb31f-openshift-default-5fb66f6676-wpbr6" podStartSLOduration=2.342093442 podStartE2EDuration="2.342093442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 23:22:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:22:30.34181528 +0000 UTC m=+1274.374944497" watchObservedRunningTime="2026-09-11 23:22:30.342093442 +0000 UTC m=+1274.375222658"
Sep 11 23:22:31.252920 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:22:31.252885    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-1ebfb31f-openshift-default-5fb66f6676-wpbr6"
Sep 11 23:22:31.257962 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:22:31.257938    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-1ebfb31f-openshift-default-5fb66f6676-wpbr6"
Sep 11 23:22:31.296671 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:22:31.296639    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-1ebfb31f-openshift-default-5fb66f6676-wpbr6"
Sep 11 23:22:31.297627 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:22:31.297610    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-1ebfb31f-openshift-default-5fb66f6676-wpbr6"
Sep 11 23:22:39.504000 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:22:39.503965    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"]
Sep 11 23:23:06.530764 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:06.530729    2546 generic.go:356] "Generic (PLEG): container finished" podID="f4575ea1-784b-4c29-9eff-a2d23d6772a8" containerID="40769837750fd21fe8bc21eb8f05b933462d6005d9c642d805920c80c87493cb" exitCode=0
Sep 11 23:23:06.531305 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:06.530809    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1ebfb31f-openshift-default-5fb66f6676-wpbr6" event={"ID":"f4575ea1-784b-4c29-9eff-a2d23d6772a8","Type":"ContainerDied","Data":"40769837750fd21fe8bc21eb8f05b933462d6005d9c642d805920c80c87493cb"}
Sep 11 23:23:06.531374 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:06.531357    2546 scope.go:117] "RemoveContainer" containerID="40769837750fd21fe8bc21eb8f05b933462d6005d9c642d805920c80c87493cb"
Sep 11 23:23:06.532689 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:06.532656    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:23:06.532762 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:06.532746    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:23:06.532810 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:06.532795    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:23:07.545881 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:07.545840    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1ebfb31f-openshift-default-5fb66f6676-wpbr6" event={"ID":"f4575ea1-784b-4c29-9eff-a2d23d6772a8","Type":"ContainerStarted","Data":"1ccf869a9fabaeea5b29c4ec9e1d172fc96e0cae9490e7435550a7221eb6e754"}
Sep 11 23:23:08.253029 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:08.252978    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-1ebfb31f-openshift-default-5fb66f6676-wpbr6"
Sep 11 23:23:08.253258 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:08.253227    2546 patch_prober.go:28] interesting pod/e2e-mt-1ebfb31f-openshift-default-5fb66f6676-wpbr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Sep 11 23:23:08.253313 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:08.253280    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1ebfb31f-openshift-default-5fb66f6676-wpbr6" podUID="f4575ea1-784b-4c29-9eff-a2d23d6772a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Sep 11 23:23:09.252799 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:09.252755    2546 patch_prober.go:28] interesting pod/e2e-mt-1ebfb31f-openshift-default-5fb66f6676-wpbr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Sep 11 23:23:09.253261 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:09.252831    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1ebfb31f-openshift-default-5fb66f6676-wpbr6" podUID="f4575ea1-784b-4c29-9eff-a2d23d6772a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Sep 11 23:23:10.253258 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:10.253216    2546 patch_prober.go:28] interesting pod/e2e-mt-1ebfb31f-openshift-default-5fb66f6676-wpbr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Sep 11 23:23:10.253650 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:10.253294    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1ebfb31f-openshift-default-5fb66f6676-wpbr6" podUID="f4575ea1-784b-4c29-9eff-a2d23d6772a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Sep 11 23:23:10.547003 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:10.546899    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7719ed-2ms9p"]
Sep 11 23:23:10.551800 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:10.551768    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7719ed-2ms9p"
Sep 11 23:23:10.555272 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:10.555246    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 23:23:10.555463 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:10.555365    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vrx7m\""
Sep 11 23:23:10.557965 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:10.557942    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7719ed-2ms9p"]
Sep 11 23:23:10.638067 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:10.638012    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0e60b80-b8cf-4bf0-a05a-17ad420bf7b3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-7719ed-2ms9p\" (UID: \"b0e60b80-b8cf-4bf0-a05a-17ad420bf7b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7719ed-2ms9p"
Sep 11 23:23:10.738764 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:10.738728    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0e60b80-b8cf-4bf0-a05a-17ad420bf7b3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-7719ed-2ms9p\" (UID: \"b0e60b80-b8cf-4bf0-a05a-17ad420bf7b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7719ed-2ms9p"
Sep 11 23:23:10.739642 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:10.739613    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0e60b80-b8cf-4bf0-a05a-17ad420bf7b3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-7719ed-2ms9p\" (UID: \"b0e60b80-b8cf-4bf0-a05a-17ad420bf7b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7719ed-2ms9p"
Sep 11 23:23:10.866477 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:10.866392    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7719ed-2ms9p"
Sep 11 23:23:11.158056 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:11.158026    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7719ed-2ms9p"]
Sep 11 23:23:11.159779 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:23:11.159752    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0e60b80_b8cf_4bf0_a05a_17ad420bf7b3.slice/crio-a2c5674844129c55e6fb06ca4566f906cef6d2f778cfdfb4183a220ee31f33a8 WatchSource:0}: Error finding container a2c5674844129c55e6fb06ca4566f906cef6d2f778cfdfb4183a220ee31f33a8: Status 404 returned error can't find the container with id a2c5674844129c55e6fb06ca4566f906cef6d2f778cfdfb4183a220ee31f33a8
Sep 11 23:23:11.252865 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:11.252826    2546 patch_prober.go:28] interesting pod/e2e-mt-1ebfb31f-openshift-default-5fb66f6676-wpbr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Sep 11 23:23:11.253063 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:11.252899    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1ebfb31f-openshift-default-5fb66f6676-wpbr6" podUID="f4575ea1-784b-4c29-9eff-a2d23d6772a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Sep 11 23:23:11.578011 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:11.577974    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7719ed-2ms9p" event={"ID":"b0e60b80-b8cf-4bf0-a05a-17ad420bf7b3","Type":"ContainerStarted","Data":"0493115a5ed3eb17b5a773cf10f5a3769f598047a3310b783a67a4dc867aee03"}
Sep 11 23:23:11.578426 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:11.578035    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7719ed-2ms9p" event={"ID":"b0e60b80-b8cf-4bf0-a05a-17ad420bf7b3","Type":"ContainerStarted","Data":"a2c5674844129c55e6fb06ca4566f906cef6d2f778cfdfb4183a220ee31f33a8"}
Sep 11 23:23:11.595072 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:11.595024    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7719ed-2ms9p" podStartSLOduration=1.5950078429999999 podStartE2EDuration="1.595007843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 23:23:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:23:11.593584556 +0000 UTC m=+1315.626713776" watchObservedRunningTime="2026-09-11 23:23:11.595007843 +0000 UTC m=+1315.628137058"
Sep 11 23:23:12.253627 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:12.253579    2546 patch_prober.go:28] interesting pod/e2e-mt-1ebfb31f-openshift-default-5fb66f6676-wpbr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Sep 11 23:23:12.253815 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:12.253656    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1ebfb31f-openshift-default-5fb66f6676-wpbr6" podUID="f4575ea1-784b-4c29-9eff-a2d23d6772a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Sep 11 23:23:12.587782 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:12.587740    2546 generic.go:356] "Generic (PLEG): container finished" podID="b0e60b80-b8cf-4bf0-a05a-17ad420bf7b3" containerID="0493115a5ed3eb17b5a773cf10f5a3769f598047a3310b783a67a4dc867aee03" exitCode=0
Sep 11 23:23:12.588258 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:12.587833    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7719ed-2ms9p" event={"ID":"b0e60b80-b8cf-4bf0-a05a-17ad420bf7b3","Type":"ContainerDied","Data":"0493115a5ed3eb17b5a773cf10f5a3769f598047a3310b783a67a4dc867aee03"}
Sep 11 23:23:13.257062 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:13.257030    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-1ebfb31f-openshift-default-5fb66f6676-wpbr6"
Sep 11 23:23:13.257428 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:13.257410    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-1ebfb31f-openshift-default-5fb66f6676-wpbr6"
Sep 11 23:23:13.258218 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:13.258193    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-1ebfb31f-openshift-default-5fb66f6676-wpbr6"
Sep 11 23:23:13.743196 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:13.743171    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7719ed-2ms9p"
Sep 11 23:23:13.879096 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:13.879003    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0e60b80-b8cf-4bf0-a05a-17ad420bf7b3-maas-api-service-ca\") pod \"b0e60b80-b8cf-4bf0-a05a-17ad420bf7b3\" (UID: \"b0e60b80-b8cf-4bf0-a05a-17ad420bf7b3\") "
Sep 11 23:23:13.879418 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:13.879397    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0e60b80-b8cf-4bf0-a05a-17ad420bf7b3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b0e60b80-b8cf-4bf0-a05a-17ad420bf7b3" (UID: "b0e60b80-b8cf-4bf0-a05a-17ad420bf7b3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 23:23:13.980207 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:13.980168    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0e60b80-b8cf-4bf0-a05a-17ad420bf7b3-maas-api-service-ca\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:23:14.603375 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:14.603348    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7719ed-2ms9p"
Sep 11 23:23:14.603537 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:14.603387    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7719ed-2ms9p" event={"ID":"b0e60b80-b8cf-4bf0-a05a-17ad420bf7b3","Type":"ContainerDied","Data":"a2c5674844129c55e6fb06ca4566f906cef6d2f778cfdfb4183a220ee31f33a8"}
Sep 11 23:23:14.603537 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:14.603435    2546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a2c5674844129c55e6fb06ca4566f906cef6d2f778cfdfb4183a220ee31f33a8"
Sep 11 23:23:19.604739 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:19.604702    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"]
Sep 11 23:23:21.606650 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:21.606608    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7719ed-2ms9p"]
Sep 11 23:23:21.609311 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:21.609288    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7719ed-2ms9p"]
Sep 11 23:23:22.495037 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:22.494998    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0e60b80-b8cf-4bf0-a05a-17ad420bf7b3" path="/var/lib/kubelet/pods/b0e60b80-b8cf-4bf0-a05a-17ad420bf7b3/volumes"
Sep 11 23:23:31.710897 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:31.710863    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1ebfb31f-openshift-default-5fb66f6676-wpbr6"]
Sep 11 23:23:31.711344 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:31.711121    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-1ebfb31f-openshift-default-5fb66f6676-wpbr6" podUID="f4575ea1-784b-4c29-9eff-a2d23d6772a8" containerName="istio-proxy" containerID="cri-o://1ccf869a9fabaeea5b29c4ec9e1d172fc96e0cae9490e7435550a7221eb6e754" gracePeriod=30
Sep 11 23:23:33.928715 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:33.928673    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-7719ed-openshift-default-7cf684688d-bjzms"]
Sep 11 23:23:33.929229 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:33.928979    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-7719ed-openshift-default-7cf684688d-bjzms" podUID="0511d07e-cb77-4534-bc61-48f06dfd3bf3" containerName="istio-proxy" containerID="cri-o://5011aeb3872e42b718d85a3664e15fda79e9576ae31647d6431ea43ae00b97b2" gracePeriod=30
Sep 11 23:23:36.975364 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:36.975306    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1ebfb31f-openshift-default-5fb66f6676-wpbr6"
Sep 11 23:23:37.027626 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:37.027593    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4575ea1-784b-4c29-9eff-a2d23d6772a8-istio-data\") pod \"f4575ea1-784b-4c29-9eff-a2d23d6772a8\" (UID: \"f4575ea1-784b-4c29-9eff-a2d23d6772a8\") "
Sep 11 23:23:37.027815 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:37.027674    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4575ea1-784b-4c29-9eff-a2d23d6772a8-istio-token\") pod \"f4575ea1-784b-4c29-9eff-a2d23d6772a8\" (UID: \"f4575ea1-784b-4c29-9eff-a2d23d6772a8\") "
Sep 11 23:23:37.027815 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:37.027735    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4575ea1-784b-4c29-9eff-a2d23d6772a8-credential-socket\") pod \"f4575ea1-784b-4c29-9eff-a2d23d6772a8\" (UID: \"f4575ea1-784b-4c29-9eff-a2d23d6772a8\") "
Sep 11 23:23:37.027815 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:37.027753    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hh2ks\" (UniqueName: \"kubernetes.io/projected/f4575ea1-784b-4c29-9eff-a2d23d6772a8-kube-api-access-hh2ks\") pod \"f4575ea1-784b-4c29-9eff-a2d23d6772a8\" (UID: \"f4575ea1-784b-4c29-9eff-a2d23d6772a8\") "
Sep 11 23:23:37.027815 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:37.027798    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4575ea1-784b-4c29-9eff-a2d23d6772a8-istio-envoy\") pod \"f4575ea1-784b-4c29-9eff-a2d23d6772a8\" (UID: \"f4575ea1-784b-4c29-9eff-a2d23d6772a8\") "
Sep 11 23:23:37.028050 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:37.027830    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4575ea1-784b-4c29-9eff-a2d23d6772a8-istio-podinfo\") pod \"f4575ea1-784b-4c29-9eff-a2d23d6772a8\" (UID: \"f4575ea1-784b-4c29-9eff-a2d23d6772a8\") "
Sep 11 23:23:37.028050 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:37.027860    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4575ea1-784b-4c29-9eff-a2d23d6772a8-workload-socket\") pod \"f4575ea1-784b-4c29-9eff-a2d23d6772a8\" (UID: \"f4575ea1-784b-4c29-9eff-a2d23d6772a8\") "
Sep 11 23:23:37.028050 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:37.027907    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4575ea1-784b-4c29-9eff-a2d23d6772a8-istiod-ca-cert\") pod \"f4575ea1-784b-4c29-9eff-a2d23d6772a8\" (UID: \"f4575ea1-784b-4c29-9eff-a2d23d6772a8\") "
Sep 11 23:23:37.028050 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:37.027914    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4575ea1-784b-4c29-9eff-a2d23d6772a8-istio-data" (OuterVolumeSpecName: "istio-data") pod "f4575ea1-784b-4c29-9eff-a2d23d6772a8" (UID: "f4575ea1-784b-4c29-9eff-a2d23d6772a8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:23:37.028050 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:37.027936    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4575ea1-784b-4c29-9eff-a2d23d6772a8-workload-certs\") pod \"f4575ea1-784b-4c29-9eff-a2d23d6772a8\" (UID: \"f4575ea1-784b-4c29-9eff-a2d23d6772a8\") "
Sep 11 23:23:37.028366 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:37.028327    2546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4575ea1-784b-4c29-9eff-a2d23d6772a8-istio-data\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:23:37.028506 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:37.028441    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4575ea1-784b-4c29-9eff-a2d23d6772a8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f4575ea1-784b-4c29-9eff-a2d23d6772a8" (UID: "f4575ea1-784b-4c29-9eff-a2d23d6772a8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:23:37.028506 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:37.028497    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4575ea1-784b-4c29-9eff-a2d23d6772a8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f4575ea1-784b-4c29-9eff-a2d23d6772a8" (UID: "f4575ea1-784b-4c29-9eff-a2d23d6772a8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:23:37.028628 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:37.028563    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4575ea1-784b-4c29-9eff-a2d23d6772a8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f4575ea1-784b-4c29-9eff-a2d23d6772a8" (UID: "f4575ea1-784b-4c29-9eff-a2d23d6772a8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:23:37.028805 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:37.028781    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4575ea1-784b-4c29-9eff-a2d23d6772a8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f4575ea1-784b-4c29-9eff-a2d23d6772a8" (UID: "f4575ea1-784b-4c29-9eff-a2d23d6772a8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 23:23:37.030573 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:37.030527    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4575ea1-784b-4c29-9eff-a2d23d6772a8-kube-api-access-hh2ks" (OuterVolumeSpecName: "kube-api-access-hh2ks") pod "f4575ea1-784b-4c29-9eff-a2d23d6772a8" (UID: "f4575ea1-784b-4c29-9eff-a2d23d6772a8"). InnerVolumeSpecName "kube-api-access-hh2ks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 23:23:37.030814 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:37.030777    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4575ea1-784b-4c29-9eff-a2d23d6772a8-istio-token" (OuterVolumeSpecName: "istio-token") pod "f4575ea1-784b-4c29-9eff-a2d23d6772a8" (UID: "f4575ea1-784b-4c29-9eff-a2d23d6772a8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 23:23:37.030924 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:37.030822    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f4575ea1-784b-4c29-9eff-a2d23d6772a8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f4575ea1-784b-4c29-9eff-a2d23d6772a8" (UID: "f4575ea1-784b-4c29-9eff-a2d23d6772a8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 23:23:37.031391 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:37.031357    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4575ea1-784b-4c29-9eff-a2d23d6772a8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f4575ea1-784b-4c29-9eff-a2d23d6772a8" (UID: "f4575ea1-784b-4c29-9eff-a2d23d6772a8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:23:37.130026 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:37.129987    2546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4575ea1-784b-4c29-9eff-a2d23d6772a8-istiod-ca-cert\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:23:37.130209 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:37.130046    2546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4575ea1-784b-4c29-9eff-a2d23d6772a8-workload-certs\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:23:37.130209 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:37.130061    2546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4575ea1-784b-4c29-9eff-a2d23d6772a8-istio-token\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:23:37.130209 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:37.130073    2546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4575ea1-784b-4c29-9eff-a2d23d6772a8-credential-socket\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:23:37.130209 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:37.130087    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hh2ks\" (UniqueName: \"kubernetes.io/projected/f4575ea1-784b-4c29-9eff-a2d23d6772a8-kube-api-access-hh2ks\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:23:37.130209 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:37.130101    2546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4575ea1-784b-4c29-9eff-a2d23d6772a8-istio-envoy\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:23:37.130209 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:37.130115    2546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4575ea1-784b-4c29-9eff-a2d23d6772a8-istio-podinfo\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:23:37.130209 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:37.130127    2546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4575ea1-784b-4c29-9eff-a2d23d6772a8-workload-socket\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:23:37.765102 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:37.765062    2546 generic.go:356] "Generic (PLEG): container finished" podID="f4575ea1-784b-4c29-9eff-a2d23d6772a8" containerID="1ccf869a9fabaeea5b29c4ec9e1d172fc96e0cae9490e7435550a7221eb6e754" exitCode=0
Sep 11 23:23:37.765321 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:37.765112    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1ebfb31f-openshift-default-5fb66f6676-wpbr6"
Sep 11 23:23:37.765321 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:37.765149    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1ebfb31f-openshift-default-5fb66f6676-wpbr6" event={"ID":"f4575ea1-784b-4c29-9eff-a2d23d6772a8","Type":"ContainerDied","Data":"1ccf869a9fabaeea5b29c4ec9e1d172fc96e0cae9490e7435550a7221eb6e754"}
Sep 11 23:23:37.765321 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:37.765216    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1ebfb31f-openshift-default-5fb66f6676-wpbr6" event={"ID":"f4575ea1-784b-4c29-9eff-a2d23d6772a8","Type":"ContainerDied","Data":"0cec7bc653df8b71260852aa7dee6113acfb034e259d0971798959f75039716e"}
Sep 11 23:23:37.765321 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:37.765239    2546 scope.go:117] "RemoveContainer" containerID="1ccf869a9fabaeea5b29c4ec9e1d172fc96e0cae9490e7435550a7221eb6e754"
Sep 11 23:23:37.775822 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:37.775804    2546 scope.go:117] "RemoveContainer" containerID="40769837750fd21fe8bc21eb8f05b933462d6005d9c642d805920c80c87493cb"
Sep 11 23:23:37.785333 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:37.785311    2546 scope.go:117] "RemoveContainer" containerID="1ccf869a9fabaeea5b29c4ec9e1d172fc96e0cae9490e7435550a7221eb6e754"
Sep 11 23:23:37.785605 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:23:37.785585    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ccf869a9fabaeea5b29c4ec9e1d172fc96e0cae9490e7435550a7221eb6e754\": container with ID starting with 1ccf869a9fabaeea5b29c4ec9e1d172fc96e0cae9490e7435550a7221eb6e754 not found: ID does not exist" containerID="1ccf869a9fabaeea5b29c4ec9e1d172fc96e0cae9490e7435550a7221eb6e754"
Sep 11 23:23:37.785680 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:37.785612    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ccf869a9fabaeea5b29c4ec9e1d172fc96e0cae9490e7435550a7221eb6e754"} err="failed to get container status \"1ccf869a9fabaeea5b29c4ec9e1d172fc96e0cae9490e7435550a7221eb6e754\": rpc error: code = NotFound desc = could not find container \"1ccf869a9fabaeea5b29c4ec9e1d172fc96e0cae9490e7435550a7221eb6e754\": container with ID starting with 1ccf869a9fabaeea5b29c4ec9e1d172fc96e0cae9490e7435550a7221eb6e754 not found: ID does not exist"
Sep 11 23:23:37.785680 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:37.785629    2546 scope.go:117] "RemoveContainer" containerID="40769837750fd21fe8bc21eb8f05b933462d6005d9c642d805920c80c87493cb"
Sep 11 23:23:37.785939 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:23:37.785919    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40769837750fd21fe8bc21eb8f05b933462d6005d9c642d805920c80c87493cb\": container with ID starting with 40769837750fd21fe8bc21eb8f05b933462d6005d9c642d805920c80c87493cb not found: ID does not exist" containerID="40769837750fd21fe8bc21eb8f05b933462d6005d9c642d805920c80c87493cb"
Sep 11 23:23:37.786008 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:37.785945    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40769837750fd21fe8bc21eb8f05b933462d6005d9c642d805920c80c87493cb"} err="failed to get container status \"40769837750fd21fe8bc21eb8f05b933462d6005d9c642d805920c80c87493cb\": rpc error: code = NotFound desc = could not find container \"40769837750fd21fe8bc21eb8f05b933462d6005d9c642d805920c80c87493cb\": container with ID starting with 40769837750fd21fe8bc21eb8f05b933462d6005d9c642d805920c80c87493cb not found: ID does not exist"
Sep 11 23:23:37.789696 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:37.789143    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1ebfb31f-openshift-default-5fb66f6676-wpbr6"]
Sep 11 23:23:37.792870 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:37.792847    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-1ebfb31f-openshift-default-5fb66f6676-wpbr6"]
Sep 11 23:23:38.495516 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:38.495485    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4575ea1-784b-4c29-9eff-a2d23d6772a8" path="/var/lib/kubelet/pods/f4575ea1-784b-4c29-9eff-a2d23d6772a8/volumes"
Sep 11 23:23:39.187979 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:39.187954    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-7719ed-openshift-default-7cf684688d-bjzms"
Sep 11 23:23:39.252548 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:39.252453    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k7sp7\" (UniqueName: \"kubernetes.io/projected/0511d07e-cb77-4534-bc61-48f06dfd3bf3-kube-api-access-k7sp7\") pod \"0511d07e-cb77-4534-bc61-48f06dfd3bf3\" (UID: \"0511d07e-cb77-4534-bc61-48f06dfd3bf3\") "
Sep 11 23:23:39.252548 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:39.252510    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0511d07e-cb77-4534-bc61-48f06dfd3bf3-credential-socket\") pod \"0511d07e-cb77-4534-bc61-48f06dfd3bf3\" (UID: \"0511d07e-cb77-4534-bc61-48f06dfd3bf3\") "
Sep 11 23:23:39.252548 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:39.252539    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0511d07e-cb77-4534-bc61-48f06dfd3bf3-workload-certs\") pod \"0511d07e-cb77-4534-bc61-48f06dfd3bf3\" (UID: \"0511d07e-cb77-4534-bc61-48f06dfd3bf3\") "
Sep 11 23:23:39.252818 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:39.252607    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0511d07e-cb77-4534-bc61-48f06dfd3bf3-istio-data\") pod \"0511d07e-cb77-4534-bc61-48f06dfd3bf3\" (UID: \"0511d07e-cb77-4534-bc61-48f06dfd3bf3\") "
Sep 11 23:23:39.252818 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:39.252630    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0511d07e-cb77-4534-bc61-48f06dfd3bf3-istio-podinfo\") pod \"0511d07e-cb77-4534-bc61-48f06dfd3bf3\" (UID: \"0511d07e-cb77-4534-bc61-48f06dfd3bf3\") "
Sep 11 23:23:39.252818 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:39.252708    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0511d07e-cb77-4534-bc61-48f06dfd3bf3-workload-socket\") pod \"0511d07e-cb77-4534-bc61-48f06dfd3bf3\" (UID: \"0511d07e-cb77-4534-bc61-48f06dfd3bf3\") "
Sep 11 23:23:39.252818 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:39.252741    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0511d07e-cb77-4534-bc61-48f06dfd3bf3-istiod-ca-cert\") pod \"0511d07e-cb77-4534-bc61-48f06dfd3bf3\" (UID: \"0511d07e-cb77-4534-bc61-48f06dfd3bf3\") "
Sep 11 23:23:39.252818 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:39.252805    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0511d07e-cb77-4534-bc61-48f06dfd3bf3-istio-envoy\") pod \"0511d07e-cb77-4534-bc61-48f06dfd3bf3\" (UID: \"0511d07e-cb77-4534-bc61-48f06dfd3bf3\") "
Sep 11 23:23:39.253060 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:39.252852    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0511d07e-cb77-4534-bc61-48f06dfd3bf3-istio-token\") pod \"0511d07e-cb77-4534-bc61-48f06dfd3bf3\" (UID: \"0511d07e-cb77-4534-bc61-48f06dfd3bf3\") "
Sep 11 23:23:39.253060 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:39.252800    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0511d07e-cb77-4534-bc61-48f06dfd3bf3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0511d07e-cb77-4534-bc61-48f06dfd3bf3" (UID: "0511d07e-cb77-4534-bc61-48f06dfd3bf3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:23:39.253060 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:39.252844    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0511d07e-cb77-4534-bc61-48f06dfd3bf3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0511d07e-cb77-4534-bc61-48f06dfd3bf3" (UID: "0511d07e-cb77-4534-bc61-48f06dfd3bf3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:23:39.253060 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:39.252875    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0511d07e-cb77-4534-bc61-48f06dfd3bf3-istio-data" (OuterVolumeSpecName: "istio-data") pod "0511d07e-cb77-4534-bc61-48f06dfd3bf3" (UID: "0511d07e-cb77-4534-bc61-48f06dfd3bf3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:23:39.253060 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:39.253018    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0511d07e-cb77-4534-bc61-48f06dfd3bf3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0511d07e-cb77-4534-bc61-48f06dfd3bf3" (UID: "0511d07e-cb77-4534-bc61-48f06dfd3bf3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:23:39.253357 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:39.253090    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0511d07e-cb77-4534-bc61-48f06dfd3bf3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0511d07e-cb77-4534-bc61-48f06dfd3bf3" (UID: "0511d07e-cb77-4534-bc61-48f06dfd3bf3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 23:23:39.253493 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:39.253432    2546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0511d07e-cb77-4534-bc61-48f06dfd3bf3-istio-data\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:23:39.253493 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:39.253454    2546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0511d07e-cb77-4534-bc61-48f06dfd3bf3-workload-socket\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:23:39.253493 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:39.253468    2546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0511d07e-cb77-4534-bc61-48f06dfd3bf3-istiod-ca-cert\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:23:39.253493 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:39.253483    2546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0511d07e-cb77-4534-bc61-48f06dfd3bf3-credential-socket\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:23:39.253493 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:39.253497    2546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0511d07e-cb77-4534-bc61-48f06dfd3bf3-workload-certs\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:23:39.254885 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:39.254859    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0511d07e-cb77-4534-bc61-48f06dfd3bf3-kube-api-access-k7sp7" (OuterVolumeSpecName: "kube-api-access-k7sp7") pod "0511d07e-cb77-4534-bc61-48f06dfd3bf3" (UID: "0511d07e-cb77-4534-bc61-48f06dfd3bf3"). InnerVolumeSpecName "kube-api-access-k7sp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 23:23:39.255038 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:39.255019    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0511d07e-cb77-4534-bc61-48f06dfd3bf3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0511d07e-cb77-4534-bc61-48f06dfd3bf3" (UID: "0511d07e-cb77-4534-bc61-48f06dfd3bf3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 23:23:39.255038 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:39.255023    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0511d07e-cb77-4534-bc61-48f06dfd3bf3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0511d07e-cb77-4534-bc61-48f06dfd3bf3" (UID: "0511d07e-cb77-4534-bc61-48f06dfd3bf3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:23:39.255126 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:39.255039    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0511d07e-cb77-4534-bc61-48f06dfd3bf3-istio-token" (OuterVolumeSpecName: "istio-token") pod "0511d07e-cb77-4534-bc61-48f06dfd3bf3" (UID: "0511d07e-cb77-4534-bc61-48f06dfd3bf3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 23:23:39.354611 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:39.354569    2546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0511d07e-cb77-4534-bc61-48f06dfd3bf3-istio-envoy\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:23:39.354611 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:39.354604    2546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0511d07e-cb77-4534-bc61-48f06dfd3bf3-istio-token\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:23:39.354611 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:39.354614    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7sp7\" (UniqueName: \"kubernetes.io/projected/0511d07e-cb77-4534-bc61-48f06dfd3bf3-kube-api-access-k7sp7\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:23:39.354843 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:39.354624    2546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0511d07e-cb77-4534-bc61-48f06dfd3bf3-istio-podinfo\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:23:39.660608 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:39.660574    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-c0d4bf-openshift-default-948f987d-4kbcq"]
Sep 11 23:23:39.661089 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:39.661050    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0511d07e-cb77-4534-bc61-48f06dfd3bf3" containerName="istio-proxy"
Sep 11 23:23:39.661089 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:39.661064    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="0511d07e-cb77-4534-bc61-48f06dfd3bf3" containerName="istio-proxy"
Sep 11 23:23:39.661089 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:39.661071    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0e60b80-b8cf-4bf0-a05a-17ad420bf7b3" containerName="revoke-keys"
Sep 11 23:23:39.661089 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:39.661077    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0e60b80-b8cf-4bf0-a05a-17ad420bf7b3" containerName="revoke-keys"
Sep 11 23:23:39.661089 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:39.661089    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4575ea1-784b-4c29-9eff-a2d23d6772a8" containerName="istio-proxy"
Sep 11 23:23:39.661089 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:39.661095    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4575ea1-784b-4c29-9eff-a2d23d6772a8" containerName="istio-proxy"
Sep 11 23:23:39.661423 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:39.661108    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4575ea1-784b-4c29-9eff-a2d23d6772a8" containerName="istio-proxy"
Sep 11 23:23:39.661423 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:39.661113    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4575ea1-784b-4c29-9eff-a2d23d6772a8" containerName="istio-proxy"
Sep 11 23:23:39.661423 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:39.661234    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4575ea1-784b-4c29-9eff-a2d23d6772a8" containerName="istio-proxy"
Sep 11 23:23:39.661423 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:39.661244    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="0511d07e-cb77-4534-bc61-48f06dfd3bf3" containerName="istio-proxy"
Sep 11 23:23:39.661423 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:39.661253    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0e60b80-b8cf-4bf0-a05a-17ad420bf7b3" containerName="revoke-keys"
Sep 11 23:23:39.664828 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:39.664809    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-c0d4bf-openshift-default-948f987d-4kbcq"
Sep 11 23:23:39.668437 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:39.668417    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-c0d4bf-openshift-default-dockercfg-sr658\""
Sep 11 23:23:39.680320 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:39.680300    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-c0d4bf-openshift-default-948f987d-4kbcq"]
Sep 11 23:23:39.762176 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:39.761688    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f621a2e1-7039-44a2-aa9a-a66e3e5e4e25-workload-socket\") pod \"e2e-noten-c0d4bf-openshift-default-948f987d-4kbcq\" (UID: \"f621a2e1-7039-44a2-aa9a-a66e3e5e4e25\") " pod="openshift-ingress/e2e-noten-c0d4bf-openshift-default-948f987d-4kbcq"
Sep 11 23:23:39.762176 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:39.761748    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f621a2e1-7039-44a2-aa9a-a66e3e5e4e25-workload-certs\") pod \"e2e-noten-c0d4bf-openshift-default-948f987d-4kbcq\" (UID: \"f621a2e1-7039-44a2-aa9a-a66e3e5e4e25\") " pod="openshift-ingress/e2e-noten-c0d4bf-openshift-default-948f987d-4kbcq"
Sep 11 23:23:39.762176 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:39.761805    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f621a2e1-7039-44a2-aa9a-a66e3e5e4e25-istiod-ca-cert\") pod \"e2e-noten-c0d4bf-openshift-default-948f987d-4kbcq\" (UID: \"f621a2e1-7039-44a2-aa9a-a66e3e5e4e25\") " pod="openshift-ingress/e2e-noten-c0d4bf-openshift-default-948f987d-4kbcq"
Sep 11 23:23:39.762176 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:39.761840    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f621a2e1-7039-44a2-aa9a-a66e3e5e4e25-credential-socket\") pod \"e2e-noten-c0d4bf-openshift-default-948f987d-4kbcq\" (UID: \"f621a2e1-7039-44a2-aa9a-a66e3e5e4e25\") " pod="openshift-ingress/e2e-noten-c0d4bf-openshift-default-948f987d-4kbcq"
Sep 11 23:23:39.762176 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:39.761863    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f621a2e1-7039-44a2-aa9a-a66e3e5e4e25-istio-podinfo\") pod \"e2e-noten-c0d4bf-openshift-default-948f987d-4kbcq\" (UID: \"f621a2e1-7039-44a2-aa9a-a66e3e5e4e25\") " pod="openshift-ingress/e2e-noten-c0d4bf-openshift-default-948f987d-4kbcq"
Sep 11 23:23:39.762176 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:39.761906    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f621a2e1-7039-44a2-aa9a-a66e3e5e4e25-istio-envoy\") pod \"e2e-noten-c0d4bf-openshift-default-948f987d-4kbcq\" (UID: \"f621a2e1-7039-44a2-aa9a-a66e3e5e4e25\") " pod="openshift-ingress/e2e-noten-c0d4bf-openshift-default-948f987d-4kbcq"
Sep 11 23:23:39.762176 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:39.761942    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f621a2e1-7039-44a2-aa9a-a66e3e5e4e25-istio-data\") pod \"e2e-noten-c0d4bf-openshift-default-948f987d-4kbcq\" (UID: \"f621a2e1-7039-44a2-aa9a-a66e3e5e4e25\") " pod="openshift-ingress/e2e-noten-c0d4bf-openshift-default-948f987d-4kbcq"
Sep 11 23:23:39.762176 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:39.762015    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f621a2e1-7039-44a2-aa9a-a66e3e5e4e25-istio-token\") pod \"e2e-noten-c0d4bf-openshift-default-948f987d-4kbcq\" (UID: \"f621a2e1-7039-44a2-aa9a-a66e3e5e4e25\") " pod="openshift-ingress/e2e-noten-c0d4bf-openshift-default-948f987d-4kbcq"
Sep 11 23:23:39.762176 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:39.762040    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cskkd\" (UniqueName: \"kubernetes.io/projected/f621a2e1-7039-44a2-aa9a-a66e3e5e4e25-kube-api-access-cskkd\") pod \"e2e-noten-c0d4bf-openshift-default-948f987d-4kbcq\" (UID: \"f621a2e1-7039-44a2-aa9a-a66e3e5e4e25\") " pod="openshift-ingress/e2e-noten-c0d4bf-openshift-default-948f987d-4kbcq"
Sep 11 23:23:39.788128 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:39.788087    2546 generic.go:356] "Generic (PLEG): container finished" podID="0511d07e-cb77-4534-bc61-48f06dfd3bf3" containerID="5011aeb3872e42b718d85a3664e15fda79e9576ae31647d6431ea43ae00b97b2" exitCode=0
Sep 11 23:23:39.788371 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:39.788284    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-7719ed-openshift-default-7cf684688d-bjzms" event={"ID":"0511d07e-cb77-4534-bc61-48f06dfd3bf3","Type":"ContainerDied","Data":"5011aeb3872e42b718d85a3664e15fda79e9576ae31647d6431ea43ae00b97b2"}
Sep 11 23:23:39.788371 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:39.788338    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-7719ed-openshift-default-7cf684688d-bjzms" event={"ID":"0511d07e-cb77-4534-bc61-48f06dfd3bf3","Type":"ContainerDied","Data":"bca92236df4b5fe25d2729801d505d2643a00bc404fae1ec003313ea405cc206"}
Sep 11 23:23:39.788371 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:39.788368    2546 scope.go:117] "RemoveContainer" containerID="5011aeb3872e42b718d85a3664e15fda79e9576ae31647d6431ea43ae00b97b2"
Sep 11 23:23:39.788562 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:39.788291    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-7719ed-openshift-default-7cf684688d-bjzms"
Sep 11 23:23:39.811047 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:39.810927    2546 scope.go:117] "RemoveContainer" containerID="61783ba24aeb6addc8bddbcdeca4e8e7766b1e530d4ba7b43a29ce538d685630"
Sep 11 23:23:39.821988 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:39.821959    2546 scope.go:117] "RemoveContainer" containerID="5011aeb3872e42b718d85a3664e15fda79e9576ae31647d6431ea43ae00b97b2"
Sep 11 23:23:39.822318 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:23:39.822298    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5011aeb3872e42b718d85a3664e15fda79e9576ae31647d6431ea43ae00b97b2\": container with ID starting with 5011aeb3872e42b718d85a3664e15fda79e9576ae31647d6431ea43ae00b97b2 not found: ID does not exist" containerID="5011aeb3872e42b718d85a3664e15fda79e9576ae31647d6431ea43ae00b97b2"
Sep 11 23:23:39.822399 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:39.822328    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5011aeb3872e42b718d85a3664e15fda79e9576ae31647d6431ea43ae00b97b2"} err="failed to get container status \"5011aeb3872e42b718d85a3664e15fda79e9576ae31647d6431ea43ae00b97b2\": rpc error: code = NotFound desc = could not find container \"5011aeb3872e42b718d85a3664e15fda79e9576ae31647d6431ea43ae00b97b2\": container with ID starting with 5011aeb3872e42b718d85a3664e15fda79e9576ae31647d6431ea43ae00b97b2 not found: ID does not exist"
Sep 11 23:23:39.822399 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:39.822349    2546 scope.go:117] "RemoveContainer" containerID="61783ba24aeb6addc8bddbcdeca4e8e7766b1e530d4ba7b43a29ce538d685630"
Sep 11 23:23:39.822714 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:23:39.822667    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61783ba24aeb6addc8bddbcdeca4e8e7766b1e530d4ba7b43a29ce538d685630\": container with ID starting with 61783ba24aeb6addc8bddbcdeca4e8e7766b1e530d4ba7b43a29ce538d685630 not found: ID does not exist" containerID="61783ba24aeb6addc8bddbcdeca4e8e7766b1e530d4ba7b43a29ce538d685630"
Sep 11 23:23:39.822714 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:39.822704    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61783ba24aeb6addc8bddbcdeca4e8e7766b1e530d4ba7b43a29ce538d685630"} err="failed to get container status \"61783ba24aeb6addc8bddbcdeca4e8e7766b1e530d4ba7b43a29ce538d685630\": rpc error: code = NotFound desc = could not find container \"61783ba24aeb6addc8bddbcdeca4e8e7766b1e530d4ba7b43a29ce538d685630\": container with ID starting with 61783ba24aeb6addc8bddbcdeca4e8e7766b1e530d4ba7b43a29ce538d685630 not found: ID does not exist"
Sep 11 23:23:39.829264 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:39.829244    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-7719ed-openshift-default-7cf684688d-bjzms"]
Sep 11 23:23:39.833587 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:39.833568    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-7719ed-openshift-default-7cf684688d-bjzms"]
Sep 11 23:23:39.862895 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:39.862873    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f621a2e1-7039-44a2-aa9a-a66e3e5e4e25-istiod-ca-cert\") pod \"e2e-noten-c0d4bf-openshift-default-948f987d-4kbcq\" (UID: \"f621a2e1-7039-44a2-aa9a-a66e3e5e4e25\") " pod="openshift-ingress/e2e-noten-c0d4bf-openshift-default-948f987d-4kbcq"
Sep 11 23:23:39.863009 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:39.862908    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f621a2e1-7039-44a2-aa9a-a66e3e5e4e25-credential-socket\") pod \"e2e-noten-c0d4bf-openshift-default-948f987d-4kbcq\" (UID: \"f621a2e1-7039-44a2-aa9a-a66e3e5e4e25\") " pod="openshift-ingress/e2e-noten-c0d4bf-openshift-default-948f987d-4kbcq"
Sep 11 23:23:39.863009 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:39.862926    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f621a2e1-7039-44a2-aa9a-a66e3e5e4e25-istio-podinfo\") pod \"e2e-noten-c0d4bf-openshift-default-948f987d-4kbcq\" (UID: \"f621a2e1-7039-44a2-aa9a-a66e3e5e4e25\") " pod="openshift-ingress/e2e-noten-c0d4bf-openshift-default-948f987d-4kbcq"
Sep 11 23:23:39.863009 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:39.862954    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f621a2e1-7039-44a2-aa9a-a66e3e5e4e25-istio-envoy\") pod \"e2e-noten-c0d4bf-openshift-default-948f987d-4kbcq\" (UID: \"f621a2e1-7039-44a2-aa9a-a66e3e5e4e25\") " pod="openshift-ingress/e2e-noten-c0d4bf-openshift-default-948f987d-4kbcq"
Sep 11 23:23:39.863125 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:39.863087    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f621a2e1-7039-44a2-aa9a-a66e3e5e4e25-istio-data\") pod \"e2e-noten-c0d4bf-openshift-default-948f987d-4kbcq\" (UID: \"f621a2e1-7039-44a2-aa9a-a66e3e5e4e25\") " pod="openshift-ingress/e2e-noten-c0d4bf-openshift-default-948f987d-4kbcq"
Sep 11 23:23:39.863240 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:39.863223    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f621a2e1-7039-44a2-aa9a-a66e3e5e4e25-istio-token\") pod \"e2e-noten-c0d4bf-openshift-default-948f987d-4kbcq\" (UID: \"f621a2e1-7039-44a2-aa9a-a66e3e5e4e25\") " pod="openshift-ingress/e2e-noten-c0d4bf-openshift-default-948f987d-4kbcq"
Sep 11 23:23:39.863297 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:39.863256    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cskkd\" (UniqueName: \"kubernetes.io/projected/f621a2e1-7039-44a2-aa9a-a66e3e5e4e25-kube-api-access-cskkd\") pod \"e2e-noten-c0d4bf-openshift-default-948f987d-4kbcq\" (UID: \"f621a2e1-7039-44a2-aa9a-a66e3e5e4e25\") " pod="openshift-ingress/e2e-noten-c0d4bf-openshift-default-948f987d-4kbcq"
Sep 11 23:23:39.863354 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:39.863324    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f621a2e1-7039-44a2-aa9a-a66e3e5e4e25-workload-socket\") pod \"e2e-noten-c0d4bf-openshift-default-948f987d-4kbcq\" (UID: \"f621a2e1-7039-44a2-aa9a-a66e3e5e4e25\") " pod="openshift-ingress/e2e-noten-c0d4bf-openshift-default-948f987d-4kbcq"
Sep 11 23:23:39.863407 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:39.863356    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f621a2e1-7039-44a2-aa9a-a66e3e5e4e25-workload-certs\") pod \"e2e-noten-c0d4bf-openshift-default-948f987d-4kbcq\" (UID: \"f621a2e1-7039-44a2-aa9a-a66e3e5e4e25\") " pod="openshift-ingress/e2e-noten-c0d4bf-openshift-default-948f987d-4kbcq"
Sep 11 23:23:39.863460 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:39.863418    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f621a2e1-7039-44a2-aa9a-a66e3e5e4e25-credential-socket\") pod \"e2e-noten-c0d4bf-openshift-default-948f987d-4kbcq\" (UID: \"f621a2e1-7039-44a2-aa9a-a66e3e5e4e25\") " pod="openshift-ingress/e2e-noten-c0d4bf-openshift-default-948f987d-4kbcq"
Sep 11 23:23:39.863518 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:39.863459    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f621a2e1-7039-44a2-aa9a-a66e3e5e4e25-istio-data\") pod \"e2e-noten-c0d4bf-openshift-default-948f987d-4kbcq\" (UID: \"f621a2e1-7039-44a2-aa9a-a66e3e5e4e25\") " pod="openshift-ingress/e2e-noten-c0d4bf-openshift-default-948f987d-4kbcq"
Sep 11 23:23:39.863684 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:39.863653    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f621a2e1-7039-44a2-aa9a-a66e3e5e4e25-workload-certs\") pod \"e2e-noten-c0d4bf-openshift-default-948f987d-4kbcq\" (UID: \"f621a2e1-7039-44a2-aa9a-a66e3e5e4e25\") " pod="openshift-ingress/e2e-noten-c0d4bf-openshift-default-948f987d-4kbcq"
Sep 11 23:23:39.863741 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:39.863688    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f621a2e1-7039-44a2-aa9a-a66e3e5e4e25-workload-socket\") pod \"e2e-noten-c0d4bf-openshift-default-948f987d-4kbcq\" (UID: \"f621a2e1-7039-44a2-aa9a-a66e3e5e4e25\") " pod="openshift-ingress/e2e-noten-c0d4bf-openshift-default-948f987d-4kbcq"
Sep 11 23:23:39.863741 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:39.863710    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f621a2e1-7039-44a2-aa9a-a66e3e5e4e25-istiod-ca-cert\") pod \"e2e-noten-c0d4bf-openshift-default-948f987d-4kbcq\" (UID: \"f621a2e1-7039-44a2-aa9a-a66e3e5e4e25\") " pod="openshift-ingress/e2e-noten-c0d4bf-openshift-default-948f987d-4kbcq"
Sep 11 23:23:39.866012 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:39.865988    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f621a2e1-7039-44a2-aa9a-a66e3e5e4e25-istio-envoy\") pod \"e2e-noten-c0d4bf-openshift-default-948f987d-4kbcq\" (UID: \"f621a2e1-7039-44a2-aa9a-a66e3e5e4e25\") " pod="openshift-ingress/e2e-noten-c0d4bf-openshift-default-948f987d-4kbcq"
Sep 11 23:23:39.866242 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:39.866224    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f621a2e1-7039-44a2-aa9a-a66e3e5e4e25-istio-podinfo\") pod \"e2e-noten-c0d4bf-openshift-default-948f987d-4kbcq\" (UID: \"f621a2e1-7039-44a2-aa9a-a66e3e5e4e25\") " pod="openshift-ingress/e2e-noten-c0d4bf-openshift-default-948f987d-4kbcq"
Sep 11 23:23:39.876458 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:39.876439    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cskkd\" (UniqueName: \"kubernetes.io/projected/f621a2e1-7039-44a2-aa9a-a66e3e5e4e25-kube-api-access-cskkd\") pod \"e2e-noten-c0d4bf-openshift-default-948f987d-4kbcq\" (UID: \"f621a2e1-7039-44a2-aa9a-a66e3e5e4e25\") " pod="openshift-ingress/e2e-noten-c0d4bf-openshift-default-948f987d-4kbcq"
Sep 11 23:23:39.876458 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:39.876451    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f621a2e1-7039-44a2-aa9a-a66e3e5e4e25-istio-token\") pod \"e2e-noten-c0d4bf-openshift-default-948f987d-4kbcq\" (UID: \"f621a2e1-7039-44a2-aa9a-a66e3e5e4e25\") " pod="openshift-ingress/e2e-noten-c0d4bf-openshift-default-948f987d-4kbcq"
Sep 11 23:23:39.976607 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:39.976510    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-c0d4bf-openshift-default-948f987d-4kbcq"
Sep 11 23:23:40.260570 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:40.260543    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-c0d4bf-openshift-default-948f987d-4kbcq"]
Sep 11 23:23:40.261978 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:23:40.261954    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf621a2e1_7039_44a2_aa9a_a66e3e5e4e25.slice/crio-15b4bf8f1cb583890730d97ba8d347f7fce879cbad0f7a70f44c1def4a9a919a WatchSource:0}: Error finding container 15b4bf8f1cb583890730d97ba8d347f7fce879cbad0f7a70f44c1def4a9a919a: Status 404 returned error can't find the container with id 15b4bf8f1cb583890730d97ba8d347f7fce879cbad0f7a70f44c1def4a9a919a
Sep 11 23:23:40.264359 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:40.264328    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:23:40.264463 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:40.264389    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:23:40.264463 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:40.264418    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:23:40.498872 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:40.498833    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0511d07e-cb77-4534-bc61-48f06dfd3bf3" path="/var/lib/kubelet/pods/0511d07e-cb77-4534-bc61-48f06dfd3bf3/volumes"
Sep 11 23:23:40.797739 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:40.797705    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-c0d4bf-openshift-default-948f987d-4kbcq" event={"ID":"f621a2e1-7039-44a2-aa9a-a66e3e5e4e25","Type":"ContainerStarted","Data":"06d4646b62b10d2abdf17449008d5067e24eb56d9735cd3a672cceed3ef30230"}
Sep 11 23:23:40.798121 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:40.797763    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-c0d4bf-openshift-default-948f987d-4kbcq" event={"ID":"f621a2e1-7039-44a2-aa9a-a66e3e5e4e25","Type":"ContainerStarted","Data":"15b4bf8f1cb583890730d97ba8d347f7fce879cbad0f7a70f44c1def4a9a919a"}
Sep 11 23:23:40.819997 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:40.819949    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-c0d4bf-openshift-default-948f987d-4kbcq" podStartSLOduration=1.819933819 podStartE2EDuration="1.819933819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 23:23:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:23:40.818955041 +0000 UTC m=+1344.852084259" watchObservedRunningTime="2026-09-11 23:23:40.819933819 +0000 UTC m=+1344.853063034"
Sep 11 23:23:40.977525 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:40.977484    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-c0d4bf-openshift-default-948f987d-4kbcq"
Sep 11 23:23:41.983570 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:41.983540    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-c0d4bf-openshift-default-948f987d-4kbcq"
Sep 11 23:23:42.811389 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:42.811357    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-c0d4bf-openshift-default-948f987d-4kbcq"
Sep 11 23:23:42.812303 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:42.812285    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-c0d4bf-openshift-default-948f987d-4kbcq"
Sep 11 23:23:45.156233 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:45.156176    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-cd48eb-openshift-default-76cd487f58-fm5sq"]
Sep 11 23:23:45.156856 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:45.156835    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0511d07e-cb77-4534-bc61-48f06dfd3bf3" containerName="istio-proxy"
Sep 11 23:23:45.156939 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:45.156859    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="0511d07e-cb77-4534-bc61-48f06dfd3bf3" containerName="istio-proxy"
Sep 11 23:23:45.157029 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:45.157016    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="0511d07e-cb77-4534-bc61-48f06dfd3bf3" containerName="istio-proxy"
Sep 11 23:23:45.157087 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:45.157040    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4575ea1-784b-4c29-9eff-a2d23d6772a8" containerName="istio-proxy"
Sep 11 23:23:45.160266 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:45.160245    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-cd48eb-openshift-default-76cd487f58-fm5sq"
Sep 11 23:23:45.163320 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:45.163301    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-cd48eb-openshift-default-dockercfg-f7vfm\""
Sep 11 23:23:45.172801 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:45.172773    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-cd48eb-openshift-default-76cd487f58-fm5sq"]
Sep 11 23:23:45.317644 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:45.317608    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48cd2fcb-f864-4e43-bcfb-cadfd9964332-workload-socket\") pod \"e2e-shared-a-w2-cd48eb-openshift-default-76cd487f58-fm5sq\" (UID: \"48cd2fcb-f864-4e43-bcfb-cadfd9964332\") " pod="openshift-ingress/e2e-shared-a-w2-cd48eb-openshift-default-76cd487f58-fm5sq"
Sep 11 23:23:45.317644 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:45.317643    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48cd2fcb-f864-4e43-bcfb-cadfd9964332-credential-socket\") pod \"e2e-shared-a-w2-cd48eb-openshift-default-76cd487f58-fm5sq\" (UID: \"48cd2fcb-f864-4e43-bcfb-cadfd9964332\") " pod="openshift-ingress/e2e-shared-a-w2-cd48eb-openshift-default-76cd487f58-fm5sq"
Sep 11 23:23:45.317858 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:45.317675    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48cd2fcb-f864-4e43-bcfb-cadfd9964332-istio-data\") pod \"e2e-shared-a-w2-cd48eb-openshift-default-76cd487f58-fm5sq\" (UID: \"48cd2fcb-f864-4e43-bcfb-cadfd9964332\") " pod="openshift-ingress/e2e-shared-a-w2-cd48eb-openshift-default-76cd487f58-fm5sq"
Sep 11 23:23:45.317858 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:45.317754    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnv58\" (UniqueName: \"kubernetes.io/projected/48cd2fcb-f864-4e43-bcfb-cadfd9964332-kube-api-access-dnv58\") pod \"e2e-shared-a-w2-cd48eb-openshift-default-76cd487f58-fm5sq\" (UID: \"48cd2fcb-f864-4e43-bcfb-cadfd9964332\") " pod="openshift-ingress/e2e-shared-a-w2-cd48eb-openshift-default-76cd487f58-fm5sq"
Sep 11 23:23:45.317858 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:45.317845    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48cd2fcb-f864-4e43-bcfb-cadfd9964332-istio-token\") pod \"e2e-shared-a-w2-cd48eb-openshift-default-76cd487f58-fm5sq\" (UID: \"48cd2fcb-f864-4e43-bcfb-cadfd9964332\") " pod="openshift-ingress/e2e-shared-a-w2-cd48eb-openshift-default-76cd487f58-fm5sq"
Sep 11 23:23:45.317955 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:45.317871    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48cd2fcb-f864-4e43-bcfb-cadfd9964332-istio-envoy\") pod \"e2e-shared-a-w2-cd48eb-openshift-default-76cd487f58-fm5sq\" (UID: \"48cd2fcb-f864-4e43-bcfb-cadfd9964332\") " pod="openshift-ingress/e2e-shared-a-w2-cd48eb-openshift-default-76cd487f58-fm5sq"
Sep 11 23:23:45.317955 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:45.317890    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48cd2fcb-f864-4e43-bcfb-cadfd9964332-workload-certs\") pod \"e2e-shared-a-w2-cd48eb-openshift-default-76cd487f58-fm5sq\" (UID: \"48cd2fcb-f864-4e43-bcfb-cadfd9964332\") " pod="openshift-ingress/e2e-shared-a-w2-cd48eb-openshift-default-76cd487f58-fm5sq"
Sep 11 23:23:45.317955 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:45.317921    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48cd2fcb-f864-4e43-bcfb-cadfd9964332-istiod-ca-cert\") pod \"e2e-shared-a-w2-cd48eb-openshift-default-76cd487f58-fm5sq\" (UID: \"48cd2fcb-f864-4e43-bcfb-cadfd9964332\") " pod="openshift-ingress/e2e-shared-a-w2-cd48eb-openshift-default-76cd487f58-fm5sq"
Sep 11 23:23:45.317955 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:45.317940    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48cd2fcb-f864-4e43-bcfb-cadfd9964332-istio-podinfo\") pod \"e2e-shared-a-w2-cd48eb-openshift-default-76cd487f58-fm5sq\" (UID: \"48cd2fcb-f864-4e43-bcfb-cadfd9964332\") " pod="openshift-ingress/e2e-shared-a-w2-cd48eb-openshift-default-76cd487f58-fm5sq"
Sep 11 23:23:45.419213 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:45.419099    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48cd2fcb-f864-4e43-bcfb-cadfd9964332-istiod-ca-cert\") pod \"e2e-shared-a-w2-cd48eb-openshift-default-76cd487f58-fm5sq\" (UID: \"48cd2fcb-f864-4e43-bcfb-cadfd9964332\") " pod="openshift-ingress/e2e-shared-a-w2-cd48eb-openshift-default-76cd487f58-fm5sq"
Sep 11 23:23:45.419213 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:45.419142    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48cd2fcb-f864-4e43-bcfb-cadfd9964332-istio-podinfo\") pod \"e2e-shared-a-w2-cd48eb-openshift-default-76cd487f58-fm5sq\" (UID: \"48cd2fcb-f864-4e43-bcfb-cadfd9964332\") " pod="openshift-ingress/e2e-shared-a-w2-cd48eb-openshift-default-76cd487f58-fm5sq"
Sep 11 23:23:45.419457 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:45.419217    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48cd2fcb-f864-4e43-bcfb-cadfd9964332-workload-socket\") pod \"e2e-shared-a-w2-cd48eb-openshift-default-76cd487f58-fm5sq\" (UID: \"48cd2fcb-f864-4e43-bcfb-cadfd9964332\") " pod="openshift-ingress/e2e-shared-a-w2-cd48eb-openshift-default-76cd487f58-fm5sq"
Sep 11 23:23:45.419457 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:45.419233    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48cd2fcb-f864-4e43-bcfb-cadfd9964332-credential-socket\") pod \"e2e-shared-a-w2-cd48eb-openshift-default-76cd487f58-fm5sq\" (UID: \"48cd2fcb-f864-4e43-bcfb-cadfd9964332\") " pod="openshift-ingress/e2e-shared-a-w2-cd48eb-openshift-default-76cd487f58-fm5sq"
Sep 11 23:23:45.419457 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:45.419270    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48cd2fcb-f864-4e43-bcfb-cadfd9964332-istio-data\") pod \"e2e-shared-a-w2-cd48eb-openshift-default-76cd487f58-fm5sq\" (UID: \"48cd2fcb-f864-4e43-bcfb-cadfd9964332\") " pod="openshift-ingress/e2e-shared-a-w2-cd48eb-openshift-default-76cd487f58-fm5sq"
Sep 11 23:23:45.419457 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:45.419297    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnv58\" (UniqueName: \"kubernetes.io/projected/48cd2fcb-f864-4e43-bcfb-cadfd9964332-kube-api-access-dnv58\") pod \"e2e-shared-a-w2-cd48eb-openshift-default-76cd487f58-fm5sq\" (UID: \"48cd2fcb-f864-4e43-bcfb-cadfd9964332\") " pod="openshift-ingress/e2e-shared-a-w2-cd48eb-openshift-default-76cd487f58-fm5sq"
Sep 11 23:23:45.419457 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:45.419399    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48cd2fcb-f864-4e43-bcfb-cadfd9964332-istio-token\") pod \"e2e-shared-a-w2-cd48eb-openshift-default-76cd487f58-fm5sq\" (UID: \"48cd2fcb-f864-4e43-bcfb-cadfd9964332\") " pod="openshift-ingress/e2e-shared-a-w2-cd48eb-openshift-default-76cd487f58-fm5sq"
Sep 11 23:23:45.419457 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:45.419440    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48cd2fcb-f864-4e43-bcfb-cadfd9964332-istio-envoy\") pod \"e2e-shared-a-w2-cd48eb-openshift-default-76cd487f58-fm5sq\" (UID: \"48cd2fcb-f864-4e43-bcfb-cadfd9964332\") " pod="openshift-ingress/e2e-shared-a-w2-cd48eb-openshift-default-76cd487f58-fm5sq"
Sep 11 23:23:45.419778 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:45.419469    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48cd2fcb-f864-4e43-bcfb-cadfd9964332-workload-certs\") pod \"e2e-shared-a-w2-cd48eb-openshift-default-76cd487f58-fm5sq\" (UID: \"48cd2fcb-f864-4e43-bcfb-cadfd9964332\") " pod="openshift-ingress/e2e-shared-a-w2-cd48eb-openshift-default-76cd487f58-fm5sq"
Sep 11 23:23:45.419778 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:45.419644    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48cd2fcb-f864-4e43-bcfb-cadfd9964332-istio-data\") pod \"e2e-shared-a-w2-cd48eb-openshift-default-76cd487f58-fm5sq\" (UID: \"48cd2fcb-f864-4e43-bcfb-cadfd9964332\") " pod="openshift-ingress/e2e-shared-a-w2-cd48eb-openshift-default-76cd487f58-fm5sq"
Sep 11 23:23:45.419778 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:45.419696    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48cd2fcb-f864-4e43-bcfb-cadfd9964332-credential-socket\") pod \"e2e-shared-a-w2-cd48eb-openshift-default-76cd487f58-fm5sq\" (UID: \"48cd2fcb-f864-4e43-bcfb-cadfd9964332\") " pod="openshift-ingress/e2e-shared-a-w2-cd48eb-openshift-default-76cd487f58-fm5sq"
Sep 11 23:23:45.419908 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:45.419887    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48cd2fcb-f864-4e43-bcfb-cadfd9964332-istiod-ca-cert\") pod \"e2e-shared-a-w2-cd48eb-openshift-default-76cd487f58-fm5sq\" (UID: \"48cd2fcb-f864-4e43-bcfb-cadfd9964332\") " pod="openshift-ingress/e2e-shared-a-w2-cd48eb-openshift-default-76cd487f58-fm5sq"
Sep 11 23:23:45.419963 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:45.419931    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48cd2fcb-f864-4e43-bcfb-cadfd9964332-workload-certs\") pod \"e2e-shared-a-w2-cd48eb-openshift-default-76cd487f58-fm5sq\" (UID: \"48cd2fcb-f864-4e43-bcfb-cadfd9964332\") " pod="openshift-ingress/e2e-shared-a-w2-cd48eb-openshift-default-76cd487f58-fm5sq"
Sep 11 23:23:45.420016 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:45.419961    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48cd2fcb-f864-4e43-bcfb-cadfd9964332-workload-socket\") pod \"e2e-shared-a-w2-cd48eb-openshift-default-76cd487f58-fm5sq\" (UID: \"48cd2fcb-f864-4e43-bcfb-cadfd9964332\") " pod="openshift-ingress/e2e-shared-a-w2-cd48eb-openshift-default-76cd487f58-fm5sq"
Sep 11 23:23:45.421879 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:45.421851    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48cd2fcb-f864-4e43-bcfb-cadfd9964332-istio-podinfo\") pod \"e2e-shared-a-w2-cd48eb-openshift-default-76cd487f58-fm5sq\" (UID: \"48cd2fcb-f864-4e43-bcfb-cadfd9964332\") " pod="openshift-ingress/e2e-shared-a-w2-cd48eb-openshift-default-76cd487f58-fm5sq"
Sep 11 23:23:45.421981 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:45.421933    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48cd2fcb-f864-4e43-bcfb-cadfd9964332-istio-envoy\") pod \"e2e-shared-a-w2-cd48eb-openshift-default-76cd487f58-fm5sq\" (UID: \"48cd2fcb-f864-4e43-bcfb-cadfd9964332\") " pod="openshift-ingress/e2e-shared-a-w2-cd48eb-openshift-default-76cd487f58-fm5sq"
Sep 11 23:23:45.429988 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:45.429959    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48cd2fcb-f864-4e43-bcfb-cadfd9964332-istio-token\") pod \"e2e-shared-a-w2-cd48eb-openshift-default-76cd487f58-fm5sq\" (UID: \"48cd2fcb-f864-4e43-bcfb-cadfd9964332\") " pod="openshift-ingress/e2e-shared-a-w2-cd48eb-openshift-default-76cd487f58-fm5sq"
Sep 11 23:23:45.430127 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:45.430039    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnv58\" (UniqueName: \"kubernetes.io/projected/48cd2fcb-f864-4e43-bcfb-cadfd9964332-kube-api-access-dnv58\") pod \"e2e-shared-a-w2-cd48eb-openshift-default-76cd487f58-fm5sq\" (UID: \"48cd2fcb-f864-4e43-bcfb-cadfd9964332\") " pod="openshift-ingress/e2e-shared-a-w2-cd48eb-openshift-default-76cd487f58-fm5sq"
Sep 11 23:23:45.472182 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:45.472129    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-cd48eb-openshift-default-76cd487f58-fm5sq"
Sep 11 23:23:45.759091 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:45.759061    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-cd48eb-openshift-default-76cd487f58-fm5sq"]
Sep 11 23:23:45.760664 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:23:45.760631    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48cd2fcb_f864_4e43_bcfb_cadfd9964332.slice/crio-6e1000816275819923875ffb8ff369f45fff36d95e094f2f2b14874ded68629f WatchSource:0}: Error finding container 6e1000816275819923875ffb8ff369f45fff36d95e094f2f2b14874ded68629f: Status 404 returned error can't find the container with id 6e1000816275819923875ffb8ff369f45fff36d95e094f2f2b14874ded68629f
Sep 11 23:23:45.763005 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:45.762973    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:23:45.763113 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:45.763053    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:23:45.763113 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:45.763099    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:23:45.835199 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:45.835149    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-cd48eb-openshift-default-76cd487f58-fm5sq" event={"ID":"48cd2fcb-f864-4e43-bcfb-cadfd9964332","Type":"ContainerStarted","Data":"e290b7f75aac5587df44419127c3cb87ba173a649c58400ae948239528fc8896"}
Sep 11 23:23:45.835340 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:45.835217    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-cd48eb-openshift-default-76cd487f58-fm5sq" event={"ID":"48cd2fcb-f864-4e43-bcfb-cadfd9964332","Type":"ContainerStarted","Data":"6e1000816275819923875ffb8ff369f45fff36d95e094f2f2b14874ded68629f"}
Sep 11 23:23:45.857206 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:45.857130    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-cd48eb-openshift-default-76cd487f58-fm5sq" podStartSLOduration=0.857108035 podStartE2EDuration="857.108035ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 23:23:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:23:45.856356423 +0000 UTC m=+1349.889485651" watchObservedRunningTime="2026-09-11 23:23:45.857108035 +0000 UTC m=+1349.890237253"
Sep 11 23:23:46.473360 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:46.473323    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-cd48eb-openshift-default-76cd487f58-fm5sq"
Sep 11 23:23:46.478251 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:46.478227    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-cd48eb-openshift-default-76cd487f58-fm5sq"
Sep 11 23:23:46.843470 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:46.843372    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-cd48eb-openshift-default-76cd487f58-fm5sq"
Sep 11 23:23:46.844541 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:46.844518    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-cd48eb-openshift-default-76cd487f58-fm5sq"
Sep 11 23:23:58.799013 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:23:58.798975    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"]
Sep 11 23:24:10.013672 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:10.013640    2546 generic.go:356] "Generic (PLEG): container finished" podID="48cd2fcb-f864-4e43-bcfb-cadfd9964332" containerID="e290b7f75aac5587df44419127c3cb87ba173a649c58400ae948239528fc8896" exitCode=0
Sep 11 23:24:10.014084 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:10.013713    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-cd48eb-openshift-default-76cd487f58-fm5sq" event={"ID":"48cd2fcb-f864-4e43-bcfb-cadfd9964332","Type":"ContainerDied","Data":"e290b7f75aac5587df44419127c3cb87ba173a649c58400ae948239528fc8896"}
Sep 11 23:24:10.014135 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:10.014123    2546 scope.go:117] "RemoveContainer" containerID="e290b7f75aac5587df44419127c3cb87ba173a649c58400ae948239528fc8896"
Sep 11 23:24:10.015314 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:10.015286    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:24:10.015391 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:10.015357    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:24:10.015391 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:10.015387    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:24:11.021910 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:11.021879    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-cd48eb-openshift-default-76cd487f58-fm5sq" event={"ID":"48cd2fcb-f864-4e43-bcfb-cadfd9964332","Type":"ContainerStarted","Data":"39c40ddc5208abebb3b8b5bf03a456803ce38921ff8f48b6f8684349369088f5"}
Sep 11 23:24:11.472518 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:11.472475    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-cd48eb-openshift-default-76cd487f58-fm5sq"
Sep 11 23:24:11.472743 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:11.472678    2546 patch_prober.go:28] interesting pod/e2e-shared-a-w2-cd48eb-openshift-default-76cd487f58-fm5sq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Sep 11 23:24:11.472819 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:11.472731    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-cd48eb-openshift-default-76cd487f58-fm5sq" podUID="48cd2fcb-f864-4e43-bcfb-cadfd9964332" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Sep 11 23:24:12.473116 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:12.473077    2546 patch_prober.go:28] interesting pod/e2e-shared-a-w2-cd48eb-openshift-default-76cd487f58-fm5sq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Sep 11 23:24:12.473555 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:12.473135    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-cd48eb-openshift-default-76cd487f58-fm5sq" podUID="48cd2fcb-f864-4e43-bcfb-cadfd9964332" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Sep 11 23:24:13.472997 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:13.472952    2546 patch_prober.go:28] interesting pod/e2e-shared-a-w2-cd48eb-openshift-default-76cd487f58-fm5sq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Sep 11 23:24:13.473248 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:13.473025    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-cd48eb-openshift-default-76cd487f58-fm5sq" podUID="48cd2fcb-f864-4e43-bcfb-cadfd9964332" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Sep 11 23:24:14.472790 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:14.472753    2546 patch_prober.go:28] interesting pod/e2e-shared-a-w2-cd48eb-openshift-default-76cd487f58-fm5sq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Sep 11 23:24:14.472969 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:14.472823    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-cd48eb-openshift-default-76cd487f58-fm5sq" podUID="48cd2fcb-f864-4e43-bcfb-cadfd9964332" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Sep 11 23:24:15.472538 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:15.472501    2546 patch_prober.go:28] interesting pod/e2e-shared-a-w2-cd48eb-openshift-default-76cd487f58-fm5sq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Sep 11 23:24:15.473011 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:15.472565    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-cd48eb-openshift-default-76cd487f58-fm5sq" podUID="48cd2fcb-f864-4e43-bcfb-cadfd9964332" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Sep 11 23:24:16.477714 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:16.477686    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-cd48eb-openshift-default-76cd487f58-fm5sq"
Sep 11 23:24:16.478180 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:16.478086    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-cd48eb-openshift-default-76cd487f58-fm5sq"
Sep 11 23:24:16.478750 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:16.478735    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-cd48eb-openshift-default-76cd487f58-fm5sq"
Sep 11 23:24:18.280131 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:18.280092    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-97d9dc-openshift-default-7bfd65cf68-sltcx"]
Sep 11 23:24:18.285249 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:18.285228    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-97d9dc-openshift-default-7bfd65cf68-sltcx"
Sep 11 23:24:18.290190 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:18.290163    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-97d9dc-openshift-default-dockercfg-dclwn\""
Sep 11 23:24:18.322437 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:18.322398    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-97d9dc-openshift-default-7bfd65cf68-sltcx"]
Sep 11 23:24:18.398429 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:18.392769    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44b0b3b3-a51c-42bf-8539-f11924e397db-istiod-ca-cert\") pod \"e2e-shared-b-w2-97d9dc-openshift-default-7bfd65cf68-sltcx\" (UID: \"44b0b3b3-a51c-42bf-8539-f11924e397db\") " pod="openshift-ingress/e2e-shared-b-w2-97d9dc-openshift-default-7bfd65cf68-sltcx"
Sep 11 23:24:18.398429 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:18.392971    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44b0b3b3-a51c-42bf-8539-f11924e397db-workload-socket\") pod \"e2e-shared-b-w2-97d9dc-openshift-default-7bfd65cf68-sltcx\" (UID: \"44b0b3b3-a51c-42bf-8539-f11924e397db\") " pod="openshift-ingress/e2e-shared-b-w2-97d9dc-openshift-default-7bfd65cf68-sltcx"
Sep 11 23:24:18.398429 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:18.393047    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44b0b3b3-a51c-42bf-8539-f11924e397db-istio-podinfo\") pod \"e2e-shared-b-w2-97d9dc-openshift-default-7bfd65cf68-sltcx\" (UID: \"44b0b3b3-a51c-42bf-8539-f11924e397db\") " pod="openshift-ingress/e2e-shared-b-w2-97d9dc-openshift-default-7bfd65cf68-sltcx"
Sep 11 23:24:18.398429 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:18.393087    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4tgf\" (UniqueName: \"kubernetes.io/projected/44b0b3b3-a51c-42bf-8539-f11924e397db-kube-api-access-h4tgf\") pod \"e2e-shared-b-w2-97d9dc-openshift-default-7bfd65cf68-sltcx\" (UID: \"44b0b3b3-a51c-42bf-8539-f11924e397db\") " pod="openshift-ingress/e2e-shared-b-w2-97d9dc-openshift-default-7bfd65cf68-sltcx"
Sep 11 23:24:18.398429 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:18.393171    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44b0b3b3-a51c-42bf-8539-f11924e397db-workload-certs\") pod \"e2e-shared-b-w2-97d9dc-openshift-default-7bfd65cf68-sltcx\" (UID: \"44b0b3b3-a51c-42bf-8539-f11924e397db\") " pod="openshift-ingress/e2e-shared-b-w2-97d9dc-openshift-default-7bfd65cf68-sltcx"
Sep 11 23:24:18.398429 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:18.393320    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44b0b3b3-a51c-42bf-8539-f11924e397db-istio-data\") pod \"e2e-shared-b-w2-97d9dc-openshift-default-7bfd65cf68-sltcx\" (UID: \"44b0b3b3-a51c-42bf-8539-f11924e397db\") " pod="openshift-ingress/e2e-shared-b-w2-97d9dc-openshift-default-7bfd65cf68-sltcx"
Sep 11 23:24:18.398429 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:18.393360    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44b0b3b3-a51c-42bf-8539-f11924e397db-istio-token\") pod \"e2e-shared-b-w2-97d9dc-openshift-default-7bfd65cf68-sltcx\" (UID: \"44b0b3b3-a51c-42bf-8539-f11924e397db\") " pod="openshift-ingress/e2e-shared-b-w2-97d9dc-openshift-default-7bfd65cf68-sltcx"
Sep 11 23:24:18.398429 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:18.393421    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44b0b3b3-a51c-42bf-8539-f11924e397db-istio-envoy\") pod \"e2e-shared-b-w2-97d9dc-openshift-default-7bfd65cf68-sltcx\" (UID: \"44b0b3b3-a51c-42bf-8539-f11924e397db\") " pod="openshift-ingress/e2e-shared-b-w2-97d9dc-openshift-default-7bfd65cf68-sltcx"
Sep 11 23:24:18.398429 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:18.393446    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44b0b3b3-a51c-42bf-8539-f11924e397db-credential-socket\") pod \"e2e-shared-b-w2-97d9dc-openshift-default-7bfd65cf68-sltcx\" (UID: \"44b0b3b3-a51c-42bf-8539-f11924e397db\") " pod="openshift-ingress/e2e-shared-b-w2-97d9dc-openshift-default-7bfd65cf68-sltcx"
Sep 11 23:24:18.494103 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:18.494063    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44b0b3b3-a51c-42bf-8539-f11924e397db-istio-data\") pod \"e2e-shared-b-w2-97d9dc-openshift-default-7bfd65cf68-sltcx\" (UID: \"44b0b3b3-a51c-42bf-8539-f11924e397db\") " pod="openshift-ingress/e2e-shared-b-w2-97d9dc-openshift-default-7bfd65cf68-sltcx"
Sep 11 23:24:18.494323 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:18.494123    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44b0b3b3-a51c-42bf-8539-f11924e397db-istio-token\") pod \"e2e-shared-b-w2-97d9dc-openshift-default-7bfd65cf68-sltcx\" (UID: \"44b0b3b3-a51c-42bf-8539-f11924e397db\") " pod="openshift-ingress/e2e-shared-b-w2-97d9dc-openshift-default-7bfd65cf68-sltcx"
Sep 11 23:24:18.494323 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:18.494210    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44b0b3b3-a51c-42bf-8539-f11924e397db-istio-envoy\") pod \"e2e-shared-b-w2-97d9dc-openshift-default-7bfd65cf68-sltcx\" (UID: \"44b0b3b3-a51c-42bf-8539-f11924e397db\") " pod="openshift-ingress/e2e-shared-b-w2-97d9dc-openshift-default-7bfd65cf68-sltcx"
Sep 11 23:24:18.494323 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:18.494240    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44b0b3b3-a51c-42bf-8539-f11924e397db-credential-socket\") pod \"e2e-shared-b-w2-97d9dc-openshift-default-7bfd65cf68-sltcx\" (UID: \"44b0b3b3-a51c-42bf-8539-f11924e397db\") " pod="openshift-ingress/e2e-shared-b-w2-97d9dc-openshift-default-7bfd65cf68-sltcx"
Sep 11 23:24:18.494492 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:18.494398    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44b0b3b3-a51c-42bf-8539-f11924e397db-istiod-ca-cert\") pod \"e2e-shared-b-w2-97d9dc-openshift-default-7bfd65cf68-sltcx\" (UID: \"44b0b3b3-a51c-42bf-8539-f11924e397db\") " pod="openshift-ingress/e2e-shared-b-w2-97d9dc-openshift-default-7bfd65cf68-sltcx"
Sep 11 23:24:18.494492 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:18.494459    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44b0b3b3-a51c-42bf-8539-f11924e397db-workload-socket\") pod \"e2e-shared-b-w2-97d9dc-openshift-default-7bfd65cf68-sltcx\" (UID: \"44b0b3b3-a51c-42bf-8539-f11924e397db\") " pod="openshift-ingress/e2e-shared-b-w2-97d9dc-openshift-default-7bfd65cf68-sltcx"
Sep 11 23:24:18.494596 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:18.494489    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44b0b3b3-a51c-42bf-8539-f11924e397db-istio-podinfo\") pod \"e2e-shared-b-w2-97d9dc-openshift-default-7bfd65cf68-sltcx\" (UID: \"44b0b3b3-a51c-42bf-8539-f11924e397db\") " pod="openshift-ingress/e2e-shared-b-w2-97d9dc-openshift-default-7bfd65cf68-sltcx"
Sep 11 23:24:18.494596 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:18.494499    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44b0b3b3-a51c-42bf-8539-f11924e397db-istio-data\") pod \"e2e-shared-b-w2-97d9dc-openshift-default-7bfd65cf68-sltcx\" (UID: \"44b0b3b3-a51c-42bf-8539-f11924e397db\") " pod="openshift-ingress/e2e-shared-b-w2-97d9dc-openshift-default-7bfd65cf68-sltcx"
Sep 11 23:24:18.494596 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:18.494528    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4tgf\" (UniqueName: \"kubernetes.io/projected/44b0b3b3-a51c-42bf-8539-f11924e397db-kube-api-access-h4tgf\") pod \"e2e-shared-b-w2-97d9dc-openshift-default-7bfd65cf68-sltcx\" (UID: \"44b0b3b3-a51c-42bf-8539-f11924e397db\") " pod="openshift-ingress/e2e-shared-b-w2-97d9dc-openshift-default-7bfd65cf68-sltcx"
Sep 11 23:24:18.494596 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:18.494585    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44b0b3b3-a51c-42bf-8539-f11924e397db-workload-certs\") pod \"e2e-shared-b-w2-97d9dc-openshift-default-7bfd65cf68-sltcx\" (UID: \"44b0b3b3-a51c-42bf-8539-f11924e397db\") " pod="openshift-ingress/e2e-shared-b-w2-97d9dc-openshift-default-7bfd65cf68-sltcx"
Sep 11 23:24:18.494826 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:18.494788    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44b0b3b3-a51c-42bf-8539-f11924e397db-workload-socket\") pod \"e2e-shared-b-w2-97d9dc-openshift-default-7bfd65cf68-sltcx\" (UID: \"44b0b3b3-a51c-42bf-8539-f11924e397db\") " pod="openshift-ingress/e2e-shared-b-w2-97d9dc-openshift-default-7bfd65cf68-sltcx"
Sep 11 23:24:18.494986 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:18.494964    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44b0b3b3-a51c-42bf-8539-f11924e397db-workload-certs\") pod \"e2e-shared-b-w2-97d9dc-openshift-default-7bfd65cf68-sltcx\" (UID: \"44b0b3b3-a51c-42bf-8539-f11924e397db\") " pod="openshift-ingress/e2e-shared-b-w2-97d9dc-openshift-default-7bfd65cf68-sltcx"
Sep 11 23:24:18.495039 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:18.495013    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44b0b3b3-a51c-42bf-8539-f11924e397db-credential-socket\") pod \"e2e-shared-b-w2-97d9dc-openshift-default-7bfd65cf68-sltcx\" (UID: \"44b0b3b3-a51c-42bf-8539-f11924e397db\") " pod="openshift-ingress/e2e-shared-b-w2-97d9dc-openshift-default-7bfd65cf68-sltcx"
Sep 11 23:24:18.495148 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:18.495105    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44b0b3b3-a51c-42bf-8539-f11924e397db-istiod-ca-cert\") pod \"e2e-shared-b-w2-97d9dc-openshift-default-7bfd65cf68-sltcx\" (UID: \"44b0b3b3-a51c-42bf-8539-f11924e397db\") " pod="openshift-ingress/e2e-shared-b-w2-97d9dc-openshift-default-7bfd65cf68-sltcx"
Sep 11 23:24:18.497408 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:18.497378    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44b0b3b3-a51c-42bf-8539-f11924e397db-istio-envoy\") pod \"e2e-shared-b-w2-97d9dc-openshift-default-7bfd65cf68-sltcx\" (UID: \"44b0b3b3-a51c-42bf-8539-f11924e397db\") " pod="openshift-ingress/e2e-shared-b-w2-97d9dc-openshift-default-7bfd65cf68-sltcx"
Sep 11 23:24:18.497502 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:18.497487    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44b0b3b3-a51c-42bf-8539-f11924e397db-istio-podinfo\") pod \"e2e-shared-b-w2-97d9dc-openshift-default-7bfd65cf68-sltcx\" (UID: \"44b0b3b3-a51c-42bf-8539-f11924e397db\") " pod="openshift-ingress/e2e-shared-b-w2-97d9dc-openshift-default-7bfd65cf68-sltcx"
Sep 11 23:24:18.508256 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:18.508237    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4tgf\" (UniqueName: \"kubernetes.io/projected/44b0b3b3-a51c-42bf-8539-f11924e397db-kube-api-access-h4tgf\") pod \"e2e-shared-b-w2-97d9dc-openshift-default-7bfd65cf68-sltcx\" (UID: \"44b0b3b3-a51c-42bf-8539-f11924e397db\") " pod="openshift-ingress/e2e-shared-b-w2-97d9dc-openshift-default-7bfd65cf68-sltcx"
Sep 11 23:24:18.510328 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:18.510312    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44b0b3b3-a51c-42bf-8539-f11924e397db-istio-token\") pod \"e2e-shared-b-w2-97d9dc-openshift-default-7bfd65cf68-sltcx\" (UID: \"44b0b3b3-a51c-42bf-8539-f11924e397db\") " pod="openshift-ingress/e2e-shared-b-w2-97d9dc-openshift-default-7bfd65cf68-sltcx"
Sep 11 23:24:18.595634 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:18.595541    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-97d9dc-openshift-default-7bfd65cf68-sltcx"
Sep 11 23:24:18.879674 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:18.879641    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-97d9dc-openshift-default-7bfd65cf68-sltcx"]
Sep 11 23:24:18.880188 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:24:18.880135    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44b0b3b3_a51c_42bf_8539_f11924e397db.slice/crio-f26e24eea4d0b8274f5dee70967b0edde9420ad812e09d83e7e0aaeae1aa12fd WatchSource:0}: Error finding container f26e24eea4d0b8274f5dee70967b0edde9420ad812e09d83e7e0aaeae1aa12fd: Status 404 returned error can't find the container with id f26e24eea4d0b8274f5dee70967b0edde9420ad812e09d83e7e0aaeae1aa12fd
Sep 11 23:24:18.882585 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:18.882549    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:24:18.882650 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:18.882615    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:24:18.882686 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:18.882650    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:24:19.092919 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:19.092872    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-97d9dc-openshift-default-7bfd65cf68-sltcx" event={"ID":"44b0b3b3-a51c-42bf-8539-f11924e397db","Type":"ContainerStarted","Data":"c9220478099430b9b95d3cc848e6d6b52138fa1f52c837570c992c51b53ee435"}
Sep 11 23:24:19.093098 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:19.092936    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-97d9dc-openshift-default-7bfd65cf68-sltcx" event={"ID":"44b0b3b3-a51c-42bf-8539-f11924e397db","Type":"ContainerStarted","Data":"f26e24eea4d0b8274f5dee70967b0edde9420ad812e09d83e7e0aaeae1aa12fd"}
Sep 11 23:24:19.116427 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:19.116364    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-97d9dc-openshift-default-7bfd65cf68-sltcx" podStartSLOduration=1.116341898 podStartE2EDuration="1.116341898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 23:24:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:24:19.112759902 +0000 UTC m=+1383.145889115" watchObservedRunningTime="2026-09-11 23:24:19.116341898 +0000 UTC m=+1383.149471113"
Sep 11 23:24:19.596667 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:19.596634    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-97d9dc-openshift-default-7bfd65cf68-sltcx"
Sep 11 23:24:19.601685 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:19.601659    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-97d9dc-openshift-default-7bfd65cf68-sltcx"
Sep 11 23:24:20.099494 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:20.099467    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-97d9dc-openshift-default-7bfd65cf68-sltcx"
Sep 11 23:24:20.100577 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:20.100560    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-97d9dc-openshift-default-7bfd65cf68-sltcx"
Sep 11 23:24:32.806434 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:32.806401    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"]
Sep 11 23:24:52.605027 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:52.604917    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-92d8ce-l42r7"]
Sep 11 23:24:52.608585 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:52.608568    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-92d8ce-l42r7"
Sep 11 23:24:52.611424 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:52.611399    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 23:24:52.611686 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:52.611654    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vrx7m\""
Sep 11 23:24:52.617562 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:52.617542    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-92d8ce-l42r7"]
Sep 11 23:24:52.778750 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:52.778702    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4be2238-df2d-4509-aceb-d3424645b91e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-92d8ce-l42r7\" (UID: \"f4be2238-df2d-4509-aceb-d3424645b91e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-92d8ce-l42r7"
Sep 11 23:24:52.879768 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:52.879671    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4be2238-df2d-4509-aceb-d3424645b91e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-92d8ce-l42r7\" (UID: \"f4be2238-df2d-4509-aceb-d3424645b91e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-92d8ce-l42r7"
Sep 11 23:24:52.880319 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:52.880298    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4be2238-df2d-4509-aceb-d3424645b91e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-92d8ce-l42r7\" (UID: \"f4be2238-df2d-4509-aceb-d3424645b91e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-92d8ce-l42r7"
Sep 11 23:24:52.924122 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:52.924078    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-92d8ce-l42r7"
Sep 11 23:24:53.211866 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:53.211837    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-92d8ce-l42r7"]
Sep 11 23:24:53.213311 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:24:53.213278    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4be2238_df2d_4509_aceb_d3424645b91e.slice/crio-8564ba30be4e452c2a15b4b04834a764c99dbc7dbd7d1a87a8ac46cb7367a50b WatchSource:0}: Error finding container 8564ba30be4e452c2a15b4b04834a764c99dbc7dbd7d1a87a8ac46cb7367a50b: Status 404 returned error can't find the container with id 8564ba30be4e452c2a15b4b04834a764c99dbc7dbd7d1a87a8ac46cb7367a50b
Sep 11 23:24:53.332588 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:53.332549    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-92d8ce-l42r7" event={"ID":"f4be2238-df2d-4509-aceb-d3424645b91e","Type":"ContainerStarted","Data":"8564ba30be4e452c2a15b4b04834a764c99dbc7dbd7d1a87a8ac46cb7367a50b"}
Sep 11 23:24:54.341902 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:54.341804    2546 generic.go:356] "Generic (PLEG): container finished" podID="f4be2238-df2d-4509-aceb-d3424645b91e" containerID="87d03b6d0a71f7544c3365b189b5d64b03dfa36bacb025da177a6043b3e52600" exitCode=0
Sep 11 23:24:54.342425 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:54.341900    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-92d8ce-l42r7" event={"ID":"f4be2238-df2d-4509-aceb-d3424645b91e","Type":"ContainerDied","Data":"87d03b6d0a71f7544c3365b189b5d64b03dfa36bacb025da177a6043b3e52600"}
Sep 11 23:24:54.506935 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:54.506908    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-bfa694/test-model-bfa694-kserve-6949bf5cdf-xjwpl"]
Sep 11 23:24:55.524848 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:55.524822    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-92d8ce-l42r7"
Sep 11 23:24:55.626246 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:55.621235    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4be2238-df2d-4509-aceb-d3424645b91e-maas-api-service-ca\") pod \"f4be2238-df2d-4509-aceb-d3424645b91e\" (UID: \"f4be2238-df2d-4509-aceb-d3424645b91e\") "
Sep 11 23:24:55.626246 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:55.624694    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4be2238-df2d-4509-aceb-d3424645b91e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f4be2238-df2d-4509-aceb-d3424645b91e" (UID: "f4be2238-df2d-4509-aceb-d3424645b91e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 23:24:55.725879 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:55.725834    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4be2238-df2d-4509-aceb-d3424645b91e-maas-api-service-ca\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:24:56.355927 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:56.355893    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-92d8ce-l42r7"
Sep 11 23:24:56.356117 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:56.355928    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-92d8ce-l42r7" event={"ID":"f4be2238-df2d-4509-aceb-d3424645b91e","Type":"ContainerDied","Data":"8564ba30be4e452c2a15b4b04834a764c99dbc7dbd7d1a87a8ac46cb7367a50b"}
Sep 11 23:24:56.356117 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:56.356004    2546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8564ba30be4e452c2a15b4b04834a764c99dbc7dbd7d1a87a8ac46cb7367a50b"
Sep 11 23:24:56.357872 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:56.357847    2546 generic.go:356] "Generic (PLEG): container finished" podID="44b0b3b3-a51c-42bf-8539-f11924e397db" containerID="c9220478099430b9b95d3cc848e6d6b52138fa1f52c837570c992c51b53ee435" exitCode=0
Sep 11 23:24:56.357993 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:56.357897    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-97d9dc-openshift-default-7bfd65cf68-sltcx" event={"ID":"44b0b3b3-a51c-42bf-8539-f11924e397db","Type":"ContainerDied","Data":"c9220478099430b9b95d3cc848e6d6b52138fa1f52c837570c992c51b53ee435"}
Sep 11 23:24:56.358411 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:56.358393    2546 scope.go:117] "RemoveContainer" containerID="c9220478099430b9b95d3cc848e6d6b52138fa1f52c837570c992c51b53ee435"
Sep 11 23:24:56.359703 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:56.359673    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:24:56.359761 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:56.359748    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:24:56.359800 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:56.359783    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:24:57.367064 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:57.367030    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-97d9dc-openshift-default-7bfd65cf68-sltcx" event={"ID":"44b0b3b3-a51c-42bf-8539-f11924e397db","Type":"ContainerStarted","Data":"d6b7df72abe48166ace8a9ce28eb0b3c44e83156bbf1f3d5ead3ae1052840e7d"}
Sep 11 23:24:57.596048 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:57.595975    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-97d9dc-openshift-default-7bfd65cf68-sltcx"
Sep 11 23:24:57.596298 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:57.596255    2546 patch_prober.go:28] interesting pod/e2e-shared-b-w2-97d9dc-openshift-default-7bfd65cf68-sltcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Sep 11 23:24:57.596380 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:57.596308    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-97d9dc-openshift-default-7bfd65cf68-sltcx" podUID="44b0b3b3-a51c-42bf-8539-f11924e397db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Sep 11 23:24:58.596609 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:58.596571    2546 patch_prober.go:28] interesting pod/e2e-shared-b-w2-97d9dc-openshift-default-7bfd65cf68-sltcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Sep 11 23:24:58.597006 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:58.596634    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-97d9dc-openshift-default-7bfd65cf68-sltcx" podUID="44b0b3b3-a51c-42bf-8539-f11924e397db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Sep 11 23:24:59.596906 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:59.596872    2546 patch_prober.go:28] interesting pod/e2e-shared-b-w2-97d9dc-openshift-default-7bfd65cf68-sltcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Sep 11 23:24:59.597332 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:24:59.596925    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-97d9dc-openshift-default-7bfd65cf68-sltcx" podUID="44b0b3b3-a51c-42bf-8539-f11924e397db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Sep 11 23:25:00.497308 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:00.497273    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-b96fb0e1-openshift-default-df6bbcc4d-shm4v"]
Sep 11 23:25:00.497836 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:00.497817    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4be2238-df2d-4509-aceb-d3424645b91e" containerName="revoke-keys"
Sep 11 23:25:00.497836 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:00.497839    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4be2238-df2d-4509-aceb-d3424645b91e" containerName="revoke-keys"
Sep 11 23:25:00.498009 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:00.497997    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4be2238-df2d-4509-aceb-d3424645b91e" containerName="revoke-keys"
Sep 11 23:25:00.501571 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:00.501538    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b96fb0e1-openshift-default-df6bbcc4d-shm4v"
Sep 11 23:25:00.505110 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:00.505083    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-b96fb0e1-openshift-default-dockercfg-wljnt\""
Sep 11 23:25:00.556730 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:00.556224    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b96fb0e1-openshift-default-df6bbcc4d-shm4v"]
Sep 11 23:25:00.578014 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:00.577959    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c330e7f-955c-4e6f-b4c8-5b0dee024ba5-istio-data\") pod \"e2e-mt-b96fb0e1-openshift-default-df6bbcc4d-shm4v\" (UID: \"3c330e7f-955c-4e6f-b4c8-5b0dee024ba5\") " pod="openshift-ingress/e2e-mt-b96fb0e1-openshift-default-df6bbcc4d-shm4v"
Sep 11 23:25:00.578319 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:00.578295    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c330e7f-955c-4e6f-b4c8-5b0dee024ba5-credential-socket\") pod \"e2e-mt-b96fb0e1-openshift-default-df6bbcc4d-shm4v\" (UID: \"3c330e7f-955c-4e6f-b4c8-5b0dee024ba5\") " pod="openshift-ingress/e2e-mt-b96fb0e1-openshift-default-df6bbcc4d-shm4v"
Sep 11 23:25:00.578535 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:00.578516    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c330e7f-955c-4e6f-b4c8-5b0dee024ba5-istio-envoy\") pod \"e2e-mt-b96fb0e1-openshift-default-df6bbcc4d-shm4v\" (UID: \"3c330e7f-955c-4e6f-b4c8-5b0dee024ba5\") " pod="openshift-ingress/e2e-mt-b96fb0e1-openshift-default-df6bbcc4d-shm4v"
Sep 11 23:25:00.578708 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:00.578691    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-867j4\" (UniqueName: \"kubernetes.io/projected/3c330e7f-955c-4e6f-b4c8-5b0dee024ba5-kube-api-access-867j4\") pod \"e2e-mt-b96fb0e1-openshift-default-df6bbcc4d-shm4v\" (UID: \"3c330e7f-955c-4e6f-b4c8-5b0dee024ba5\") " pod="openshift-ingress/e2e-mt-b96fb0e1-openshift-default-df6bbcc4d-shm4v"
Sep 11 23:25:00.578900 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:00.578883    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c330e7f-955c-4e6f-b4c8-5b0dee024ba5-istiod-ca-cert\") pod \"e2e-mt-b96fb0e1-openshift-default-df6bbcc4d-shm4v\" (UID: \"3c330e7f-955c-4e6f-b4c8-5b0dee024ba5\") " pod="openshift-ingress/e2e-mt-b96fb0e1-openshift-default-df6bbcc4d-shm4v"
Sep 11 23:25:00.579084 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:00.579068    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c330e7f-955c-4e6f-b4c8-5b0dee024ba5-workload-certs\") pod \"e2e-mt-b96fb0e1-openshift-default-df6bbcc4d-shm4v\" (UID: \"3c330e7f-955c-4e6f-b4c8-5b0dee024ba5\") " pod="openshift-ingress/e2e-mt-b96fb0e1-openshift-default-df6bbcc4d-shm4v"
Sep 11 23:25:00.579369 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:00.579350    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c330e7f-955c-4e6f-b4c8-5b0dee024ba5-workload-socket\") pod \"e2e-mt-b96fb0e1-openshift-default-df6bbcc4d-shm4v\" (UID: \"3c330e7f-955c-4e6f-b4c8-5b0dee024ba5\") " pod="openshift-ingress/e2e-mt-b96fb0e1-openshift-default-df6bbcc4d-shm4v"
Sep 11 23:25:00.579548 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:00.579532    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c330e7f-955c-4e6f-b4c8-5b0dee024ba5-istio-token\") pod \"e2e-mt-b96fb0e1-openshift-default-df6bbcc4d-shm4v\" (UID: \"3c330e7f-955c-4e6f-b4c8-5b0dee024ba5\") " pod="openshift-ingress/e2e-mt-b96fb0e1-openshift-default-df6bbcc4d-shm4v"
Sep 11 23:25:00.579711 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:00.579694    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c330e7f-955c-4e6f-b4c8-5b0dee024ba5-istio-podinfo\") pod \"e2e-mt-b96fb0e1-openshift-default-df6bbcc4d-shm4v\" (UID: \"3c330e7f-955c-4e6f-b4c8-5b0dee024ba5\") " pod="openshift-ingress/e2e-mt-b96fb0e1-openshift-default-df6bbcc4d-shm4v"
Sep 11 23:25:00.597037 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:00.596998    2546 patch_prober.go:28] interesting pod/e2e-shared-b-w2-97d9dc-openshift-default-7bfd65cf68-sltcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Sep 11 23:25:00.597697 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:00.597660    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-97d9dc-openshift-default-7bfd65cf68-sltcx" podUID="44b0b3b3-a51c-42bf-8539-f11924e397db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Sep 11 23:25:00.681429 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:00.681386    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c330e7f-955c-4e6f-b4c8-5b0dee024ba5-workload-certs\") pod \"e2e-mt-b96fb0e1-openshift-default-df6bbcc4d-shm4v\" (UID: \"3c330e7f-955c-4e6f-b4c8-5b0dee024ba5\") " pod="openshift-ingress/e2e-mt-b96fb0e1-openshift-default-df6bbcc4d-shm4v"
Sep 11 23:25:00.681709 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:00.681690    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c330e7f-955c-4e6f-b4c8-5b0dee024ba5-workload-socket\") pod \"e2e-mt-b96fb0e1-openshift-default-df6bbcc4d-shm4v\" (UID: \"3c330e7f-955c-4e6f-b4c8-5b0dee024ba5\") " pod="openshift-ingress/e2e-mt-b96fb0e1-openshift-default-df6bbcc4d-shm4v"
Sep 11 23:25:00.682243 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:00.682222    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c330e7f-955c-4e6f-b4c8-5b0dee024ba5-istio-token\") pod \"e2e-mt-b96fb0e1-openshift-default-df6bbcc4d-shm4v\" (UID: \"3c330e7f-955c-4e6f-b4c8-5b0dee024ba5\") " pod="openshift-ingress/e2e-mt-b96fb0e1-openshift-default-df6bbcc4d-shm4v"
Sep 11 23:25:00.682498 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:00.682471    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c330e7f-955c-4e6f-b4c8-5b0dee024ba5-istio-podinfo\") pod \"e2e-mt-b96fb0e1-openshift-default-df6bbcc4d-shm4v\" (UID: \"3c330e7f-955c-4e6f-b4c8-5b0dee024ba5\") " pod="openshift-ingress/e2e-mt-b96fb0e1-openshift-default-df6bbcc4d-shm4v"
Sep 11 23:25:00.682591 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:00.681880    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c330e7f-955c-4e6f-b4c8-5b0dee024ba5-workload-certs\") pod \"e2e-mt-b96fb0e1-openshift-default-df6bbcc4d-shm4v\" (UID: \"3c330e7f-955c-4e6f-b4c8-5b0dee024ba5\") " pod="openshift-ingress/e2e-mt-b96fb0e1-openshift-default-df6bbcc4d-shm4v"
Sep 11 23:25:00.682591 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:00.682569    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c330e7f-955c-4e6f-b4c8-5b0dee024ba5-istio-data\") pod \"e2e-mt-b96fb0e1-openshift-default-df6bbcc4d-shm4v\" (UID: \"3c330e7f-955c-4e6f-b4c8-5b0dee024ba5\") " pod="openshift-ingress/e2e-mt-b96fb0e1-openshift-default-df6bbcc4d-shm4v"
Sep 11 23:25:00.682591 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:00.682118    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c330e7f-955c-4e6f-b4c8-5b0dee024ba5-workload-socket\") pod \"e2e-mt-b96fb0e1-openshift-default-df6bbcc4d-shm4v\" (UID: \"3c330e7f-955c-4e6f-b4c8-5b0dee024ba5\") " pod="openshift-ingress/e2e-mt-b96fb0e1-openshift-default-df6bbcc4d-shm4v"
Sep 11 23:25:00.682746 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:00.682619    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c330e7f-955c-4e6f-b4c8-5b0dee024ba5-credential-socket\") pod \"e2e-mt-b96fb0e1-openshift-default-df6bbcc4d-shm4v\" (UID: \"3c330e7f-955c-4e6f-b4c8-5b0dee024ba5\") " pod="openshift-ingress/e2e-mt-b96fb0e1-openshift-default-df6bbcc4d-shm4v"
Sep 11 23:25:00.682746 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:00.682671    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c330e7f-955c-4e6f-b4c8-5b0dee024ba5-istio-envoy\") pod \"e2e-mt-b96fb0e1-openshift-default-df6bbcc4d-shm4v\" (UID: \"3c330e7f-955c-4e6f-b4c8-5b0dee024ba5\") " pod="openshift-ingress/e2e-mt-b96fb0e1-openshift-default-df6bbcc4d-shm4v"
Sep 11 23:25:00.682746 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:00.682733    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-867j4\" (UniqueName: \"kubernetes.io/projected/3c330e7f-955c-4e6f-b4c8-5b0dee024ba5-kube-api-access-867j4\") pod \"e2e-mt-b96fb0e1-openshift-default-df6bbcc4d-shm4v\" (UID: \"3c330e7f-955c-4e6f-b4c8-5b0dee024ba5\") " pod="openshift-ingress/e2e-mt-b96fb0e1-openshift-default-df6bbcc4d-shm4v"
Sep 11 23:25:00.682901 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:00.682788    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c330e7f-955c-4e6f-b4c8-5b0dee024ba5-istiod-ca-cert\") pod \"e2e-mt-b96fb0e1-openshift-default-df6bbcc4d-shm4v\" (UID: \"3c330e7f-955c-4e6f-b4c8-5b0dee024ba5\") " pod="openshift-ingress/e2e-mt-b96fb0e1-openshift-default-df6bbcc4d-shm4v"
Sep 11 23:25:00.682901 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:00.682840    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c330e7f-955c-4e6f-b4c8-5b0dee024ba5-istio-data\") pod \"e2e-mt-b96fb0e1-openshift-default-df6bbcc4d-shm4v\" (UID: \"3c330e7f-955c-4e6f-b4c8-5b0dee024ba5\") " pod="openshift-ingress/e2e-mt-b96fb0e1-openshift-default-df6bbcc4d-shm4v"
Sep 11 23:25:00.683326 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:00.683309    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c330e7f-955c-4e6f-b4c8-5b0dee024ba5-credential-socket\") pod \"e2e-mt-b96fb0e1-openshift-default-df6bbcc4d-shm4v\" (UID: \"3c330e7f-955c-4e6f-b4c8-5b0dee024ba5\") " pod="openshift-ingress/e2e-mt-b96fb0e1-openshift-default-df6bbcc4d-shm4v"
Sep 11 23:25:00.683431 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:00.683342    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c330e7f-955c-4e6f-b4c8-5b0dee024ba5-istiod-ca-cert\") pod \"e2e-mt-b96fb0e1-openshift-default-df6bbcc4d-shm4v\" (UID: \"3c330e7f-955c-4e6f-b4c8-5b0dee024ba5\") " pod="openshift-ingress/e2e-mt-b96fb0e1-openshift-default-df6bbcc4d-shm4v"
Sep 11 23:25:00.685106 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:00.685088    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c330e7f-955c-4e6f-b4c8-5b0dee024ba5-istio-podinfo\") pod \"e2e-mt-b96fb0e1-openshift-default-df6bbcc4d-shm4v\" (UID: \"3c330e7f-955c-4e6f-b4c8-5b0dee024ba5\") " pod="openshift-ingress/e2e-mt-b96fb0e1-openshift-default-df6bbcc4d-shm4v"
Sep 11 23:25:00.685181 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:00.685111    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c330e7f-955c-4e6f-b4c8-5b0dee024ba5-istio-envoy\") pod \"e2e-mt-b96fb0e1-openshift-default-df6bbcc4d-shm4v\" (UID: \"3c330e7f-955c-4e6f-b4c8-5b0dee024ba5\") " pod="openshift-ingress/e2e-mt-b96fb0e1-openshift-default-df6bbcc4d-shm4v"
Sep 11 23:25:00.692897 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:00.692857    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-867j4\" (UniqueName: \"kubernetes.io/projected/3c330e7f-955c-4e6f-b4c8-5b0dee024ba5-kube-api-access-867j4\") pod \"e2e-mt-b96fb0e1-openshift-default-df6bbcc4d-shm4v\" (UID: \"3c330e7f-955c-4e6f-b4c8-5b0dee024ba5\") " pod="openshift-ingress/e2e-mt-b96fb0e1-openshift-default-df6bbcc4d-shm4v"
Sep 11 23:25:00.692897 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:00.692876    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c330e7f-955c-4e6f-b4c8-5b0dee024ba5-istio-token\") pod \"e2e-mt-b96fb0e1-openshift-default-df6bbcc4d-shm4v\" (UID: \"3c330e7f-955c-4e6f-b4c8-5b0dee024ba5\") " pod="openshift-ingress/e2e-mt-b96fb0e1-openshift-default-df6bbcc4d-shm4v"
Sep 11 23:25:00.813316 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:00.813228    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b96fb0e1-openshift-default-df6bbcc4d-shm4v"
Sep 11 23:25:01.112648 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:01.112619    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b96fb0e1-openshift-default-df6bbcc4d-shm4v"]
Sep 11 23:25:01.113467 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:25:01.113441    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c330e7f_955c_4e6f_b4c8_5b0dee024ba5.slice/crio-04fe5627a8e7a7eec4cbd2942646239ac840a587e8041802aec7338f051d1848 WatchSource:0}: Error finding container 04fe5627a8e7a7eec4cbd2942646239ac840a587e8041802aec7338f051d1848: Status 404 returned error can't find the container with id 04fe5627a8e7a7eec4cbd2942646239ac840a587e8041802aec7338f051d1848
Sep 11 23:25:01.116212 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:01.116179    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:25:01.116321 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:01.116258    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:25:01.116321 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:01.116302    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:25:01.397445 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:01.397337    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b96fb0e1-openshift-default-df6bbcc4d-shm4v" event={"ID":"3c330e7f-955c-4e6f-b4c8-5b0dee024ba5","Type":"ContainerStarted","Data":"a1bf2018af1bc1ccbdac0ba0ca08734bd30b10cc2303dfcff0520ed9be0cabef"}
Sep 11 23:25:01.397445 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:01.397394    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b96fb0e1-openshift-default-df6bbcc4d-shm4v" event={"ID":"3c330e7f-955c-4e6f-b4c8-5b0dee024ba5","Type":"ContainerStarted","Data":"04fe5627a8e7a7eec4cbd2942646239ac840a587e8041802aec7338f051d1848"}
Sep 11 23:25:01.422466 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:01.422410    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-b96fb0e1-openshift-default-df6bbcc4d-shm4v" podStartSLOduration=1.422394671 podStartE2EDuration="1.422394671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 23:25:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:25:01.420296424 +0000 UTC m=+1425.453425639" watchObservedRunningTime="2026-09-11 23:25:01.422394671 +0000 UTC m=+1425.455523885"
Sep 11 23:25:01.596863 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:01.596824    2546 patch_prober.go:28] interesting pod/e2e-shared-b-w2-97d9dc-openshift-default-7bfd65cf68-sltcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Sep 11 23:25:01.597053 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:01.596879    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-97d9dc-openshift-default-7bfd65cf68-sltcx" podUID="44b0b3b3-a51c-42bf-8539-f11924e397db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Sep 11 23:25:01.813846 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:01.813810    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b96fb0e1-openshift-default-df6bbcc4d-shm4v"
Sep 11 23:25:02.600628 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:02.600600    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-97d9dc-openshift-default-7bfd65cf68-sltcx"
Sep 11 23:25:02.601100 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:02.601076    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-97d9dc-openshift-default-7bfd65cf68-sltcx"
Sep 11 23:25:02.601807 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:02.601788    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-97d9dc-openshift-default-7bfd65cf68-sltcx"
Sep 11 23:25:02.818835 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:02.818805    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b96fb0e1-openshift-default-df6bbcc4d-shm4v"
Sep 11 23:25:03.411057 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:03.411018    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b96fb0e1-openshift-default-df6bbcc4d-shm4v"
Sep 11 23:25:03.411891 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:03.411872    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b96fb0e1-openshift-default-df6bbcc4d-shm4v"
Sep 11 23:25:05.190429 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:05.190383    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"]
Sep 11 23:25:09.527538 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:09.527449    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-bfa694/test-model-bfa694-kserve-6949bf5cdf-xjwpl" podUID="2abb209f-6719-4bbb-acc5-3e9a1d9b2f94" containerName="main" containerID="cri-o://3db1d9c21696faccf6528f7153118955d82c36574446ee3040c199fc55c7851d" gracePeriod=45
Sep 11 23:25:09.582874 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:09.582838    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-92d8ce-l42r7"]
Sep 11 23:25:09.585619 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:09.585595    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-92d8ce-l42r7"]
Sep 11 23:25:09.843895 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:09.843862    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-bfa694/test-model-bfa694-kserve-6949bf5cdf-xjwpl"
Sep 11 23:25:09.892453 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:09.892424    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2abb209f-6719-4bbb-acc5-3e9a1d9b2f94-tls-certs\") pod \"2abb209f-6719-4bbb-acc5-3e9a1d9b2f94\" (UID: \"2abb209f-6719-4bbb-acc5-3e9a1d9b2f94\") "
Sep 11 23:25:09.892635 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:09.892468    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2abb209f-6719-4bbb-acc5-3e9a1d9b2f94-dshm\") pod \"2abb209f-6719-4bbb-acc5-3e9a1d9b2f94\" (UID: \"2abb209f-6719-4bbb-acc5-3e9a1d9b2f94\") "
Sep 11 23:25:09.892635 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:09.892488    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2abb209f-6719-4bbb-acc5-3e9a1d9b2f94-home\") pod \"2abb209f-6719-4bbb-acc5-3e9a1d9b2f94\" (UID: \"2abb209f-6719-4bbb-acc5-3e9a1d9b2f94\") "
Sep 11 23:25:09.892635 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:09.892615    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2abb209f-6719-4bbb-acc5-3e9a1d9b2f94-model-cache\") pod \"2abb209f-6719-4bbb-acc5-3e9a1d9b2f94\" (UID: \"2abb209f-6719-4bbb-acc5-3e9a1d9b2f94\") "
Sep 11 23:25:09.892789 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:09.892663    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2abb209f-6719-4bbb-acc5-3e9a1d9b2f94-tmp-dir\") pod \"2abb209f-6719-4bbb-acc5-3e9a1d9b2f94\" (UID: \"2abb209f-6719-4bbb-acc5-3e9a1d9b2f94\") "
Sep 11 23:25:09.892789 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:09.892770    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2abb209f-6719-4bbb-acc5-3e9a1d9b2f94-home" (OuterVolumeSpecName: "home") pod "2abb209f-6719-4bbb-acc5-3e9a1d9b2f94" (UID: "2abb209f-6719-4bbb-acc5-3e9a1d9b2f94"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:25:09.892898 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:09.892803    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lwvz8\" (UniqueName: \"kubernetes.io/projected/2abb209f-6719-4bbb-acc5-3e9a1d9b2f94-kube-api-access-lwvz8\") pod \"2abb209f-6719-4bbb-acc5-3e9a1d9b2f94\" (UID: \"2abb209f-6719-4bbb-acc5-3e9a1d9b2f94\") "
Sep 11 23:25:09.892950 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:09.892910    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2abb209f-6719-4bbb-acc5-3e9a1d9b2f94-model-cache" (OuterVolumeSpecName: "model-cache") pod "2abb209f-6719-4bbb-acc5-3e9a1d9b2f94" (UID: "2abb209f-6719-4bbb-acc5-3e9a1d9b2f94"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:25:09.893026 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:09.892991    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2abb209f-6719-4bbb-acc5-3e9a1d9b2f94-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2abb209f-6719-4bbb-acc5-3e9a1d9b2f94" (UID: "2abb209f-6719-4bbb-acc5-3e9a1d9b2f94"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:25:09.893798 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:09.893771    2546 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2abb209f-6719-4bbb-acc5-3e9a1d9b2f94-model-cache\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:25:09.893798 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:09.893801    2546 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2abb209f-6719-4bbb-acc5-3e9a1d9b2f94-tmp-dir\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:25:09.893798 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:09.893815    2546 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2abb209f-6719-4bbb-acc5-3e9a1d9b2f94-home\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:25:09.894966 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:09.894939    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2abb209f-6719-4bbb-acc5-3e9a1d9b2f94-dshm" (OuterVolumeSpecName: "dshm") pod "2abb209f-6719-4bbb-acc5-3e9a1d9b2f94" (UID: "2abb209f-6719-4bbb-acc5-3e9a1d9b2f94"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:25:09.895045 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:09.894984    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2abb209f-6719-4bbb-acc5-3e9a1d9b2f94-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2abb209f-6719-4bbb-acc5-3e9a1d9b2f94" (UID: "2abb209f-6719-4bbb-acc5-3e9a1d9b2f94"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 23:25:09.895092 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:09.895071    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2abb209f-6719-4bbb-acc5-3e9a1d9b2f94-kube-api-access-lwvz8" (OuterVolumeSpecName: "kube-api-access-lwvz8") pod "2abb209f-6719-4bbb-acc5-3e9a1d9b2f94" (UID: "2abb209f-6719-4bbb-acc5-3e9a1d9b2f94"). InnerVolumeSpecName "kube-api-access-lwvz8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 23:25:09.995473 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:09.995428    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwvz8\" (UniqueName: \"kubernetes.io/projected/2abb209f-6719-4bbb-acc5-3e9a1d9b2f94-kube-api-access-lwvz8\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:25:09.995473 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:09.995462    2546 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2abb209f-6719-4bbb-acc5-3e9a1d9b2f94-tls-certs\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:25:09.995473 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:09.995471    2546 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2abb209f-6719-4bbb-acc5-3e9a1d9b2f94-dshm\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:25:10.473693 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:10.470122    2546 generic.go:356] "Generic (PLEG): container finished" podID="2abb209f-6719-4bbb-acc5-3e9a1d9b2f94" containerID="3db1d9c21696faccf6528f7153118955d82c36574446ee3040c199fc55c7851d" exitCode=0
Sep 11 23:25:10.473693 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:10.471887    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-bfa694/test-model-bfa694-kserve-6949bf5cdf-xjwpl"
Sep 11 23:25:10.473693 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:10.471892    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-bfa694/test-model-bfa694-kserve-6949bf5cdf-xjwpl" event={"ID":"2abb209f-6719-4bbb-acc5-3e9a1d9b2f94","Type":"ContainerDied","Data":"3db1d9c21696faccf6528f7153118955d82c36574446ee3040c199fc55c7851d"}
Sep 11 23:25:10.473693 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:10.472417    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-bfa694/test-model-bfa694-kserve-6949bf5cdf-xjwpl" event={"ID":"2abb209f-6719-4bbb-acc5-3e9a1d9b2f94","Type":"ContainerDied","Data":"4a248cb6e2f1792507d9236e7515f5105a0fa5f72109797f2781f35c4f0cd707"}
Sep 11 23:25:10.473693 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:10.472453    2546 scope.go:117] "RemoveContainer" containerID="3db1d9c21696faccf6528f7153118955d82c36574446ee3040c199fc55c7851d"
Sep 11 23:25:10.492692 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:10.491936    2546 scope.go:117] "RemoveContainer" containerID="3db1d9c21696faccf6528f7153118955d82c36574446ee3040c199fc55c7851d"
Sep 11 23:25:10.492937 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:25:10.492894    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3db1d9c21696faccf6528f7153118955d82c36574446ee3040c199fc55c7851d\": container with ID starting with 3db1d9c21696faccf6528f7153118955d82c36574446ee3040c199fc55c7851d not found: ID does not exist" containerID="3db1d9c21696faccf6528f7153118955d82c36574446ee3040c199fc55c7851d"
Sep 11 23:25:10.493049 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:10.492935    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3db1d9c21696faccf6528f7153118955d82c36574446ee3040c199fc55c7851d"} err="failed to get container status \"3db1d9c21696faccf6528f7153118955d82c36574446ee3040c199fc55c7851d\": rpc error: code = NotFound desc = could not find container \"3db1d9c21696faccf6528f7153118955d82c36574446ee3040c199fc55c7851d\": container with ID starting with 3db1d9c21696faccf6528f7153118955d82c36574446ee3040c199fc55c7851d not found: ID does not exist"
Sep 11 23:25:10.497729 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:10.497707    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4be2238-df2d-4509-aceb-d3424645b91e" path="/var/lib/kubelet/pods/f4be2238-df2d-4509-aceb-d3424645b91e/volumes"
Sep 11 23:25:10.506099 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:10.506050    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-bfa694/test-model-bfa694-kserve-6949bf5cdf-xjwpl"]
Sep 11 23:25:10.508870 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:10.508852    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-bfa694/test-model-bfa694-kserve-6949bf5cdf-xjwpl"]
Sep 11 23:25:12.495139 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:12.495105    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2abb209f-6719-4bbb-acc5-3e9a1d9b2f94" path="/var/lib/kubelet/pods/2abb209f-6719-4bbb-acc5-3e9a1d9b2f94/volumes"
Sep 11 23:25:16.281133 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:16.281089    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-92d8ce-openshift-default-5ff8db57b8-8xvzw"]
Sep 11 23:25:16.281666 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:16.281375    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-92d8ce-openshift-default-5ff8db57b8-8xvzw" podUID="2304922d-c8d4-4a95-a81c-4af2ccea36b0" containerName="istio-proxy" containerID="cri-o://fbf830e5fb9f4b6ad31034e7f9139fe54af5726c6b7643246f5e6026041dc61e" gracePeriod=30
Sep 11 23:25:16.600333 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:16.600255    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bfa694-mpv86"]
Sep 11 23:25:16.600720 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:16.600708    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2abb209f-6719-4bbb-acc5-3e9a1d9b2f94" containerName="main"
Sep 11 23:25:16.600771 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:16.600722    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="2abb209f-6719-4bbb-acc5-3e9a1d9b2f94" containerName="main"
Sep 11 23:25:16.600826 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:16.600818    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="2abb209f-6719-4bbb-acc5-3e9a1d9b2f94" containerName="main"
Sep 11 23:25:16.604115 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:16.604097    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bfa694-mpv86"
Sep 11 23:25:16.607168 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:16.607128    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vrx7m\""
Sep 11 23:25:16.607294 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:16.607128    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 23:25:16.611925 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:16.611899    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bfa694-mpv86"]
Sep 11 23:25:16.618923 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:16.618899    2546 scope.go:117] "RemoveContainer" containerID="92772103cb54aece30b906c43c328c7d9fbb1fc9e7c89f135812b0f9dd97b23d"
Sep 11 23:25:16.630052 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:16.630028    2546 scope.go:117] "RemoveContainer" containerID="c585325c822bf8d156c8b260d471f8699b35c8841bd94930370b00eac40461ac"
Sep 11 23:25:16.780261 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:16.780224    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd62c1ac-0a15-4ad2-82b3-fd73eb6e247f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-bfa694-mpv86\" (UID: \"bd62c1ac-0a15-4ad2-82b3-fd73eb6e247f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bfa694-mpv86"
Sep 11 23:25:16.881015 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:16.880909    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd62c1ac-0a15-4ad2-82b3-fd73eb6e247f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-bfa694-mpv86\" (UID: \"bd62c1ac-0a15-4ad2-82b3-fd73eb6e247f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bfa694-mpv86"
Sep 11 23:25:16.881646 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:16.881615    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd62c1ac-0a15-4ad2-82b3-fd73eb6e247f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-bfa694-mpv86\" (UID: \"bd62c1ac-0a15-4ad2-82b3-fd73eb6e247f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bfa694-mpv86"
Sep 11 23:25:16.916333 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:16.916296    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bfa694-mpv86"
Sep 11 23:25:17.217222 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:17.217192    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bfa694-mpv86"]
Sep 11 23:25:17.217691 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:25:17.217669    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd62c1ac_0a15_4ad2_82b3_fd73eb6e247f.slice/crio-4d7cfdb9249ee6c3c188d91b0dba4c13a4e492228546d0f34b7448422582f594 WatchSource:0}: Error finding container 4d7cfdb9249ee6c3c188d91b0dba4c13a4e492228546d0f34b7448422582f594: Status 404 returned error can't find the container with id 4d7cfdb9249ee6c3c188d91b0dba4c13a4e492228546d0f34b7448422582f594
Sep 11 23:25:17.539143 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:17.539111    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bfa694-mpv86" event={"ID":"bd62c1ac-0a15-4ad2-82b3-fd73eb6e247f","Type":"ContainerStarted","Data":"4d7cfdb9249ee6c3c188d91b0dba4c13a4e492228546d0f34b7448422582f594"}
Sep 11 23:25:18.548036 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:18.547993    2546 generic.go:356] "Generic (PLEG): container finished" podID="bd62c1ac-0a15-4ad2-82b3-fd73eb6e247f" containerID="65c29795831866cf799de0411ba7e8ad6517b2a1e2a46eab790fd720b4639101" exitCode=0
Sep 11 23:25:18.548450 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:18.548052    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bfa694-mpv86" event={"ID":"bd62c1ac-0a15-4ad2-82b3-fd73eb6e247f","Type":"ContainerDied","Data":"65c29795831866cf799de0411ba7e8ad6517b2a1e2a46eab790fd720b4639101"}
Sep 11 23:25:19.698716 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:19.698691    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bfa694-mpv86"
Sep 11 23:25:19.811134 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:19.811039    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd62c1ac-0a15-4ad2-82b3-fd73eb6e247f-maas-api-service-ca\") pod \"bd62c1ac-0a15-4ad2-82b3-fd73eb6e247f\" (UID: \"bd62c1ac-0a15-4ad2-82b3-fd73eb6e247f\") "
Sep 11 23:25:19.811458 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:19.811434    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd62c1ac-0a15-4ad2-82b3-fd73eb6e247f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bd62c1ac-0a15-4ad2-82b3-fd73eb6e247f" (UID: "bd62c1ac-0a15-4ad2-82b3-fd73eb6e247f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 23:25:19.912754 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:19.912719    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd62c1ac-0a15-4ad2-82b3-fd73eb6e247f-maas-api-service-ca\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:25:20.568650 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:20.568592    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bfa694-mpv86" event={"ID":"bd62c1ac-0a15-4ad2-82b3-fd73eb6e247f","Type":"ContainerDied","Data":"4d7cfdb9249ee6c3c188d91b0dba4c13a4e492228546d0f34b7448422582f594"}
Sep 11 23:25:20.568825 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:20.568666    2546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d7cfdb9249ee6c3c188d91b0dba4c13a4e492228546d0f34b7448422582f594"
Sep 11 23:25:20.568825 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:20.568738    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bfa694-mpv86"
Sep 11 23:25:21.577246 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:21.577219    2546 generic.go:356] "Generic (PLEG): container finished" podID="2304922d-c8d4-4a95-a81c-4af2ccea36b0" containerID="fbf830e5fb9f4b6ad31034e7f9139fe54af5726c6b7643246f5e6026041dc61e" exitCode=0
Sep 11 23:25:21.577568 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:21.577300    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-92d8ce-openshift-default-5ff8db57b8-8xvzw" event={"ID":"2304922d-c8d4-4a95-a81c-4af2ccea36b0","Type":"ContainerDied","Data":"fbf830e5fb9f4b6ad31034e7f9139fe54af5726c6b7643246f5e6026041dc61e"}
Sep 11 23:25:21.577568 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:21.577372    2546 scope.go:117] "RemoveContainer" containerID="bf23cb14ab75188574c3d578b03c4980958fb1aa6d7f7abf440d6a2f07f1bcec"
Sep 11 23:25:21.602355 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:21.602331    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-92d8ce-openshift-default-5ff8db57b8-8xvzw"
Sep 11 23:25:21.755278 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:21.755245    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2304922d-c8d4-4a95-a81c-4af2ccea36b0-istio-data\") pod \"2304922d-c8d4-4a95-a81c-4af2ccea36b0\" (UID: \"2304922d-c8d4-4a95-a81c-4af2ccea36b0\") "
Sep 11 23:25:21.755487 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:21.755312    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2304922d-c8d4-4a95-a81c-4af2ccea36b0-credential-socket\") pod \"2304922d-c8d4-4a95-a81c-4af2ccea36b0\" (UID: \"2304922d-c8d4-4a95-a81c-4af2ccea36b0\") "
Sep 11 23:25:21.755487 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:21.755359    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2304922d-c8d4-4a95-a81c-4af2ccea36b0-istio-envoy\") pod \"2304922d-c8d4-4a95-a81c-4af2ccea36b0\" (UID: \"2304922d-c8d4-4a95-a81c-4af2ccea36b0\") "
Sep 11 23:25:21.755487 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:21.755383    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2304922d-c8d4-4a95-a81c-4af2ccea36b0-workload-certs\") pod \"2304922d-c8d4-4a95-a81c-4af2ccea36b0\" (UID: \"2304922d-c8d4-4a95-a81c-4af2ccea36b0\") "
Sep 11 23:25:21.755487 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:21.755427    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-46bbf\" (UniqueName: \"kubernetes.io/projected/2304922d-c8d4-4a95-a81c-4af2ccea36b0-kube-api-access-46bbf\") pod \"2304922d-c8d4-4a95-a81c-4af2ccea36b0\" (UID: \"2304922d-c8d4-4a95-a81c-4af2ccea36b0\") "
Sep 11 23:25:21.755487 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:21.755483    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2304922d-c8d4-4a95-a81c-4af2ccea36b0-istio-podinfo\") pod \"2304922d-c8d4-4a95-a81c-4af2ccea36b0\" (UID: \"2304922d-c8d4-4a95-a81c-4af2ccea36b0\") "
Sep 11 23:25:21.755760 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:21.755509    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2304922d-c8d4-4a95-a81c-4af2ccea36b0-workload-socket\") pod \"2304922d-c8d4-4a95-a81c-4af2ccea36b0\" (UID: \"2304922d-c8d4-4a95-a81c-4af2ccea36b0\") "
Sep 11 23:25:21.755760 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:21.755542    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2304922d-c8d4-4a95-a81c-4af2ccea36b0-istio-data" (OuterVolumeSpecName: "istio-data") pod "2304922d-c8d4-4a95-a81c-4af2ccea36b0" (UID: "2304922d-c8d4-4a95-a81c-4af2ccea36b0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:25:21.755760 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:21.755548    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2304922d-c8d4-4a95-a81c-4af2ccea36b0-istio-token\") pod \"2304922d-c8d4-4a95-a81c-4af2ccea36b0\" (UID: \"2304922d-c8d4-4a95-a81c-4af2ccea36b0\") "
Sep 11 23:25:21.755760 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:21.755586    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2304922d-c8d4-4a95-a81c-4af2ccea36b0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2304922d-c8d4-4a95-a81c-4af2ccea36b0" (UID: "2304922d-c8d4-4a95-a81c-4af2ccea36b0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:25:21.755760 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:21.755613    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2304922d-c8d4-4a95-a81c-4af2ccea36b0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2304922d-c8d4-4a95-a81c-4af2ccea36b0" (UID: "2304922d-c8d4-4a95-a81c-4af2ccea36b0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:25:21.755760 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:21.755707    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2304922d-c8d4-4a95-a81c-4af2ccea36b0-istiod-ca-cert\") pod \"2304922d-c8d4-4a95-a81c-4af2ccea36b0\" (UID: \"2304922d-c8d4-4a95-a81c-4af2ccea36b0\") "
Sep 11 23:25:21.756071 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:21.755938    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2304922d-c8d4-4a95-a81c-4af2ccea36b0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2304922d-c8d4-4a95-a81c-4af2ccea36b0" (UID: "2304922d-c8d4-4a95-a81c-4af2ccea36b0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:25:21.756071 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:21.756054    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2304922d-c8d4-4a95-a81c-4af2ccea36b0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2304922d-c8d4-4a95-a81c-4af2ccea36b0" (UID: "2304922d-c8d4-4a95-a81c-4af2ccea36b0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 23:25:21.756401 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:21.756356    2546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2304922d-c8d4-4a95-a81c-4af2ccea36b0-workload-socket\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:25:21.756456 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:21.756407    2546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2304922d-c8d4-4a95-a81c-4af2ccea36b0-istiod-ca-cert\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:25:21.756456 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:21.756421    2546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2304922d-c8d4-4a95-a81c-4af2ccea36b0-istio-data\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:25:21.756456 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:21.756433    2546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2304922d-c8d4-4a95-a81c-4af2ccea36b0-credential-socket\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:25:21.756456 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:21.756444    2546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2304922d-c8d4-4a95-a81c-4af2ccea36b0-workload-certs\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:25:21.757886 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:21.757855    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2304922d-c8d4-4a95-a81c-4af2ccea36b0-istio-token" (OuterVolumeSpecName: "istio-token") pod "2304922d-c8d4-4a95-a81c-4af2ccea36b0" (UID: "2304922d-c8d4-4a95-a81c-4af2ccea36b0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 23:25:21.757991 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:21.757904    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2304922d-c8d4-4a95-a81c-4af2ccea36b0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2304922d-c8d4-4a95-a81c-4af2ccea36b0" (UID: "2304922d-c8d4-4a95-a81c-4af2ccea36b0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:25:21.757991 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:21.757939    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2304922d-c8d4-4a95-a81c-4af2ccea36b0-kube-api-access-46bbf" (OuterVolumeSpecName: "kube-api-access-46bbf") pod "2304922d-c8d4-4a95-a81c-4af2ccea36b0" (UID: "2304922d-c8d4-4a95-a81c-4af2ccea36b0"). InnerVolumeSpecName "kube-api-access-46bbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 23:25:21.757991 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:21.757971    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2304922d-c8d4-4a95-a81c-4af2ccea36b0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2304922d-c8d4-4a95-a81c-4af2ccea36b0" (UID: "2304922d-c8d4-4a95-a81c-4af2ccea36b0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 23:25:21.857644 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:21.857549    2546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2304922d-c8d4-4a95-a81c-4af2ccea36b0-istio-podinfo\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:25:21.857644 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:21.857583    2546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2304922d-c8d4-4a95-a81c-4af2ccea36b0-istio-token\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:25:21.857644 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:21.857593    2546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2304922d-c8d4-4a95-a81c-4af2ccea36b0-istio-envoy\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:25:21.857644 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:21.857601    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46bbf\" (UniqueName: \"kubernetes.io/projected/2304922d-c8d4-4a95-a81c-4af2ccea36b0-kube-api-access-46bbf\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:25:22.585881 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:22.585845    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-92d8ce-openshift-default-5ff8db57b8-8xvzw"
Sep 11 23:25:22.586351 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:22.585903    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-92d8ce-openshift-default-5ff8db57b8-8xvzw" event={"ID":"2304922d-c8d4-4a95-a81c-4af2ccea36b0","Type":"ContainerDied","Data":"c09cd6895a18a7914d1752a70535cabe1e2f3ec13f70b3d8bffe669a4b2d2904"}
Sep 11 23:25:22.586351 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:22.585959    2546 scope.go:117] "RemoveContainer" containerID="fbf830e5fb9f4b6ad31034e7f9139fe54af5726c6b7643246f5e6026041dc61e"
Sep 11 23:25:22.632852 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:22.632807    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-92d8ce-openshift-default-5ff8db57b8-8xvzw"]
Sep 11 23:25:22.650837 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:22.650790    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-92d8ce-openshift-default-5ff8db57b8-8xvzw"]
Sep 11 23:25:24.304227 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:24.304188    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-c0d4bf-openshift-default-948f987d-4kbcq"]
Sep 11 23:25:24.305115 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:24.305079    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-c0d4bf-openshift-default-948f987d-4kbcq" podUID="f621a2e1-7039-44a2-aa9a-a66e3e5e4e25" containerName="istio-proxy" containerID="cri-o://06d4646b62b10d2abdf17449008d5067e24eb56d9735cd3a672cceed3ef30230" gracePeriod=30
Sep 11 23:25:24.502803 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:24.502743    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2304922d-c8d4-4a95-a81c-4af2ccea36b0" path="/var/lib/kubelet/pods/2304922d-c8d4-4a95-a81c-4af2ccea36b0/volumes"
Sep 11 23:25:26.704726 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:26.704634    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-bfa694-7679c8678b-zzpqs"]
Sep 11 23:25:26.705259 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:26.704896    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bfa694-7679c8678b-zzpqs" podUID="fe27a86b-83cf-4afa-a295-f832ded073a0" containerName="payload-pre-processing" containerID="cri-o://4d716905040f2fb8536590a7e9f7a09405ef88ad3372085c7b0f58a0cc304a5c" gracePeriod=30
Sep 11 23:25:26.866757 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:26.866723    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bfa694-mpv86"]
Sep 11 23:25:26.869640 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:26.869610    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bfa694-mpv86"]
Sep 11 23:25:26.985951 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:26.985921    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bfa694-7679c8678b-zzpqs"
Sep 11 23:25:27.025320 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:27.025285    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dq9vk\" (UniqueName: \"kubernetes.io/projected/fe27a86b-83cf-4afa-a295-f832ded073a0-kube-api-access-dq9vk\") pod \"fe27a86b-83cf-4afa-a295-f832ded073a0\" (UID: \"fe27a86b-83cf-4afa-a295-f832ded073a0\") "
Sep 11 23:25:27.025534 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:27.025464    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe27a86b-83cf-4afa-a295-f832ded073a0-plugins-config-volume\") pod \"fe27a86b-83cf-4afa-a295-f832ded073a0\" (UID: \"fe27a86b-83cf-4afa-a295-f832ded073a0\") "
Sep 11 23:25:27.026232 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:27.026201    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe27a86b-83cf-4afa-a295-f832ded073a0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fe27a86b-83cf-4afa-a295-f832ded073a0" (UID: "fe27a86b-83cf-4afa-a295-f832ded073a0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 23:25:27.028501 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:27.028474    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe27a86b-83cf-4afa-a295-f832ded073a0-kube-api-access-dq9vk" (OuterVolumeSpecName: "kube-api-access-dq9vk") pod "fe27a86b-83cf-4afa-a295-f832ded073a0" (UID: "fe27a86b-83cf-4afa-a295-f832ded073a0"). InnerVolumeSpecName "kube-api-access-dq9vk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 23:25:27.127320 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:27.127284    2546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe27a86b-83cf-4afa-a295-f832ded073a0-plugins-config-volume\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:25:27.127320 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:27.127319    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dq9vk\" (UniqueName: \"kubernetes.io/projected/fe27a86b-83cf-4afa-a295-f832ded073a0-kube-api-access-dq9vk\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:25:27.623891 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:27.623848    2546 generic.go:356] "Generic (PLEG): container finished" podID="fe27a86b-83cf-4afa-a295-f832ded073a0" containerID="4d716905040f2fb8536590a7e9f7a09405ef88ad3372085c7b0f58a0cc304a5c" exitCode=0
Sep 11 23:25:27.624101 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:27.623905    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bfa694-7679c8678b-zzpqs"
Sep 11 23:25:27.624101 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:27.623931    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bfa694-7679c8678b-zzpqs" event={"ID":"fe27a86b-83cf-4afa-a295-f832ded073a0","Type":"ContainerDied","Data":"4d716905040f2fb8536590a7e9f7a09405ef88ad3372085c7b0f58a0cc304a5c"}
Sep 11 23:25:27.624101 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:27.623998    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bfa694-7679c8678b-zzpqs" event={"ID":"fe27a86b-83cf-4afa-a295-f832ded073a0","Type":"ContainerDied","Data":"b2019883c85f75e719ee0725939439122f4c41879369ba4e712e54109cb856a3"}
Sep 11 23:25:27.624101 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:27.624029    2546 scope.go:117] "RemoveContainer" containerID="4d716905040f2fb8536590a7e9f7a09405ef88ad3372085c7b0f58a0cc304a5c"
Sep 11 23:25:27.636721 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:27.636698    2546 scope.go:117] "RemoveContainer" containerID="4d716905040f2fb8536590a7e9f7a09405ef88ad3372085c7b0f58a0cc304a5c"
Sep 11 23:25:27.637102 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:25:27.637083    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d716905040f2fb8536590a7e9f7a09405ef88ad3372085c7b0f58a0cc304a5c\": container with ID starting with 4d716905040f2fb8536590a7e9f7a09405ef88ad3372085c7b0f58a0cc304a5c not found: ID does not exist" containerID="4d716905040f2fb8536590a7e9f7a09405ef88ad3372085c7b0f58a0cc304a5c"
Sep 11 23:25:27.637205 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:27.637107    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d716905040f2fb8536590a7e9f7a09405ef88ad3372085c7b0f58a0cc304a5c"} err="failed to get container status \"4d716905040f2fb8536590a7e9f7a09405ef88ad3372085c7b0f58a0cc304a5c\": rpc error: code = NotFound desc = could not find container \"4d716905040f2fb8536590a7e9f7a09405ef88ad3372085c7b0f58a0cc304a5c\": container with ID starting with 4d716905040f2fb8536590a7e9f7a09405ef88ad3372085c7b0f58a0cc304a5c not found: ID does not exist"
Sep 11 23:25:27.650829 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:27.650797    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-bfa694-7679c8678b-zzpqs"]
Sep 11 23:25:27.655007 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:27.654976    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-bfa694-7679c8678b-zzpqs"]
Sep 11 23:25:27.813888 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:27.813851    2546 patch_prober.go:28] interesting pod/e2e-noten-c0d4bf-openshift-default-948f987d-4kbcq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 11 23:25:27.814300 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:27.813924    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-noten-c0d4bf-openshift-default-948f987d-4kbcq" podUID="f621a2e1-7039-44a2-aa9a-a66e3e5e4e25" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 23:25:28.494928 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:28.494892    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd62c1ac-0a15-4ad2-82b3-fd73eb6e247f" path="/var/lib/kubelet/pods/bd62c1ac-0a15-4ad2-82b3-fd73eb6e247f/volumes"
Sep 11 23:25:28.495291 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:28.495277    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe27a86b-83cf-4afa-a295-f832ded073a0" path="/var/lib/kubelet/pods/fe27a86b-83cf-4afa-a295-f832ded073a0/volumes"
Sep 11 23:25:29.611984 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:29.611956    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-c0d4bf-openshift-default-948f987d-4kbcq"
Sep 11 23:25:29.645031 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:29.644983    2546 generic.go:356] "Generic (PLEG): container finished" podID="f621a2e1-7039-44a2-aa9a-a66e3e5e4e25" containerID="06d4646b62b10d2abdf17449008d5067e24eb56d9735cd3a672cceed3ef30230" exitCode=0
Sep 11 23:25:29.645244 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:29.645039    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-c0d4bf-openshift-default-948f987d-4kbcq"
Sep 11 23:25:29.645244 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:29.645074    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-c0d4bf-openshift-default-948f987d-4kbcq" event={"ID":"f621a2e1-7039-44a2-aa9a-a66e3e5e4e25","Type":"ContainerDied","Data":"06d4646b62b10d2abdf17449008d5067e24eb56d9735cd3a672cceed3ef30230"}
Sep 11 23:25:29.645244 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:29.645145    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-c0d4bf-openshift-default-948f987d-4kbcq" event={"ID":"f621a2e1-7039-44a2-aa9a-a66e3e5e4e25","Type":"ContainerDied","Data":"15b4bf8f1cb583890730d97ba8d347f7fce879cbad0f7a70f44c1def4a9a919a"}
Sep 11 23:25:29.645244 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:29.645236    2546 scope.go:117] "RemoveContainer" containerID="06d4646b62b10d2abdf17449008d5067e24eb56d9735cd3a672cceed3ef30230"
Sep 11 23:25:29.647506 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:29.647485    2546 generic.go:356] "Generic (PLEG): container finished" podID="3c330e7f-955c-4e6f-b4c8-5b0dee024ba5" containerID="a1bf2018af1bc1ccbdac0ba0ca08734bd30b10cc2303dfcff0520ed9be0cabef" exitCode=0
Sep 11 23:25:29.647595 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:29.647543    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b96fb0e1-openshift-default-df6bbcc4d-shm4v" event={"ID":"3c330e7f-955c-4e6f-b4c8-5b0dee024ba5","Type":"ContainerDied","Data":"a1bf2018af1bc1ccbdac0ba0ca08734bd30b10cc2303dfcff0520ed9be0cabef"}
Sep 11 23:25:29.648111 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:29.648017    2546 scope.go:117] "RemoveContainer" containerID="a1bf2018af1bc1ccbdac0ba0ca08734bd30b10cc2303dfcff0520ed9be0cabef"
Sep 11 23:25:29.650105 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:29.650069    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:25:29.650218 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:29.650182    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:25:29.650272 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:29.650234    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:25:29.656616 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:29.656596    2546 scope.go:117] "RemoveContainer" containerID="06d4646b62b10d2abdf17449008d5067e24eb56d9735cd3a672cceed3ef30230"
Sep 11 23:25:29.657005 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:25:29.656972    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06d4646b62b10d2abdf17449008d5067e24eb56d9735cd3a672cceed3ef30230\": container with ID starting with 06d4646b62b10d2abdf17449008d5067e24eb56d9735cd3a672cceed3ef30230 not found: ID does not exist" containerID="06d4646b62b10d2abdf17449008d5067e24eb56d9735cd3a672cceed3ef30230"
Sep 11 23:25:29.657101 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:29.657021    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06d4646b62b10d2abdf17449008d5067e24eb56d9735cd3a672cceed3ef30230"} err="failed to get container status \"06d4646b62b10d2abdf17449008d5067e24eb56d9735cd3a672cceed3ef30230\": rpc error: code = NotFound desc = could not find container \"06d4646b62b10d2abdf17449008d5067e24eb56d9735cd3a672cceed3ef30230\": container with ID starting with 06d4646b62b10d2abdf17449008d5067e24eb56d9735cd3a672cceed3ef30230 not found: ID does not exist"
Sep 11 23:25:29.757881 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:29.757844    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f621a2e1-7039-44a2-aa9a-a66e3e5e4e25-istio-podinfo\") pod \"f621a2e1-7039-44a2-aa9a-a66e3e5e4e25\" (UID: \"f621a2e1-7039-44a2-aa9a-a66e3e5e4e25\") "
Sep 11 23:25:29.758063 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:29.757940    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f621a2e1-7039-44a2-aa9a-a66e3e5e4e25-credential-socket\") pod \"f621a2e1-7039-44a2-aa9a-a66e3e5e4e25\" (UID: \"f621a2e1-7039-44a2-aa9a-a66e3e5e4e25\") "
Sep 11 23:25:29.758063 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:29.757986    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f621a2e1-7039-44a2-aa9a-a66e3e5e4e25-istiod-ca-cert\") pod \"f621a2e1-7039-44a2-aa9a-a66e3e5e4e25\" (UID: \"f621a2e1-7039-44a2-aa9a-a66e3e5e4e25\") "
Sep 11 23:25:29.758063 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:29.758028    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f621a2e1-7039-44a2-aa9a-a66e3e5e4e25-istio-token\") pod \"f621a2e1-7039-44a2-aa9a-a66e3e5e4e25\" (UID: \"f621a2e1-7039-44a2-aa9a-a66e3e5e4e25\") "
Sep 11 23:25:29.758063 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:29.758055    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f621a2e1-7039-44a2-aa9a-a66e3e5e4e25-istio-envoy\") pod \"f621a2e1-7039-44a2-aa9a-a66e3e5e4e25\" (UID: \"f621a2e1-7039-44a2-aa9a-a66e3e5e4e25\") "
Sep 11 23:25:29.758299 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:29.758100    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cskkd\" (UniqueName: \"kubernetes.io/projected/f621a2e1-7039-44a2-aa9a-a66e3e5e4e25-kube-api-access-cskkd\") pod \"f621a2e1-7039-44a2-aa9a-a66e3e5e4e25\" (UID: \"f621a2e1-7039-44a2-aa9a-a66e3e5e4e25\") "
Sep 11 23:25:29.758299 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:29.758140    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f621a2e1-7039-44a2-aa9a-a66e3e5e4e25-workload-certs\") pod \"f621a2e1-7039-44a2-aa9a-a66e3e5e4e25\" (UID: \"f621a2e1-7039-44a2-aa9a-a66e3e5e4e25\") "
Sep 11 23:25:29.758299 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:29.758198    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f621a2e1-7039-44a2-aa9a-a66e3e5e4e25-workload-socket\") pod \"f621a2e1-7039-44a2-aa9a-a66e3e5e4e25\" (UID: \"f621a2e1-7039-44a2-aa9a-a66e3e5e4e25\") "
Sep 11 23:25:29.758299 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:29.758273    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f621a2e1-7039-44a2-aa9a-a66e3e5e4e25-istio-data\") pod \"f621a2e1-7039-44a2-aa9a-a66e3e5e4e25\" (UID: \"f621a2e1-7039-44a2-aa9a-a66e3e5e4e25\") "
Sep 11 23:25:29.758513 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:29.758321    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f621a2e1-7039-44a2-aa9a-a66e3e5e4e25-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f621a2e1-7039-44a2-aa9a-a66e3e5e4e25" (UID: "f621a2e1-7039-44a2-aa9a-a66e3e5e4e25"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:25:29.758513 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:29.758379    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f621a2e1-7039-44a2-aa9a-a66e3e5e4e25-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f621a2e1-7039-44a2-aa9a-a66e3e5e4e25" (UID: "f621a2e1-7039-44a2-aa9a-a66e3e5e4e25"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 23:25:29.758780 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:29.758648    2546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f621a2e1-7039-44a2-aa9a-a66e3e5e4e25-credential-socket\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:25:29.758780 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:29.758669    2546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f621a2e1-7039-44a2-aa9a-a66e3e5e4e25-istiod-ca-cert\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:25:29.759016 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:29.758991    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f621a2e1-7039-44a2-aa9a-a66e3e5e4e25-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f621a2e1-7039-44a2-aa9a-a66e3e5e4e25" (UID: "f621a2e1-7039-44a2-aa9a-a66e3e5e4e25"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:25:29.759207 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:29.759186    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f621a2e1-7039-44a2-aa9a-a66e3e5e4e25-istio-data" (OuterVolumeSpecName: "istio-data") pod "f621a2e1-7039-44a2-aa9a-a66e3e5e4e25" (UID: "f621a2e1-7039-44a2-aa9a-a66e3e5e4e25"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:25:29.759413 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:29.759353    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f621a2e1-7039-44a2-aa9a-a66e3e5e4e25-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f621a2e1-7039-44a2-aa9a-a66e3e5e4e25" (UID: "f621a2e1-7039-44a2-aa9a-a66e3e5e4e25"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:25:29.760557 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:29.760535    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f621a2e1-7039-44a2-aa9a-a66e3e5e4e25-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f621a2e1-7039-44a2-aa9a-a66e3e5e4e25" (UID: "f621a2e1-7039-44a2-aa9a-a66e3e5e4e25"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 23:25:29.760928 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:29.760903    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f621a2e1-7039-44a2-aa9a-a66e3e5e4e25-kube-api-access-cskkd" (OuterVolumeSpecName: "kube-api-access-cskkd") pod "f621a2e1-7039-44a2-aa9a-a66e3e5e4e25" (UID: "f621a2e1-7039-44a2-aa9a-a66e3e5e4e25"). InnerVolumeSpecName "kube-api-access-cskkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 23:25:29.761248 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:29.761224    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f621a2e1-7039-44a2-aa9a-a66e3e5e4e25-istio-token" (OuterVolumeSpecName: "istio-token") pod "f621a2e1-7039-44a2-aa9a-a66e3e5e4e25" (UID: "f621a2e1-7039-44a2-aa9a-a66e3e5e4e25"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 23:25:29.761368 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:29.761256    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f621a2e1-7039-44a2-aa9a-a66e3e5e4e25-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f621a2e1-7039-44a2-aa9a-a66e3e5e4e25" (UID: "f621a2e1-7039-44a2-aa9a-a66e3e5e4e25"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:25:29.859282 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:29.859191    2546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f621a2e1-7039-44a2-aa9a-a66e3e5e4e25-istio-token\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:25:29.859282 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:29.859218    2546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f621a2e1-7039-44a2-aa9a-a66e3e5e4e25-istio-envoy\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:25:29.859282 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:29.859228    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cskkd\" (UniqueName: \"kubernetes.io/projected/f621a2e1-7039-44a2-aa9a-a66e3e5e4e25-kube-api-access-cskkd\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:25:29.859282 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:29.859237    2546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f621a2e1-7039-44a2-aa9a-a66e3e5e4e25-workload-certs\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:25:29.859282 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:29.859248    2546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f621a2e1-7039-44a2-aa9a-a66e3e5e4e25-workload-socket\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:25:29.859282 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:29.859255    2546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f621a2e1-7039-44a2-aa9a-a66e3e5e4e25-istio-data\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:25:29.859282 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:29.859263    2546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f621a2e1-7039-44a2-aa9a-a66e3e5e4e25-istio-podinfo\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:25:29.968926 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:29.968893    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-c0d4bf-openshift-default-948f987d-4kbcq"]
Sep 11 23:25:29.973813 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:29.973782    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-c0d4bf-openshift-default-948f987d-4kbcq"]
Sep 11 23:25:30.496718 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:30.496687    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f621a2e1-7039-44a2-aa9a-a66e3e5e4e25" path="/var/lib/kubelet/pods/f621a2e1-7039-44a2-aa9a-a66e3e5e4e25/volumes"
Sep 11 23:25:30.660228 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:30.660194    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b96fb0e1-openshift-default-df6bbcc4d-shm4v" event={"ID":"3c330e7f-955c-4e6f-b4c8-5b0dee024ba5","Type":"ContainerStarted","Data":"7985807caa2afa5f34824ac4e77fcb377d5fc7da5561325d3c7c636ec7e6c65f"}
Sep 11 23:25:30.813603 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:30.813515    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b96fb0e1-openshift-default-df6bbcc4d-shm4v"
Sep 11 23:25:30.814571 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:30.814541    2546 patch_prober.go:28] interesting pod/e2e-mt-b96fb0e1-openshift-default-df6bbcc4d-shm4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Sep 11 23:25:30.814693 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:30.814600    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b96fb0e1-openshift-default-df6bbcc4d-shm4v" podUID="3c330e7f-955c-4e6f-b4c8-5b0dee024ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Sep 11 23:25:31.814250 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:31.814211    2546 patch_prober.go:28] interesting pod/e2e-mt-b96fb0e1-openshift-default-df6bbcc4d-shm4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Sep 11 23:25:31.814727 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:31.814276    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b96fb0e1-openshift-default-df6bbcc4d-shm4v" podUID="3c330e7f-955c-4e6f-b4c8-5b0dee024ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Sep 11 23:25:32.813976 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:32.813935    2546 patch_prober.go:28] interesting pod/e2e-mt-b96fb0e1-openshift-default-df6bbcc4d-shm4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Sep 11 23:25:32.814182 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:32.813999    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b96fb0e1-openshift-default-df6bbcc4d-shm4v" podUID="3c330e7f-955c-4e6f-b4c8-5b0dee024ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Sep 11 23:25:33.300057 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:33.300029    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq"]
Sep 11 23:25:33.300743 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:33.300706    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq" podUID="dcd4c593-5a49-4cdc-bd1a-27c49cbe594c" containerName="istio-proxy" containerID="cri-o://e3f7a3859610c7b831ae66497695213320dac08ba33e63db7db8a05d0943538e" gracePeriod=30
Sep 11 23:25:33.411472 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:33.411433    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b96fb0e1-openshift-default-df6bbcc4d-shm4v"
Sep 11 23:25:33.813705 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:33.813664    2546 patch_prober.go:28] interesting pod/e2e-mt-b96fb0e1-openshift-default-df6bbcc4d-shm4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Sep 11 23:25:33.813891 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:33.813721    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b96fb0e1-openshift-default-df6bbcc4d-shm4v" podUID="3c330e7f-955c-4e6f-b4c8-5b0dee024ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Sep 11 23:25:34.160485 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:34.160447    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k"]
Sep 11 23:25:34.161192 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:34.161168    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe27a86b-83cf-4afa-a295-f832ded073a0" containerName="payload-pre-processing"
Sep 11 23:25:34.161192 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:34.161189    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe27a86b-83cf-4afa-a295-f832ded073a0" containerName="payload-pre-processing"
Sep 11 23:25:34.161336 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:34.161213    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2304922d-c8d4-4a95-a81c-4af2ccea36b0" containerName="istio-proxy"
Sep 11 23:25:34.161336 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:34.161219    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="2304922d-c8d4-4a95-a81c-4af2ccea36b0" containerName="istio-proxy"
Sep 11 23:25:34.161336 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:34.161225    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f621a2e1-7039-44a2-aa9a-a66e3e5e4e25" containerName="istio-proxy"
Sep 11 23:25:34.161336 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:34.161230    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="f621a2e1-7039-44a2-aa9a-a66e3e5e4e25" containerName="istio-proxy"
Sep 11 23:25:34.161336 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:34.161247    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd62c1ac-0a15-4ad2-82b3-fd73eb6e247f" containerName="revoke-keys"
Sep 11 23:25:34.161336 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:34.161252    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd62c1ac-0a15-4ad2-82b3-fd73eb6e247f" containerName="revoke-keys"
Sep 11 23:25:34.161336 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:34.161323    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe27a86b-83cf-4afa-a295-f832ded073a0" containerName="payload-pre-processing"
Sep 11 23:25:34.161605 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:34.161344    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="2304922d-c8d4-4a95-a81c-4af2ccea36b0" containerName="istio-proxy"
Sep 11 23:25:34.161605 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:34.161351    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="2304922d-c8d4-4a95-a81c-4af2ccea36b0" containerName="istio-proxy"
Sep 11 23:25:34.161605 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:34.161357    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="f621a2e1-7039-44a2-aa9a-a66e3e5e4e25" containerName="istio-proxy"
Sep 11 23:25:34.161605 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:34.161367    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd62c1ac-0a15-4ad2-82b3-fd73eb6e247f" containerName="revoke-keys"
Sep 11 23:25:34.166121 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:34.166100    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k"
Sep 11 23:25:34.171048 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:34.171032    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-1c4628-openshift-default-dockercfg-tp2z2\""
Sep 11 23:25:34.201253 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:34.201218    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb3c41dd-091c-4f4e-94e5-55d389088418-istio-data\") pod \"e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k\" (UID: \"bb3c41dd-091c-4f4e-94e5-55d389088418\") " pod="openshift-ingress/e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k"
Sep 11 23:25:34.201253 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:34.201257    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb3c41dd-091c-4f4e-94e5-55d389088418-istio-podinfo\") pod \"e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k\" (UID: \"bb3c41dd-091c-4f4e-94e5-55d389088418\") " pod="openshift-ingress/e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k"
Sep 11 23:25:34.201454 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:34.201296    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnw5p\" (UniqueName: \"kubernetes.io/projected/bb3c41dd-091c-4f4e-94e5-55d389088418-kube-api-access-hnw5p\") pod \"e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k\" (UID: \"bb3c41dd-091c-4f4e-94e5-55d389088418\") " pod="openshift-ingress/e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k"
Sep 11 23:25:34.201454 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:34.201317    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb3c41dd-091c-4f4e-94e5-55d389088418-credential-socket\") pod \"e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k\" (UID: \"bb3c41dd-091c-4f4e-94e5-55d389088418\") " pod="openshift-ingress/e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k"
Sep 11 23:25:34.201454 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:34.201334    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb3c41dd-091c-4f4e-94e5-55d389088418-istio-envoy\") pod \"e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k\" (UID: \"bb3c41dd-091c-4f4e-94e5-55d389088418\") " pod="openshift-ingress/e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k"
Sep 11 23:25:34.201454 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:34.201385    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb3c41dd-091c-4f4e-94e5-55d389088418-workload-socket\") pod \"e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k\" (UID: \"bb3c41dd-091c-4f4e-94e5-55d389088418\") " pod="openshift-ingress/e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k"
Sep 11 23:25:34.201454 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:34.201403    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb3c41dd-091c-4f4e-94e5-55d389088418-istiod-ca-cert\") pod \"e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k\" (UID: \"bb3c41dd-091c-4f4e-94e5-55d389088418\") " pod="openshift-ingress/e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k"
Sep 11 23:25:34.201454 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:34.201422    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb3c41dd-091c-4f4e-94e5-55d389088418-workload-certs\") pod \"e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k\" (UID: \"bb3c41dd-091c-4f4e-94e5-55d389088418\") " pod="openshift-ingress/e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k"
Sep 11 23:25:34.201642 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:34.201507    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb3c41dd-091c-4f4e-94e5-55d389088418-istio-token\") pod \"e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k\" (UID: \"bb3c41dd-091c-4f4e-94e5-55d389088418\") " pod="openshift-ingress/e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k"
Sep 11 23:25:34.219636 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:34.219601    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k"]
Sep 11 23:25:34.305839 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:34.305555    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb3c41dd-091c-4f4e-94e5-55d389088418-credential-socket\") pod \"e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k\" (UID: \"bb3c41dd-091c-4f4e-94e5-55d389088418\") " pod="openshift-ingress/e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k"
Sep 11 23:25:34.305839 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:34.305604    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb3c41dd-091c-4f4e-94e5-55d389088418-istio-envoy\") pod \"e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k\" (UID: \"bb3c41dd-091c-4f4e-94e5-55d389088418\") " pod="openshift-ingress/e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k"
Sep 11 23:25:34.305839 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:34.305684    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb3c41dd-091c-4f4e-94e5-55d389088418-workload-socket\") pod \"e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k\" (UID: \"bb3c41dd-091c-4f4e-94e5-55d389088418\") " pod="openshift-ingress/e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k"
Sep 11 23:25:34.305839 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:34.305718    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb3c41dd-091c-4f4e-94e5-55d389088418-istiod-ca-cert\") pod \"e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k\" (UID: \"bb3c41dd-091c-4f4e-94e5-55d389088418\") " pod="openshift-ingress/e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k"
Sep 11 23:25:34.305839 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:34.305751    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb3c41dd-091c-4f4e-94e5-55d389088418-workload-certs\") pod \"e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k\" (UID: \"bb3c41dd-091c-4f4e-94e5-55d389088418\") " pod="openshift-ingress/e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k"
Sep 11 23:25:34.305839 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:34.305818    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb3c41dd-091c-4f4e-94e5-55d389088418-istio-token\") pod \"e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k\" (UID: \"bb3c41dd-091c-4f4e-94e5-55d389088418\") " pod="openshift-ingress/e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k"
Sep 11 23:25:34.306638 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:34.305908    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb3c41dd-091c-4f4e-94e5-55d389088418-istio-data\") pod \"e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k\" (UID: \"bb3c41dd-091c-4f4e-94e5-55d389088418\") " pod="openshift-ingress/e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k"
Sep 11 23:25:34.306638 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:34.305950    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb3c41dd-091c-4f4e-94e5-55d389088418-istio-podinfo\") pod \"e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k\" (UID: \"bb3c41dd-091c-4f4e-94e5-55d389088418\") " pod="openshift-ingress/e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k"
Sep 11 23:25:34.306638 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:34.305984    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnw5p\" (UniqueName: \"kubernetes.io/projected/bb3c41dd-091c-4f4e-94e5-55d389088418-kube-api-access-hnw5p\") pod \"e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k\" (UID: \"bb3c41dd-091c-4f4e-94e5-55d389088418\") " pod="openshift-ingress/e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k"
Sep 11 23:25:34.306801 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:34.306731    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb3c41dd-091c-4f4e-94e5-55d389088418-credential-socket\") pod \"e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k\" (UID: \"bb3c41dd-091c-4f4e-94e5-55d389088418\") " pod="openshift-ingress/e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k"
Sep 11 23:25:34.307757 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:34.307044    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb3c41dd-091c-4f4e-94e5-55d389088418-workload-socket\") pod \"e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k\" (UID: \"bb3c41dd-091c-4f4e-94e5-55d389088418\") " pod="openshift-ingress/e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k"
Sep 11 23:25:34.307757 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:34.307184    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb3c41dd-091c-4f4e-94e5-55d389088418-istio-data\") pod \"e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k\" (UID: \"bb3c41dd-091c-4f4e-94e5-55d389088418\") " pod="openshift-ingress/e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k"
Sep 11 23:25:34.307757 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:34.307661    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb3c41dd-091c-4f4e-94e5-55d389088418-workload-certs\") pod \"e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k\" (UID: \"bb3c41dd-091c-4f4e-94e5-55d389088418\") " pod="openshift-ingress/e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k"
Sep 11 23:25:34.307757 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:34.307671    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb3c41dd-091c-4f4e-94e5-55d389088418-istiod-ca-cert\") pod \"e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k\" (UID: \"bb3c41dd-091c-4f4e-94e5-55d389088418\") " pod="openshift-ingress/e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k"
Sep 11 23:25:34.310487 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:34.310178    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb3c41dd-091c-4f4e-94e5-55d389088418-istio-envoy\") pod \"e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k\" (UID: \"bb3c41dd-091c-4f4e-94e5-55d389088418\") " pod="openshift-ingress/e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k"
Sep 11 23:25:34.313341 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:34.313311    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb3c41dd-091c-4f4e-94e5-55d389088418-istio-podinfo\") pod \"e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k\" (UID: \"bb3c41dd-091c-4f4e-94e5-55d389088418\") " pod="openshift-ingress/e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k"
Sep 11 23:25:34.316482 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:34.316457    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb3c41dd-091c-4f4e-94e5-55d389088418-istio-token\") pod \"e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k\" (UID: \"bb3c41dd-091c-4f4e-94e5-55d389088418\") " pod="openshift-ingress/e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k"
Sep 11 23:25:34.316796 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:34.316781    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnw5p\" (UniqueName: \"kubernetes.io/projected/bb3c41dd-091c-4f4e-94e5-55d389088418-kube-api-access-hnw5p\") pod \"e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k\" (UID: \"bb3c41dd-091c-4f4e-94e5-55d389088418\") " pod="openshift-ingress/e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k"
Sep 11 23:25:34.476572 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:34.476483    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k"
Sep 11 23:25:34.794274 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:34.794237    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k"]
Sep 11 23:25:34.794477 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:25:34.794453    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb3c41dd_091c_4f4e_94e5_55d389088418.slice/crio-130ad9cc4bf416a85d4c916ac4787cf031bc7511c12b12d7f31eb1422633c5f8 WatchSource:0}: Error finding container 130ad9cc4bf416a85d4c916ac4787cf031bc7511c12b12d7f31eb1422633c5f8: Status 404 returned error can't find the container with id 130ad9cc4bf416a85d4c916ac4787cf031bc7511c12b12d7f31eb1422633c5f8
Sep 11 23:25:34.796929 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:34.796901    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:25:34.797073 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:34.797058    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:25:34.797196 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:34.797183    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:25:34.814255 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:34.814182    2546 patch_prober.go:28] interesting pod/e2e-mt-b96fb0e1-openshift-default-df6bbcc4d-shm4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Sep 11 23:25:34.814255 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:34.814245    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b96fb0e1-openshift-default-df6bbcc4d-shm4v" podUID="3c330e7f-955c-4e6f-b4c8-5b0dee024ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Sep 11 23:25:35.700227 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:35.700197    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k" event={"ID":"bb3c41dd-091c-4f4e-94e5-55d389088418","Type":"ContainerStarted","Data":"5f14e88f52446d1883b5f8e093a7154e7a38ed1b80238ce72f2c461a26eed13b"}
Sep 11 23:25:35.700630 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:35.700248    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k" event={"ID":"bb3c41dd-091c-4f4e-94e5-55d389088418","Type":"ContainerStarted","Data":"130ad9cc4bf416a85d4c916ac4787cf031bc7511c12b12d7f31eb1422633c5f8"}
Sep 11 23:25:35.747967 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:35.747882    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k" podStartSLOduration=1.747858848 podStartE2EDuration="1.747858848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 23:25:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:25:35.743554095 +0000 UTC m=+1459.776683310" watchObservedRunningTime="2026-09-11 23:25:35.747858848 +0000 UTC m=+1459.780988064"
Sep 11 23:25:35.829937 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:35.829886    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b96fb0e1-openshift-default-df6bbcc4d-shm4v"
Sep 11 23:25:35.832052 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:35.832013    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b96fb0e1-openshift-default-df6bbcc4d-shm4v"
Sep 11 23:25:36.477275 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:36.477092    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k"
Sep 11 23:25:36.482135 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:36.482107    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k"
Sep 11 23:25:36.707708 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:36.707675    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k"
Sep 11 23:25:36.708790 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:36.708770    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k"
Sep 11 23:25:37.072984 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:37.072933    2546 patch_prober.go:28] interesting pod/e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 11 23:25:37.073199 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:37.073006    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq" podUID="dcd4c593-5a49-4cdc-bd1a-27c49cbe594c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 23:25:38.573233 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:38.573137    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq"
Sep 11 23:25:38.661940 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:38.658124    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcd4c593-5a49-4cdc-bd1a-27c49cbe594c-istio-podinfo\") pod \"dcd4c593-5a49-4cdc-bd1a-27c49cbe594c\" (UID: \"dcd4c593-5a49-4cdc-bd1a-27c49cbe594c\") "
Sep 11 23:25:38.661940 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:38.658536    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcd4c593-5a49-4cdc-bd1a-27c49cbe594c-istio-token\") pod \"dcd4c593-5a49-4cdc-bd1a-27c49cbe594c\" (UID: \"dcd4c593-5a49-4cdc-bd1a-27c49cbe594c\") "
Sep 11 23:25:38.661940 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:38.658624    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcd4c593-5a49-4cdc-bd1a-27c49cbe594c-istio-data\") pod \"dcd4c593-5a49-4cdc-bd1a-27c49cbe594c\" (UID: \"dcd4c593-5a49-4cdc-bd1a-27c49cbe594c\") "
Sep 11 23:25:38.661940 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:38.658705    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcd4c593-5a49-4cdc-bd1a-27c49cbe594c-istiod-ca-cert\") pod \"dcd4c593-5a49-4cdc-bd1a-27c49cbe594c\" (UID: \"dcd4c593-5a49-4cdc-bd1a-27c49cbe594c\") "
Sep 11 23:25:38.661940 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:38.658747    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zwbk2\" (UniqueName: \"kubernetes.io/projected/dcd4c593-5a49-4cdc-bd1a-27c49cbe594c-kube-api-access-zwbk2\") pod \"dcd4c593-5a49-4cdc-bd1a-27c49cbe594c\" (UID: \"dcd4c593-5a49-4cdc-bd1a-27c49cbe594c\") "
Sep 11 23:25:38.661940 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:38.658927    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcd4c593-5a49-4cdc-bd1a-27c49cbe594c-istio-envoy\") pod \"dcd4c593-5a49-4cdc-bd1a-27c49cbe594c\" (UID: \"dcd4c593-5a49-4cdc-bd1a-27c49cbe594c\") "
Sep 11 23:25:38.661940 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:38.658986    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcd4c593-5a49-4cdc-bd1a-27c49cbe594c-credential-socket\") pod \"dcd4c593-5a49-4cdc-bd1a-27c49cbe594c\" (UID: \"dcd4c593-5a49-4cdc-bd1a-27c49cbe594c\") "
Sep 11 23:25:38.661940 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:38.659026    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcd4c593-5a49-4cdc-bd1a-27c49cbe594c-workload-socket\") pod \"dcd4c593-5a49-4cdc-bd1a-27c49cbe594c\" (UID: \"dcd4c593-5a49-4cdc-bd1a-27c49cbe594c\") "
Sep 11 23:25:38.661940 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:38.659078    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcd4c593-5a49-4cdc-bd1a-27c49cbe594c-workload-certs\") pod \"dcd4c593-5a49-4cdc-bd1a-27c49cbe594c\" (UID: \"dcd4c593-5a49-4cdc-bd1a-27c49cbe594c\") "
Sep 11 23:25:38.661940 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:38.659999    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcd4c593-5a49-4cdc-bd1a-27c49cbe594c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "dcd4c593-5a49-4cdc-bd1a-27c49cbe594c" (UID: "dcd4c593-5a49-4cdc-bd1a-27c49cbe594c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:25:38.661940 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:38.660212    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcd4c593-5a49-4cdc-bd1a-27c49cbe594c-istio-data" (OuterVolumeSpecName: "istio-data") pod "dcd4c593-5a49-4cdc-bd1a-27c49cbe594c" (UID: "dcd4c593-5a49-4cdc-bd1a-27c49cbe594c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:25:38.661940 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:38.660477    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcd4c593-5a49-4cdc-bd1a-27c49cbe594c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "dcd4c593-5a49-4cdc-bd1a-27c49cbe594c" (UID: "dcd4c593-5a49-4cdc-bd1a-27c49cbe594c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 23:25:38.663346 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:38.663050    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcd4c593-5a49-4cdc-bd1a-27c49cbe594c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "dcd4c593-5a49-4cdc-bd1a-27c49cbe594c" (UID: "dcd4c593-5a49-4cdc-bd1a-27c49cbe594c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:25:38.663890 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:38.663845    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcd4c593-5a49-4cdc-bd1a-27c49cbe594c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "dcd4c593-5a49-4cdc-bd1a-27c49cbe594c" (UID: "dcd4c593-5a49-4cdc-bd1a-27c49cbe594c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:25:38.664419 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:38.664383    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcd4c593-5a49-4cdc-bd1a-27c49cbe594c-kube-api-access-zwbk2" (OuterVolumeSpecName: "kube-api-access-zwbk2") pod "dcd4c593-5a49-4cdc-bd1a-27c49cbe594c" (UID: "dcd4c593-5a49-4cdc-bd1a-27c49cbe594c"). InnerVolumeSpecName "kube-api-access-zwbk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 23:25:38.665314 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:38.665288    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcd4c593-5a49-4cdc-bd1a-27c49cbe594c-istio-token" (OuterVolumeSpecName: "istio-token") pod "dcd4c593-5a49-4cdc-bd1a-27c49cbe594c" (UID: "dcd4c593-5a49-4cdc-bd1a-27c49cbe594c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 23:25:38.665431 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:38.665388    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dcd4c593-5a49-4cdc-bd1a-27c49cbe594c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "dcd4c593-5a49-4cdc-bd1a-27c49cbe594c" (UID: "dcd4c593-5a49-4cdc-bd1a-27c49cbe594c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 23:25:38.666718 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:38.666688    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcd4c593-5a49-4cdc-bd1a-27c49cbe594c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "dcd4c593-5a49-4cdc-bd1a-27c49cbe594c" (UID: "dcd4c593-5a49-4cdc-bd1a-27c49cbe594c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:25:38.723295 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:38.723259    2546 generic.go:356] "Generic (PLEG): container finished" podID="dcd4c593-5a49-4cdc-bd1a-27c49cbe594c" containerID="e3f7a3859610c7b831ae66497695213320dac08ba33e63db7db8a05d0943538e" exitCode=0
Sep 11 23:25:38.723496 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:38.723326    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq"
Sep 11 23:25:38.723496 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:38.723359    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq" event={"ID":"dcd4c593-5a49-4cdc-bd1a-27c49cbe594c","Type":"ContainerDied","Data":"e3f7a3859610c7b831ae66497695213320dac08ba33e63db7db8a05d0943538e"}
Sep 11 23:25:38.723496 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:38.723415    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq" event={"ID":"dcd4c593-5a49-4cdc-bd1a-27c49cbe594c","Type":"ContainerDied","Data":"39fadfa9acc6c4867d82dfb77e6408892e5eddb41fe0c5387c7bb4d2f632c07f"}
Sep 11 23:25:38.723496 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:38.723437    2546 scope.go:117] "RemoveContainer" containerID="e3f7a3859610c7b831ae66497695213320dac08ba33e63db7db8a05d0943538e"
Sep 11 23:25:38.733854 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:38.733822    2546 scope.go:117] "RemoveContainer" containerID="b8252117d86680331cf009c4a07735c80a2714ef52eccd148c2cab7e5aae94e7"
Sep 11 23:25:38.744053 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:38.744034    2546 scope.go:117] "RemoveContainer" containerID="e3f7a3859610c7b831ae66497695213320dac08ba33e63db7db8a05d0943538e"
Sep 11 23:25:38.744439 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:25:38.744417    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3f7a3859610c7b831ae66497695213320dac08ba33e63db7db8a05d0943538e\": container with ID starting with e3f7a3859610c7b831ae66497695213320dac08ba33e63db7db8a05d0943538e not found: ID does not exist" containerID="e3f7a3859610c7b831ae66497695213320dac08ba33e63db7db8a05d0943538e"
Sep 11 23:25:38.744544 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:38.744452    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3f7a3859610c7b831ae66497695213320dac08ba33e63db7db8a05d0943538e"} err="failed to get container status \"e3f7a3859610c7b831ae66497695213320dac08ba33e63db7db8a05d0943538e\": rpc error: code = NotFound desc = could not find container \"e3f7a3859610c7b831ae66497695213320dac08ba33e63db7db8a05d0943538e\": container with ID starting with e3f7a3859610c7b831ae66497695213320dac08ba33e63db7db8a05d0943538e not found: ID does not exist"
Sep 11 23:25:38.744544 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:38.744478    2546 scope.go:117] "RemoveContainer" containerID="b8252117d86680331cf009c4a07735c80a2714ef52eccd148c2cab7e5aae94e7"
Sep 11 23:25:38.744785 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:25:38.744763    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8252117d86680331cf009c4a07735c80a2714ef52eccd148c2cab7e5aae94e7\": container with ID starting with b8252117d86680331cf009c4a07735c80a2714ef52eccd148c2cab7e5aae94e7 not found: ID does not exist" containerID="b8252117d86680331cf009c4a07735c80a2714ef52eccd148c2cab7e5aae94e7"
Sep 11 23:25:38.744825 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:38.744795    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8252117d86680331cf009c4a07735c80a2714ef52eccd148c2cab7e5aae94e7"} err="failed to get container status \"b8252117d86680331cf009c4a07735c80a2714ef52eccd148c2cab7e5aae94e7\": rpc error: code = NotFound desc = could not find container \"b8252117d86680331cf009c4a07735c80a2714ef52eccd148c2cab7e5aae94e7\": container with ID starting with b8252117d86680331cf009c4a07735c80a2714ef52eccd148c2cab7e5aae94e7 not found: ID does not exist"
Sep 11 23:25:38.749726 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:38.749687    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq"]
Sep 11 23:25:38.753579 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:38.753556    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-bfa694-openshift-default-6c9f8d96d-7mmsq"]
Sep 11 23:25:38.761310 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:38.761288    2546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcd4c593-5a49-4cdc-bd1a-27c49cbe594c-workload-socket\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:25:38.761425 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:38.761311    2546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcd4c593-5a49-4cdc-bd1a-27c49cbe594c-workload-certs\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:25:38.761425 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:38.761325    2546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcd4c593-5a49-4cdc-bd1a-27c49cbe594c-istio-podinfo\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:25:38.761425 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:38.761336    2546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcd4c593-5a49-4cdc-bd1a-27c49cbe594c-istio-token\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:25:38.761425 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:38.761348    2546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcd4c593-5a49-4cdc-bd1a-27c49cbe594c-istio-data\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:25:38.761425 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:38.761358    2546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcd4c593-5a49-4cdc-bd1a-27c49cbe594c-istiod-ca-cert\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:25:38.761425 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:38.761369    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwbk2\" (UniqueName: \"kubernetes.io/projected/dcd4c593-5a49-4cdc-bd1a-27c49cbe594c-kube-api-access-zwbk2\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:25:38.761425 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:38.761382    2546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcd4c593-5a49-4cdc-bd1a-27c49cbe594c-istio-envoy\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:25:38.761425 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:38.761394    2546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcd4c593-5a49-4cdc-bd1a-27c49cbe594c-credential-socket\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:25:40.495576 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:40.495537    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dcd4c593-5a49-4cdc-bd1a-27c49cbe594c" path="/var/lib/kubelet/pods/dcd4c593-5a49-4cdc-bd1a-27c49cbe594c/volumes"
Sep 11 23:25:43.620542 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:25:43.620508    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"]
Sep 11 23:26:00.507811 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:00.507777    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-12fb6b/distinct-2-w0-12fb6b-kserve-56c4887795-bf8sz"]
Sep 11 23:26:00.508294 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:00.508280    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcd4c593-5a49-4cdc-bd1a-27c49cbe594c" containerName="istio-proxy"
Sep 11 23:26:00.508294 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:00.508295    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcd4c593-5a49-4cdc-bd1a-27c49cbe594c" containerName="istio-proxy"
Sep 11 23:26:00.508372 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:00.508310    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2304922d-c8d4-4a95-a81c-4af2ccea36b0" containerName="istio-proxy"
Sep 11 23:26:00.508372 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:00.508316    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="2304922d-c8d4-4a95-a81c-4af2ccea36b0" containerName="istio-proxy"
Sep 11 23:26:00.508372 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:00.508331    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcd4c593-5a49-4cdc-bd1a-27c49cbe594c" containerName="istio-proxy"
Sep 11 23:26:00.508372 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:00.508336    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcd4c593-5a49-4cdc-bd1a-27c49cbe594c" containerName="istio-proxy"
Sep 11 23:26:00.508536 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:00.508406    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcd4c593-5a49-4cdc-bd1a-27c49cbe594c" containerName="istio-proxy"
Sep 11 23:26:00.508536 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:00.508416    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcd4c593-5a49-4cdc-bd1a-27c49cbe594c" containerName="istio-proxy"
Sep 11 23:26:00.511652 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:00.511633    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-12fb6b/distinct-2-w0-12fb6b-kserve-56c4887795-bf8sz"
Sep 11 23:26:00.514858 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:00.514840    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-12fb6b\"/\"distinct-2-w0-12fb6b-kserve-self-signed-certs\""
Sep 11 23:26:00.521979 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:00.521954    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-12fb6b/distinct-2-w0-12fb6b-kserve-56c4887795-bf8sz"]
Sep 11 23:26:00.695208 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:00.695146    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9da6fe34-d6b1-4b92-a067-e2677f71bca9-tmp-dir\") pod \"distinct-2-w0-12fb6b-kserve-56c4887795-bf8sz\" (UID: \"9da6fe34-d6b1-4b92-a067-e2677f71bca9\") " pod="e2e-models-e2e-worker-w0-12fb6b/distinct-2-w0-12fb6b-kserve-56c4887795-bf8sz"
Sep 11 23:26:00.696547 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:00.696515    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9da6fe34-d6b1-4b92-a067-e2677f71bca9-dshm\") pod \"distinct-2-w0-12fb6b-kserve-56c4887795-bf8sz\" (UID: \"9da6fe34-d6b1-4b92-a067-e2677f71bca9\") " pod="e2e-models-e2e-worker-w0-12fb6b/distinct-2-w0-12fb6b-kserve-56c4887795-bf8sz"
Sep 11 23:26:00.696694 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:00.696596    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9da6fe34-d6b1-4b92-a067-e2677f71bca9-model-cache\") pod \"distinct-2-w0-12fb6b-kserve-56c4887795-bf8sz\" (UID: \"9da6fe34-d6b1-4b92-a067-e2677f71bca9\") " pod="e2e-models-e2e-worker-w0-12fb6b/distinct-2-w0-12fb6b-kserve-56c4887795-bf8sz"
Sep 11 23:26:00.696694 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:00.696645    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9da6fe34-d6b1-4b92-a067-e2677f71bca9-tls-certs\") pod \"distinct-2-w0-12fb6b-kserve-56c4887795-bf8sz\" (UID: \"9da6fe34-d6b1-4b92-a067-e2677f71bca9\") " pod="e2e-models-e2e-worker-w0-12fb6b/distinct-2-w0-12fb6b-kserve-56c4887795-bf8sz"
Sep 11 23:26:00.696811 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:00.696734    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9da6fe34-d6b1-4b92-a067-e2677f71bca9-home\") pod \"distinct-2-w0-12fb6b-kserve-56c4887795-bf8sz\" (UID: \"9da6fe34-d6b1-4b92-a067-e2677f71bca9\") " pod="e2e-models-e2e-worker-w0-12fb6b/distinct-2-w0-12fb6b-kserve-56c4887795-bf8sz"
Sep 11 23:26:00.696811 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:00.696803    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qql8n\" (UniqueName: \"kubernetes.io/projected/9da6fe34-d6b1-4b92-a067-e2677f71bca9-kube-api-access-qql8n\") pod \"distinct-2-w0-12fb6b-kserve-56c4887795-bf8sz\" (UID: \"9da6fe34-d6b1-4b92-a067-e2677f71bca9\") " pod="e2e-models-e2e-worker-w0-12fb6b/distinct-2-w0-12fb6b-kserve-56c4887795-bf8sz"
Sep 11 23:26:00.798418 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:00.798327    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9da6fe34-d6b1-4b92-a067-e2677f71bca9-tmp-dir\") pod \"distinct-2-w0-12fb6b-kserve-56c4887795-bf8sz\" (UID: \"9da6fe34-d6b1-4b92-a067-e2677f71bca9\") " pod="e2e-models-e2e-worker-w0-12fb6b/distinct-2-w0-12fb6b-kserve-56c4887795-bf8sz"
Sep 11 23:26:00.798418 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:00.798372    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9da6fe34-d6b1-4b92-a067-e2677f71bca9-dshm\") pod \"distinct-2-w0-12fb6b-kserve-56c4887795-bf8sz\" (UID: \"9da6fe34-d6b1-4b92-a067-e2677f71bca9\") " pod="e2e-models-e2e-worker-w0-12fb6b/distinct-2-w0-12fb6b-kserve-56c4887795-bf8sz"
Sep 11 23:26:00.798418 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:00.798401    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9da6fe34-d6b1-4b92-a067-e2677f71bca9-model-cache\") pod \"distinct-2-w0-12fb6b-kserve-56c4887795-bf8sz\" (UID: \"9da6fe34-d6b1-4b92-a067-e2677f71bca9\") " pod="e2e-models-e2e-worker-w0-12fb6b/distinct-2-w0-12fb6b-kserve-56c4887795-bf8sz"
Sep 11 23:26:00.798691 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:00.798521    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9da6fe34-d6b1-4b92-a067-e2677f71bca9-tls-certs\") pod \"distinct-2-w0-12fb6b-kserve-56c4887795-bf8sz\" (UID: \"9da6fe34-d6b1-4b92-a067-e2677f71bca9\") " pod="e2e-models-e2e-worker-w0-12fb6b/distinct-2-w0-12fb6b-kserve-56c4887795-bf8sz"
Sep 11 23:26:00.798691 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:00.798605    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9da6fe34-d6b1-4b92-a067-e2677f71bca9-home\") pod \"distinct-2-w0-12fb6b-kserve-56c4887795-bf8sz\" (UID: \"9da6fe34-d6b1-4b92-a067-e2677f71bca9\") " pod="e2e-models-e2e-worker-w0-12fb6b/distinct-2-w0-12fb6b-kserve-56c4887795-bf8sz"
Sep 11 23:26:00.798691 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:00.798650    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qql8n\" (UniqueName: \"kubernetes.io/projected/9da6fe34-d6b1-4b92-a067-e2677f71bca9-kube-api-access-qql8n\") pod \"distinct-2-w0-12fb6b-kserve-56c4887795-bf8sz\" (UID: \"9da6fe34-d6b1-4b92-a067-e2677f71bca9\") " pod="e2e-models-e2e-worker-w0-12fb6b/distinct-2-w0-12fb6b-kserve-56c4887795-bf8sz"
Sep 11 23:26:00.798841 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:00.798754    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9da6fe34-d6b1-4b92-a067-e2677f71bca9-tmp-dir\") pod \"distinct-2-w0-12fb6b-kserve-56c4887795-bf8sz\" (UID: \"9da6fe34-d6b1-4b92-a067-e2677f71bca9\") " pod="e2e-models-e2e-worker-w0-12fb6b/distinct-2-w0-12fb6b-kserve-56c4887795-bf8sz"
Sep 11 23:26:00.798841 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:00.798800    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9da6fe34-d6b1-4b92-a067-e2677f71bca9-model-cache\") pod \"distinct-2-w0-12fb6b-kserve-56c4887795-bf8sz\" (UID: \"9da6fe34-d6b1-4b92-a067-e2677f71bca9\") " pod="e2e-models-e2e-worker-w0-12fb6b/distinct-2-w0-12fb6b-kserve-56c4887795-bf8sz"
Sep 11 23:26:00.798961 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:00.798943    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9da6fe34-d6b1-4b92-a067-e2677f71bca9-home\") pod \"distinct-2-w0-12fb6b-kserve-56c4887795-bf8sz\" (UID: \"9da6fe34-d6b1-4b92-a067-e2677f71bca9\") " pod="e2e-models-e2e-worker-w0-12fb6b/distinct-2-w0-12fb6b-kserve-56c4887795-bf8sz"
Sep 11 23:26:00.801001 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:00.800965    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9da6fe34-d6b1-4b92-a067-e2677f71bca9-dshm\") pod \"distinct-2-w0-12fb6b-kserve-56c4887795-bf8sz\" (UID: \"9da6fe34-d6b1-4b92-a067-e2677f71bca9\") " pod="e2e-models-e2e-worker-w0-12fb6b/distinct-2-w0-12fb6b-kserve-56c4887795-bf8sz"
Sep 11 23:26:00.801261 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:00.801244    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9da6fe34-d6b1-4b92-a067-e2677f71bca9-tls-certs\") pod \"distinct-2-w0-12fb6b-kserve-56c4887795-bf8sz\" (UID: \"9da6fe34-d6b1-4b92-a067-e2677f71bca9\") " pod="e2e-models-e2e-worker-w0-12fb6b/distinct-2-w0-12fb6b-kserve-56c4887795-bf8sz"
Sep 11 23:26:00.807003 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:00.806984    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qql8n\" (UniqueName: \"kubernetes.io/projected/9da6fe34-d6b1-4b92-a067-e2677f71bca9-kube-api-access-qql8n\") pod \"distinct-2-w0-12fb6b-kserve-56c4887795-bf8sz\" (UID: \"9da6fe34-d6b1-4b92-a067-e2677f71bca9\") " pod="e2e-models-e2e-worker-w0-12fb6b/distinct-2-w0-12fb6b-kserve-56c4887795-bf8sz"
Sep 11 23:26:00.821312 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:00.821278    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-12fb6b/distinct-2-w0-12fb6b-kserve-56c4887795-bf8sz"
Sep 11 23:26:01.118432 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:01.118407    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-12fb6b/distinct-2-w0-12fb6b-kserve-56c4887795-bf8sz"]
Sep 11 23:26:01.119647 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:26:01.119609    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9da6fe34_d6b1_4b92_a067_e2677f71bca9.slice/crio-3853204736b987171cba6e482e492fefedd3961087dfc5f7e136a964d516d994 WatchSource:0}: Error finding container 3853204736b987171cba6e482e492fefedd3961087dfc5f7e136a964d516d994: Status 404 returned error can't find the container with id 3853204736b987171cba6e482e492fefedd3961087dfc5f7e136a964d516d994
Sep 11 23:26:01.902402 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:01.902365    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-12fb6b/distinct-2-w0-12fb6b-kserve-56c4887795-bf8sz" event={"ID":"9da6fe34-d6b1-4b92-a067-e2677f71bca9","Type":"ContainerStarted","Data":"328ba10a98d3d824ec19b37471e0a837842a2a0f8bd38a173fad6257b723a171"}
Sep 11 23:26:01.902781 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:01.902411    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-12fb6b/distinct-2-w0-12fb6b-kserve-56c4887795-bf8sz" event={"ID":"9da6fe34-d6b1-4b92-a067-e2677f71bca9","Type":"ContainerStarted","Data":"3853204736b987171cba6e482e492fefedd3961087dfc5f7e136a964d516d994"}
Sep 11 23:26:01.922167 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:01.922103    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-12fb6b/distinct-2-w0-12fb6b-kserve-56c4887795-bf8sz" podStartSLOduration=1.9220871590000002 podStartE2EDuration="1.922087159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 23:26:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:26:01.921793433 +0000 UTC m=+1485.954922649" watchObservedRunningTime="2026-09-11 23:26:01.922087159 +0000 UTC m=+1485.955216374"
Sep 11 23:26:08.311045 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:08.311013    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw"]
Sep 11 23:26:08.314860 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:08.314837    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw"
Sep 11 23:26:08.318021 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:08.317994    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-da9451-openshift-default-dockercfg-2nhzb\""
Sep 11 23:26:08.330180 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:08.330128    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw"]
Sep 11 23:26:08.384903 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:08.384868    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c437d61-e321-42e7-bb35-9dd7119a09ac-credential-socket\") pod \"e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw\" (UID: \"8c437d61-e321-42e7-bb35-9dd7119a09ac\") " pod="openshift-ingress/e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw"
Sep 11 23:26:08.384903 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:08.384905    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c437d61-e321-42e7-bb35-9dd7119a09ac-istio-podinfo\") pod \"e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw\" (UID: \"8c437d61-e321-42e7-bb35-9dd7119a09ac\") " pod="openshift-ingress/e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw"
Sep 11 23:26:08.385258 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:08.384980    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c437d61-e321-42e7-bb35-9dd7119a09ac-istio-envoy\") pod \"e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw\" (UID: \"8c437d61-e321-42e7-bb35-9dd7119a09ac\") " pod="openshift-ingress/e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw"
Sep 11 23:26:08.385258 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:08.385072    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c437d61-e321-42e7-bb35-9dd7119a09ac-istio-data\") pod \"e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw\" (UID: \"8c437d61-e321-42e7-bb35-9dd7119a09ac\") " pod="openshift-ingress/e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw"
Sep 11 23:26:08.385258 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:08.385107    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44g9l\" (UniqueName: \"kubernetes.io/projected/8c437d61-e321-42e7-bb35-9dd7119a09ac-kube-api-access-44g9l\") pod \"e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw\" (UID: \"8c437d61-e321-42e7-bb35-9dd7119a09ac\") " pod="openshift-ingress/e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw"
Sep 11 23:26:08.385258 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:08.385175    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c437d61-e321-42e7-bb35-9dd7119a09ac-workload-certs\") pod \"e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw\" (UID: \"8c437d61-e321-42e7-bb35-9dd7119a09ac\") " pod="openshift-ingress/e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw"
Sep 11 23:26:08.385258 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:08.385214    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c437d61-e321-42e7-bb35-9dd7119a09ac-istiod-ca-cert\") pod \"e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw\" (UID: \"8c437d61-e321-42e7-bb35-9dd7119a09ac\") " pod="openshift-ingress/e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw"
Sep 11 23:26:08.385258 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:08.385245    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c437d61-e321-42e7-bb35-9dd7119a09ac-istio-token\") pod \"e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw\" (UID: \"8c437d61-e321-42e7-bb35-9dd7119a09ac\") " pod="openshift-ingress/e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw"
Sep 11 23:26:08.385534 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:08.385313    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c437d61-e321-42e7-bb35-9dd7119a09ac-workload-socket\") pod \"e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw\" (UID: \"8c437d61-e321-42e7-bb35-9dd7119a09ac\") " pod="openshift-ingress/e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw"
Sep 11 23:26:08.486277 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:08.486237    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c437d61-e321-42e7-bb35-9dd7119a09ac-workload-socket\") pod \"e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw\" (UID: \"8c437d61-e321-42e7-bb35-9dd7119a09ac\") " pod="openshift-ingress/e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw"
Sep 11 23:26:08.486478 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:08.486303    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c437d61-e321-42e7-bb35-9dd7119a09ac-credential-socket\") pod \"e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw\" (UID: \"8c437d61-e321-42e7-bb35-9dd7119a09ac\") " pod="openshift-ingress/e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw"
Sep 11 23:26:08.486478 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:08.486332    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c437d61-e321-42e7-bb35-9dd7119a09ac-istio-podinfo\") pod \"e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw\" (UID: \"8c437d61-e321-42e7-bb35-9dd7119a09ac\") " pod="openshift-ingress/e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw"
Sep 11 23:26:08.486478 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:08.486362    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c437d61-e321-42e7-bb35-9dd7119a09ac-istio-envoy\") pod \"e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw\" (UID: \"8c437d61-e321-42e7-bb35-9dd7119a09ac\") " pod="openshift-ingress/e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw"
Sep 11 23:26:08.486478 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:08.486413    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c437d61-e321-42e7-bb35-9dd7119a09ac-istio-data\") pod \"e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw\" (UID: \"8c437d61-e321-42e7-bb35-9dd7119a09ac\") " pod="openshift-ingress/e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw"
Sep 11 23:26:08.486478 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:08.486437    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44g9l\" (UniqueName: \"kubernetes.io/projected/8c437d61-e321-42e7-bb35-9dd7119a09ac-kube-api-access-44g9l\") pod \"e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw\" (UID: \"8c437d61-e321-42e7-bb35-9dd7119a09ac\") " pod="openshift-ingress/e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw"
Sep 11 23:26:08.486738 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:08.486481    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c437d61-e321-42e7-bb35-9dd7119a09ac-workload-certs\") pod \"e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw\" (UID: \"8c437d61-e321-42e7-bb35-9dd7119a09ac\") " pod="openshift-ingress/e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw"
Sep 11 23:26:08.486738 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:08.486520    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c437d61-e321-42e7-bb35-9dd7119a09ac-istiod-ca-cert\") pod \"e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw\" (UID: \"8c437d61-e321-42e7-bb35-9dd7119a09ac\") " pod="openshift-ingress/e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw"
Sep 11 23:26:08.486931 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:08.486904    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c437d61-e321-42e7-bb35-9dd7119a09ac-workload-socket\") pod \"e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw\" (UID: \"8c437d61-e321-42e7-bb35-9dd7119a09ac\") " pod="openshift-ingress/e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw"
Sep 11 23:26:08.487607 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:08.487569    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c437d61-e321-42e7-bb35-9dd7119a09ac-istio-data\") pod \"e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw\" (UID: \"8c437d61-e321-42e7-bb35-9dd7119a09ac\") " pod="openshift-ingress/e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw"
Sep 11 23:26:08.487769 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:08.487736    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c437d61-e321-42e7-bb35-9dd7119a09ac-istiod-ca-cert\") pod \"e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw\" (UID: \"8c437d61-e321-42e7-bb35-9dd7119a09ac\") " pod="openshift-ingress/e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw"
Sep 11 23:26:08.487903 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:08.486625    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c437d61-e321-42e7-bb35-9dd7119a09ac-istio-token\") pod \"e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw\" (UID: \"8c437d61-e321-42e7-bb35-9dd7119a09ac\") " pod="openshift-ingress/e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw"
Sep 11 23:26:08.489563 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:08.489538    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c437d61-e321-42e7-bb35-9dd7119a09ac-workload-certs\") pod \"e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw\" (UID: \"8c437d61-e321-42e7-bb35-9dd7119a09ac\") " pod="openshift-ingress/e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw"
Sep 11 23:26:08.491200 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:08.489716    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c437d61-e321-42e7-bb35-9dd7119a09ac-credential-socket\") pod \"e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw\" (UID: \"8c437d61-e321-42e7-bb35-9dd7119a09ac\") " pod="openshift-ingress/e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw"
Sep 11 23:26:08.493705 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:08.491995    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c437d61-e321-42e7-bb35-9dd7119a09ac-istio-podinfo\") pod \"e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw\" (UID: \"8c437d61-e321-42e7-bb35-9dd7119a09ac\") " pod="openshift-ingress/e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw"
Sep 11 23:26:08.499890 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:08.498238    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44g9l\" (UniqueName: \"kubernetes.io/projected/8c437d61-e321-42e7-bb35-9dd7119a09ac-kube-api-access-44g9l\") pod \"e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw\" (UID: \"8c437d61-e321-42e7-bb35-9dd7119a09ac\") " pod="openshift-ingress/e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw"
Sep 11 23:26:08.499890 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:08.498315    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c437d61-e321-42e7-bb35-9dd7119a09ac-istio-envoy\") pod \"e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw\" (UID: \"8c437d61-e321-42e7-bb35-9dd7119a09ac\") " pod="openshift-ingress/e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw"
Sep 11 23:26:08.500788 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:08.500756    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c437d61-e321-42e7-bb35-9dd7119a09ac-istio-token\") pod \"e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw\" (UID: \"8c437d61-e321-42e7-bb35-9dd7119a09ac\") " pod="openshift-ingress/e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw"
Sep 11 23:26:08.628123 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:08.628014    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw"
Sep 11 23:26:08.924293 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:08.924256    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw"]
Sep 11 23:26:08.925606 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:26:08.925575    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c437d61_e321_42e7_bb35_9dd7119a09ac.slice/crio-cdd27f0c367022eb40264c25305ee96e8b24724c0e980af8da3190545be8ebdb WatchSource:0}: Error finding container cdd27f0c367022eb40264c25305ee96e8b24724c0e980af8da3190545be8ebdb: Status 404 returned error can't find the container with id cdd27f0c367022eb40264c25305ee96e8b24724c0e980af8da3190545be8ebdb
Sep 11 23:26:08.928403 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:08.928357    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:26:08.928512 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:08.928439    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:26:08.928512 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:08.928506    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:26:08.957565 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:08.957536    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw" event={"ID":"8c437d61-e321-42e7-bb35-9dd7119a09ac","Type":"ContainerStarted","Data":"cdd27f0c367022eb40264c25305ee96e8b24724c0e980af8da3190545be8ebdb"}
Sep 11 23:26:09.965870 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:09.965829    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw" event={"ID":"8c437d61-e321-42e7-bb35-9dd7119a09ac","Type":"ContainerStarted","Data":"266652491fac3416440db75183e30f6877b828bc9b7e33b385644284ed3f89fa"}
Sep 11 23:26:09.988088 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:09.988035    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw" podStartSLOduration=1.988018704 podStartE2EDuration="1.988018704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 23:26:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:26:09.986234118 +0000 UTC m=+1494.019363333" watchObservedRunningTime="2026-09-11 23:26:09.988018704 +0000 UTC m=+1494.021147919"
Sep 11 23:26:10.628170 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:10.628116    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw"
Sep 11 23:26:10.634453 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:10.634427    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw"
Sep 11 23:26:10.822096 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:10.822000    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-12fb6b/distinct-2-w0-12fb6b-kserve-56c4887795-bf8sz"
Sep 11 23:26:10.822296 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:10.822097    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-12fb6b/distinct-2-w0-12fb6b-kserve-56c4887795-bf8sz"
Sep 11 23:26:10.834870 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:10.834839    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-12fb6b/distinct-2-w0-12fb6b-kserve-56c4887795-bf8sz"
Sep 11 23:26:10.973774 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:10.973740    2546 generic.go:356] "Generic (PLEG): container finished" podID="bb3c41dd-091c-4f4e-94e5-55d389088418" containerID="5f14e88f52446d1883b5f8e093a7154e7a38ed1b80238ce72f2c461a26eed13b" exitCode=0
Sep 11 23:26:10.974197 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:10.973819    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k" event={"ID":"bb3c41dd-091c-4f4e-94e5-55d389088418","Type":"ContainerDied","Data":"5f14e88f52446d1883b5f8e093a7154e7a38ed1b80238ce72f2c461a26eed13b"}
Sep 11 23:26:10.974549 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:10.974362    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw"
Sep 11 23:26:10.974549 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:10.974363    2546 scope.go:117] "RemoveContainer" containerID="5f14e88f52446d1883b5f8e093a7154e7a38ed1b80238ce72f2c461a26eed13b"
Sep 11 23:26:10.975464 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:10.975447    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw"
Sep 11 23:26:10.975810 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:10.975784    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:26:10.975891 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:10.975853    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:26:10.975891 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:10.975887    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:26:10.985740 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:10.985717    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-12fb6b/distinct-2-w0-12fb6b-kserve-56c4887795-bf8sz"
Sep 11 23:26:11.984214 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:11.984177    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k" event={"ID":"bb3c41dd-091c-4f4e-94e5-55d389088418","Type":"ContainerStarted","Data":"44e6e308a812f60817f5b350e9f9975444ab25d327b67e84059aa329a97dd152"}
Sep 11 23:26:12.477293 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:12.477256    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k"
Sep 11 23:26:12.477562 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:12.477541    2546 patch_prober.go:28] interesting pod/e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Sep 11 23:26:12.477621 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:12.477599    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k" podUID="bb3c41dd-091c-4f4e-94e5-55d389088418" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Sep 11 23:26:13.476936 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:13.476898    2546 patch_prober.go:28] interesting pod/e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Sep 11 23:26:13.477390 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:13.476957    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k" podUID="bb3c41dd-091c-4f4e-94e5-55d389088418" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Sep 11 23:26:14.477250 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:14.477212    2546 patch_prober.go:28] interesting pod/e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Sep 11 23:26:14.477652 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:14.477269    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k" podUID="bb3c41dd-091c-4f4e-94e5-55d389088418" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Sep 11 23:26:15.477371 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:15.477334    2546 patch_prober.go:28] interesting pod/e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Sep 11 23:26:15.477770 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:15.477394    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k" podUID="bb3c41dd-091c-4f4e-94e5-55d389088418" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Sep 11 23:26:16.477147 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:16.477106    2546 patch_prober.go:28] interesting pod/e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Sep 11 23:26:16.477360 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:16.477215    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k" podUID="bb3c41dd-091c-4f4e-94e5-55d389088418" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Sep 11 23:26:16.594927 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:16.594900    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7244_6b0a8af3-932c-4091-b7a0-180fe1751a3a/ovn-acl-logging/0.log"
Sep 11 23:26:16.595374 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:16.594901    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7244_6b0a8af3-932c-4091-b7a0-180fe1751a3a/ovn-acl-logging/0.log"
Sep 11 23:26:16.664322 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:16.664293    2546 scope.go:117] "RemoveContainer" containerID="2b84fc5efe24af4b23f526eafd61dde7c9cdb697789bf077b21b8dbd59eb21b3"
Sep 11 23:26:17.481545 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:17.481515    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k"
Sep 11 23:26:17.481832 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:17.481811    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k"
Sep 11 23:26:17.482497 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:17.482476    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k"
Sep 11 23:26:18.293320 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:18.293278    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"]
Sep 11 23:26:39.422877 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:39.422796    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-b96fb0e1/e2e-lifecycle-model-b96fb0e1-kserve-5688546d8c-4fdqw"]
Sep 11 23:26:39.429222 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:39.429186    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-b96fb0e1/e2e-lifecycle-model-b96fb0e1-kserve-5688546d8c-4fdqw"
Sep 11 23:26:39.440750 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:39.440695    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-b96fb0e1/e2e-lifecycle-model-b96fb0e1-kserve-5688546d8c-4fdqw"]
Sep 11 23:26:39.440949 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:39.440929    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-b96fb0e1\"/\"kube-root-ca.crt\""
Sep 11 23:26:39.441254 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:39.441229    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-b96fb0e1\"/\"openshift-service-ca.crt\""
Sep 11 23:26:39.441473 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:39.441453    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-b96fb0e1\"/\"default-dockercfg-hw2jn\""
Sep 11 23:26:39.441670 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:39.441648    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-b96fb0e1\"/\"e2e-lifecycle-model-b96fb0e1-kserve-self-signed-certs\""
Sep 11 23:26:39.548854 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:39.548805    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14fee1b9-b0f8-4fa4-a1f2-61b89c4d9daf-dshm\") pod \"e2e-lifecycle-model-b96fb0e1-kserve-5688546d8c-4fdqw\" (UID: \"14fee1b9-b0f8-4fa4-a1f2-61b89c4d9daf\") " pod="ai-tenant-e2e-mt-b96fb0e1/e2e-lifecycle-model-b96fb0e1-kserve-5688546d8c-4fdqw"
Sep 11 23:26:39.549085 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:39.548868    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14fee1b9-b0f8-4fa4-a1f2-61b89c4d9daf-tmp-dir\") pod \"e2e-lifecycle-model-b96fb0e1-kserve-5688546d8c-4fdqw\" (UID: \"14fee1b9-b0f8-4fa4-a1f2-61b89c4d9daf\") " pod="ai-tenant-e2e-mt-b96fb0e1/e2e-lifecycle-model-b96fb0e1-kserve-5688546d8c-4fdqw"
Sep 11 23:26:39.549085 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:39.548917    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14fee1b9-b0f8-4fa4-a1f2-61b89c4d9daf-model-cache\") pod \"e2e-lifecycle-model-b96fb0e1-kserve-5688546d8c-4fdqw\" (UID: \"14fee1b9-b0f8-4fa4-a1f2-61b89c4d9daf\") " pod="ai-tenant-e2e-mt-b96fb0e1/e2e-lifecycle-model-b96fb0e1-kserve-5688546d8c-4fdqw"
Sep 11 23:26:39.549085 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:39.548958    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n85d8\" (UniqueName: \"kubernetes.io/projected/14fee1b9-b0f8-4fa4-a1f2-61b89c4d9daf-kube-api-access-n85d8\") pod \"e2e-lifecycle-model-b96fb0e1-kserve-5688546d8c-4fdqw\" (UID: \"14fee1b9-b0f8-4fa4-a1f2-61b89c4d9daf\") " pod="ai-tenant-e2e-mt-b96fb0e1/e2e-lifecycle-model-b96fb0e1-kserve-5688546d8c-4fdqw"
Sep 11 23:26:39.549302 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:39.549096    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14fee1b9-b0f8-4fa4-a1f2-61b89c4d9daf-tls-certs\") pod \"e2e-lifecycle-model-b96fb0e1-kserve-5688546d8c-4fdqw\" (UID: \"14fee1b9-b0f8-4fa4-a1f2-61b89c4d9daf\") " pod="ai-tenant-e2e-mt-b96fb0e1/e2e-lifecycle-model-b96fb0e1-kserve-5688546d8c-4fdqw"
Sep 11 23:26:39.549302 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:39.549186    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14fee1b9-b0f8-4fa4-a1f2-61b89c4d9daf-home\") pod \"e2e-lifecycle-model-b96fb0e1-kserve-5688546d8c-4fdqw\" (UID: \"14fee1b9-b0f8-4fa4-a1f2-61b89c4d9daf\") " pod="ai-tenant-e2e-mt-b96fb0e1/e2e-lifecycle-model-b96fb0e1-kserve-5688546d8c-4fdqw"
Sep 11 23:26:39.650925 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:39.650875    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14fee1b9-b0f8-4fa4-a1f2-61b89c4d9daf-model-cache\") pod \"e2e-lifecycle-model-b96fb0e1-kserve-5688546d8c-4fdqw\" (UID: \"14fee1b9-b0f8-4fa4-a1f2-61b89c4d9daf\") " pod="ai-tenant-e2e-mt-b96fb0e1/e2e-lifecycle-model-b96fb0e1-kserve-5688546d8c-4fdqw"
Sep 11 23:26:39.651703 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:39.651677    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n85d8\" (UniqueName: \"kubernetes.io/projected/14fee1b9-b0f8-4fa4-a1f2-61b89c4d9daf-kube-api-access-n85d8\") pod \"e2e-lifecycle-model-b96fb0e1-kserve-5688546d8c-4fdqw\" (UID: \"14fee1b9-b0f8-4fa4-a1f2-61b89c4d9daf\") " pod="ai-tenant-e2e-mt-b96fb0e1/e2e-lifecycle-model-b96fb0e1-kserve-5688546d8c-4fdqw"
Sep 11 23:26:39.651876 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:39.651606    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14fee1b9-b0f8-4fa4-a1f2-61b89c4d9daf-model-cache\") pod \"e2e-lifecycle-model-b96fb0e1-kserve-5688546d8c-4fdqw\" (UID: \"14fee1b9-b0f8-4fa4-a1f2-61b89c4d9daf\") " pod="ai-tenant-e2e-mt-b96fb0e1/e2e-lifecycle-model-b96fb0e1-kserve-5688546d8c-4fdqw"
Sep 11 23:26:39.652081 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:39.652056    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14fee1b9-b0f8-4fa4-a1f2-61b89c4d9daf-tls-certs\") pod \"e2e-lifecycle-model-b96fb0e1-kserve-5688546d8c-4fdqw\" (UID: \"14fee1b9-b0f8-4fa4-a1f2-61b89c4d9daf\") " pod="ai-tenant-e2e-mt-b96fb0e1/e2e-lifecycle-model-b96fb0e1-kserve-5688546d8c-4fdqw"
Sep 11 23:26:39.652213 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:39.652119    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14fee1b9-b0f8-4fa4-a1f2-61b89c4d9daf-home\") pod \"e2e-lifecycle-model-b96fb0e1-kserve-5688546d8c-4fdqw\" (UID: \"14fee1b9-b0f8-4fa4-a1f2-61b89c4d9daf\") " pod="ai-tenant-e2e-mt-b96fb0e1/e2e-lifecycle-model-b96fb0e1-kserve-5688546d8c-4fdqw"
Sep 11 23:26:39.652213 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:39.652168    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14fee1b9-b0f8-4fa4-a1f2-61b89c4d9daf-dshm\") pod \"e2e-lifecycle-model-b96fb0e1-kserve-5688546d8c-4fdqw\" (UID: \"14fee1b9-b0f8-4fa4-a1f2-61b89c4d9daf\") " pod="ai-tenant-e2e-mt-b96fb0e1/e2e-lifecycle-model-b96fb0e1-kserve-5688546d8c-4fdqw"
Sep 11 23:26:39.652213 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:39.652191    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14fee1b9-b0f8-4fa4-a1f2-61b89c4d9daf-tmp-dir\") pod \"e2e-lifecycle-model-b96fb0e1-kserve-5688546d8c-4fdqw\" (UID: \"14fee1b9-b0f8-4fa4-a1f2-61b89c4d9daf\") " pod="ai-tenant-e2e-mt-b96fb0e1/e2e-lifecycle-model-b96fb0e1-kserve-5688546d8c-4fdqw"
Sep 11 23:26:39.652446 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:39.652426    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14fee1b9-b0f8-4fa4-a1f2-61b89c4d9daf-home\") pod \"e2e-lifecycle-model-b96fb0e1-kserve-5688546d8c-4fdqw\" (UID: \"14fee1b9-b0f8-4fa4-a1f2-61b89c4d9daf\") " pod="ai-tenant-e2e-mt-b96fb0e1/e2e-lifecycle-model-b96fb0e1-kserve-5688546d8c-4fdqw"
Sep 11 23:26:39.652570 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:39.652457    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14fee1b9-b0f8-4fa4-a1f2-61b89c4d9daf-tmp-dir\") pod \"e2e-lifecycle-model-b96fb0e1-kserve-5688546d8c-4fdqw\" (UID: \"14fee1b9-b0f8-4fa4-a1f2-61b89c4d9daf\") " pod="ai-tenant-e2e-mt-b96fb0e1/e2e-lifecycle-model-b96fb0e1-kserve-5688546d8c-4fdqw"
Sep 11 23:26:39.655101 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:39.655074    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14fee1b9-b0f8-4fa4-a1f2-61b89c4d9daf-dshm\") pod \"e2e-lifecycle-model-b96fb0e1-kserve-5688546d8c-4fdqw\" (UID: \"14fee1b9-b0f8-4fa4-a1f2-61b89c4d9daf\") " pod="ai-tenant-e2e-mt-b96fb0e1/e2e-lifecycle-model-b96fb0e1-kserve-5688546d8c-4fdqw"
Sep 11 23:26:39.655101 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:39.655090    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14fee1b9-b0f8-4fa4-a1f2-61b89c4d9daf-tls-certs\") pod \"e2e-lifecycle-model-b96fb0e1-kserve-5688546d8c-4fdqw\" (UID: \"14fee1b9-b0f8-4fa4-a1f2-61b89c4d9daf\") " pod="ai-tenant-e2e-mt-b96fb0e1/e2e-lifecycle-model-b96fb0e1-kserve-5688546d8c-4fdqw"
Sep 11 23:26:39.662514 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:39.662491    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n85d8\" (UniqueName: \"kubernetes.io/projected/14fee1b9-b0f8-4fa4-a1f2-61b89c4d9daf-kube-api-access-n85d8\") pod \"e2e-lifecycle-model-b96fb0e1-kserve-5688546d8c-4fdqw\" (UID: \"14fee1b9-b0f8-4fa4-a1f2-61b89c4d9daf\") " pod="ai-tenant-e2e-mt-b96fb0e1/e2e-lifecycle-model-b96fb0e1-kserve-5688546d8c-4fdqw"
Sep 11 23:26:39.751999 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:39.751889    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-b96fb0e1/e2e-lifecycle-model-b96fb0e1-kserve-5688546d8c-4fdqw"
Sep 11 23:26:40.057248 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:40.057222    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-b96fb0e1/e2e-lifecycle-model-b96fb0e1-kserve-5688546d8c-4fdqw"]
Sep 11 23:26:40.059198 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:26:40.059010    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod14fee1b9_b0f8_4fa4_a1f2_61b89c4d9daf.slice/crio-c78ffd76732751a5999caef74ce712dc272f87e7d325957fb72a1cf4d91a2754 WatchSource:0}: Error finding container c78ffd76732751a5999caef74ce712dc272f87e7d325957fb72a1cf4d91a2754: Status 404 returned error can't find the container with id c78ffd76732751a5999caef74ce712dc272f87e7d325957fb72a1cf4d91a2754
Sep 11 23:26:40.181474 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:40.181440    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-b96fb0e1/e2e-lifecycle-model-b96fb0e1-kserve-5688546d8c-4fdqw" event={"ID":"14fee1b9-b0f8-4fa4-a1f2-61b89c4d9daf","Type":"ContainerStarted","Data":"a26835f1611c9022531a8d6c8d4efb495eba246fb7bfa5eb3c616b2f4a6872f2"}
Sep 11 23:26:40.181659 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:40.181482    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-b96fb0e1/e2e-lifecycle-model-b96fb0e1-kserve-5688546d8c-4fdqw" event={"ID":"14fee1b9-b0f8-4fa4-a1f2-61b89c4d9daf","Type":"ContainerStarted","Data":"c78ffd76732751a5999caef74ce712dc272f87e7d325957fb72a1cf4d91a2754"}
Sep 11 23:26:40.201074 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:40.201021    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-b96fb0e1/e2e-lifecycle-model-b96fb0e1-kserve-5688546d8c-4fdqw" podStartSLOduration=1.201006385 podStartE2EDuration="1.201006385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 23:26:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:26:40.199297371 +0000 UTC m=+1524.232426589" watchObservedRunningTime="2026-09-11 23:26:40.201006385 +0000 UTC m=+1524.234135600"
Sep 11 23:26:47.236108 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:47.236021    2546 generic.go:356] "Generic (PLEG): container finished" podID="8c437d61-e321-42e7-bb35-9dd7119a09ac" containerID="266652491fac3416440db75183e30f6877b828bc9b7e33b385644284ed3f89fa" exitCode=0
Sep 11 23:26:47.236516 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:47.236100    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw" event={"ID":"8c437d61-e321-42e7-bb35-9dd7119a09ac","Type":"ContainerDied","Data":"266652491fac3416440db75183e30f6877b828bc9b7e33b385644284ed3f89fa"}
Sep 11 23:26:47.236580 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:47.236566    2546 scope.go:117] "RemoveContainer" containerID="266652491fac3416440db75183e30f6877b828bc9b7e33b385644284ed3f89fa"
Sep 11 23:26:47.237811 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:47.237778    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:26:47.237878 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:47.237854    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:26:47.237929 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:47.237904    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:26:48.244694 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:48.244653    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw" event={"ID":"8c437d61-e321-42e7-bb35-9dd7119a09ac","Type":"ContainerStarted","Data":"e602ccf807868d8aba6da15ef037144f5fc9e69ff8660db0ea2429d1b66ba092"}
Sep 11 23:26:48.628741 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:48.628646    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw"
Sep 11 23:26:48.628901 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:48.628836    2546 patch_prober.go:28] interesting pod/e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Sep 11 23:26:48.628943 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:48.628902    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw" podUID="8c437d61-e321-42e7-bb35-9dd7119a09ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Sep 11 23:26:49.631918 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:49.631878    2546 patch_prober.go:28] interesting pod/e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Sep 11 23:26:49.632544 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:49.632507    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw" podUID="8c437d61-e321-42e7-bb35-9dd7119a09ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Sep 11 23:26:49.752168 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:49.752122    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-b96fb0e1/e2e-lifecycle-model-b96fb0e1-kserve-5688546d8c-4fdqw"
Sep 11 23:26:49.752429 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:49.752407    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-b96fb0e1/e2e-lifecycle-model-b96fb0e1-kserve-5688546d8c-4fdqw"
Sep 11 23:26:49.765114 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:49.765083    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-b96fb0e1/e2e-lifecycle-model-b96fb0e1-kserve-5688546d8c-4fdqw"
Sep 11 23:26:50.274235 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:50.274196    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-b96fb0e1/e2e-lifecycle-model-b96fb0e1-kserve-5688546d8c-4fdqw"
Sep 11 23:26:50.628764 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:50.628654    2546 patch_prober.go:28] interesting pod/e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Sep 11 23:26:50.628764 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:50.628734    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw" podUID="8c437d61-e321-42e7-bb35-9dd7119a09ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Sep 11 23:26:51.629437 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:51.629396    2546 patch_prober.go:28] interesting pod/e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Sep 11 23:26:51.630060 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:51.630026    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw" podUID="8c437d61-e321-42e7-bb35-9dd7119a09ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Sep 11 23:26:52.628814 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:52.628767    2546 patch_prober.go:28] interesting pod/e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Sep 11 23:26:52.629044 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:52.628838    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw" podUID="8c437d61-e321-42e7-bb35-9dd7119a09ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Sep 11 23:26:53.628835 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:53.628790    2546 patch_prober.go:28] interesting pod/e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Sep 11 23:26:53.629396 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:53.628879    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw" podUID="8c437d61-e321-42e7-bb35-9dd7119a09ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Sep 11 23:26:53.794639 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:53.794608    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"]
Sep 11 23:26:54.629352 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:54.629309    2546 patch_prober.go:28] interesting pod/e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Sep 11 23:26:54.629982 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:54.629950    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw" podUID="8c437d61-e321-42e7-bb35-9dd7119a09ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Sep 11 23:26:55.633984 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:55.633945    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw"
Sep 11 23:26:55.634479 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:55.634340    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw"
Sep 11 23:26:55.635134 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:26:55.635119    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw"
Sep 11 23:27:16.706848 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:27:16.706815    2546 scope.go:117] "RemoveContainer" containerID="a6ae1bc7fd52882563b22bf5def45f50e527dd5c4f2e4d6be6afafcaa1c77302"
Sep 11 23:27:16.716111 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:27:16.716091    2546 scope.go:117] "RemoveContainer" containerID="c66c36a1680cab71d0b006ee4a574a011b2f00b51689f6733d52bdbf84b0f965"
Sep 11 23:27:16.725208 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:27:16.725188    2546 scope.go:117] "RemoveContainer" containerID="fff138816c6423bd5b24402d47336908f723eadeaed54805ea5432701347152f"
Sep 11 23:27:28.889675 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:27:28.889631    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"]
Sep 11 23:27:35.592316 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:27:35.592275    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b96fb0e1-wpzsb"]
Sep 11 23:27:35.598037 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:27:35.598015    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b96fb0e1-wpzsb"
Sep 11 23:27:35.602646 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:27:35.602616    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 23:27:35.602938 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:27:35.602912    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vrx7m\""
Sep 11 23:27:35.604101 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:27:35.604072    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b96fb0e1-wpzsb"]
Sep 11 23:27:35.776465 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:27:35.776428    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ddd3b74-bca5-4776-8b7e-f12f2f2dd8c7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-b96fb0e1-wpzsb\" (UID: \"5ddd3b74-bca5-4776-8b7e-f12f2f2dd8c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b96fb0e1-wpzsb"
Sep 11 23:27:35.877733 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:27:35.877625    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ddd3b74-bca5-4776-8b7e-f12f2f2dd8c7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-b96fb0e1-wpzsb\" (UID: \"5ddd3b74-bca5-4776-8b7e-f12f2f2dd8c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b96fb0e1-wpzsb"
Sep 11 23:27:35.878336 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:27:35.878316    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ddd3b74-bca5-4776-8b7e-f12f2f2dd8c7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-b96fb0e1-wpzsb\" (UID: \"5ddd3b74-bca5-4776-8b7e-f12f2f2dd8c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b96fb0e1-wpzsb"
Sep 11 23:27:35.911293 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:27:35.911259    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b96fb0e1-wpzsb"
Sep 11 23:27:36.187202 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:27:36.187172    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b96fb0e1-wpzsb"]
Sep 11 23:27:36.188185 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:27:36.188124    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ddd3b74_bca5_4776_8b7e_f12f2f2dd8c7.slice/crio-055b12afabaea165d4c4b94f8cf6ea8fee122617ec483390177571afbb13ba98 WatchSource:0}: Error finding container 055b12afabaea165d4c4b94f8cf6ea8fee122617ec483390177571afbb13ba98: Status 404 returned error can't find the container with id 055b12afabaea165d4c4b94f8cf6ea8fee122617ec483390177571afbb13ba98
Sep 11 23:27:36.600383 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:27:36.600339    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b96fb0e1-wpzsb" event={"ID":"5ddd3b74-bca5-4776-8b7e-f12f2f2dd8c7","Type":"ContainerStarted","Data":"78ce1543cc85dcb8ae5cbdd1308a2f714ba8f6755fc4437dac7bb881f16edc6e"}
Sep 11 23:27:36.600762 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:27:36.600439    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b96fb0e1-wpzsb" event={"ID":"5ddd3b74-bca5-4776-8b7e-f12f2f2dd8c7","Type":"ContainerStarted","Data":"055b12afabaea165d4c4b94f8cf6ea8fee122617ec483390177571afbb13ba98"}
Sep 11 23:27:36.617522 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:27:36.617472    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b96fb0e1-wpzsb" podStartSLOduration=1.617457252 podStartE2EDuration="1.617457252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 23:27:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:27:36.616501811 +0000 UTC m=+1580.649631030" watchObservedRunningTime="2026-09-11 23:27:36.617457252 +0000 UTC m=+1580.650586537"
Sep 11 23:27:37.608097 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:27:37.608065    2546 generic.go:356] "Generic (PLEG): container finished" podID="5ddd3b74-bca5-4776-8b7e-f12f2f2dd8c7" containerID="78ce1543cc85dcb8ae5cbdd1308a2f714ba8f6755fc4437dac7bb881f16edc6e" exitCode=0
Sep 11 23:27:37.608525 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:27:37.608142    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b96fb0e1-wpzsb" event={"ID":"5ddd3b74-bca5-4776-8b7e-f12f2f2dd8c7","Type":"ContainerDied","Data":"78ce1543cc85dcb8ae5cbdd1308a2f714ba8f6755fc4437dac7bb881f16edc6e"}
Sep 11 23:27:38.764023 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:27:38.763993    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b96fb0e1-wpzsb"
Sep 11 23:27:38.914708 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:27:38.914675    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ddd3b74-bca5-4776-8b7e-f12f2f2dd8c7-maas-api-service-ca\") pod \"5ddd3b74-bca5-4776-8b7e-f12f2f2dd8c7\" (UID: \"5ddd3b74-bca5-4776-8b7e-f12f2f2dd8c7\") "
Sep 11 23:27:38.915067 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:27:38.915043    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ddd3b74-bca5-4776-8b7e-f12f2f2dd8c7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5ddd3b74-bca5-4776-8b7e-f12f2f2dd8c7" (UID: "5ddd3b74-bca5-4776-8b7e-f12f2f2dd8c7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 23:27:39.016721 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:27:39.016680    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ddd3b74-bca5-4776-8b7e-f12f2f2dd8c7-maas-api-service-ca\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:27:39.623504 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:27:39.623467    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b96fb0e1-wpzsb"
Sep 11 23:27:39.623716 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:27:39.623501    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b96fb0e1-wpzsb" event={"ID":"5ddd3b74-bca5-4776-8b7e-f12f2f2dd8c7","Type":"ContainerDied","Data":"055b12afabaea165d4c4b94f8cf6ea8fee122617ec483390177571afbb13ba98"}
Sep 11 23:27:39.623716 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:27:39.623549    2546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="055b12afabaea165d4c4b94f8cf6ea8fee122617ec483390177571afbb13ba98"
Sep 11 23:27:51.579832 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:27:51.579767    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b96fb0e1-wpzsb"]
Sep 11 23:27:51.583682 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:27:51.583657    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b96fb0e1-wpzsb"]
Sep 11 23:27:52.495901 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:27:52.495864    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ddd3b74-bca5-4776-8b7e-f12f2f2dd8c7" path="/var/lib/kubelet/pods/5ddd3b74-bca5-4776-8b7e-f12f2f2dd8c7/volumes"
Sep 11 23:27:59.545955 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:27:59.545921    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-b96fb0e1/e2e-lifecycle-model-b96fb0e1-kserve-5688546d8c-4fdqw"]
Sep 11 23:28:03.786892 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:28:03.786855    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"]
Sep 11 23:28:14.567458 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:28:14.567344    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-b96fb0e1/e2e-lifecycle-model-b96fb0e1-kserve-5688546d8c-4fdqw" podUID="14fee1b9-b0f8-4fa4-a1f2-61b89c4d9daf" containerName="main" containerID="cri-o://a26835f1611c9022531a8d6c8d4efb495eba246fb7bfa5eb3c616b2f4a6872f2" gracePeriod=45
Sep 11 23:28:14.835291 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:28:14.835222    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-b96fb0e1/e2e-lifecycle-model-b96fb0e1-kserve-5688546d8c-4fdqw"
Sep 11 23:28:14.874735 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:28:14.874701    2546 generic.go:356] "Generic (PLEG): container finished" podID="14fee1b9-b0f8-4fa4-a1f2-61b89c4d9daf" containerID="a26835f1611c9022531a8d6c8d4efb495eba246fb7bfa5eb3c616b2f4a6872f2" exitCode=0
Sep 11 23:28:14.874936 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:28:14.874748    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-b96fb0e1/e2e-lifecycle-model-b96fb0e1-kserve-5688546d8c-4fdqw"
Sep 11 23:28:14.874936 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:28:14.874785    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-b96fb0e1/e2e-lifecycle-model-b96fb0e1-kserve-5688546d8c-4fdqw" event={"ID":"14fee1b9-b0f8-4fa4-a1f2-61b89c4d9daf","Type":"ContainerDied","Data":"a26835f1611c9022531a8d6c8d4efb495eba246fb7bfa5eb3c616b2f4a6872f2"}
Sep 11 23:28:14.874936 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:28:14.874831    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-b96fb0e1/e2e-lifecycle-model-b96fb0e1-kserve-5688546d8c-4fdqw" event={"ID":"14fee1b9-b0f8-4fa4-a1f2-61b89c4d9daf","Type":"ContainerDied","Data":"c78ffd76732751a5999caef74ce712dc272f87e7d325957fb72a1cf4d91a2754"}
Sep 11 23:28:14.874936 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:28:14.874853    2546 scope.go:117] "RemoveContainer" containerID="a26835f1611c9022531a8d6c8d4efb495eba246fb7bfa5eb3c616b2f4a6872f2"
Sep 11 23:28:14.886835 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:28:14.886806    2546 scope.go:117] "RemoveContainer" containerID="a26835f1611c9022531a8d6c8d4efb495eba246fb7bfa5eb3c616b2f4a6872f2"
Sep 11 23:28:14.887270 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:28:14.887243    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a26835f1611c9022531a8d6c8d4efb495eba246fb7bfa5eb3c616b2f4a6872f2\": container with ID starting with a26835f1611c9022531a8d6c8d4efb495eba246fb7bfa5eb3c616b2f4a6872f2 not found: ID does not exist" containerID="a26835f1611c9022531a8d6c8d4efb495eba246fb7bfa5eb3c616b2f4a6872f2"
Sep 11 23:28:14.887365 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:28:14.887282    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a26835f1611c9022531a8d6c8d4efb495eba246fb7bfa5eb3c616b2f4a6872f2"} err="failed to get container status \"a26835f1611c9022531a8d6c8d4efb495eba246fb7bfa5eb3c616b2f4a6872f2\": rpc error: code = NotFound desc = could not find container \"a26835f1611c9022531a8d6c8d4efb495eba246fb7bfa5eb3c616b2f4a6872f2\": container with ID starting with a26835f1611c9022531a8d6c8d4efb495eba246fb7bfa5eb3c616b2f4a6872f2 not found: ID does not exist"
Sep 11 23:28:14.928021 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:28:14.927979    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14fee1b9-b0f8-4fa4-a1f2-61b89c4d9daf-home\") pod \"14fee1b9-b0f8-4fa4-a1f2-61b89c4d9daf\" (UID: \"14fee1b9-b0f8-4fa4-a1f2-61b89c4d9daf\") "
Sep 11 23:28:14.928245 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:28:14.928030    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14fee1b9-b0f8-4fa4-a1f2-61b89c4d9daf-tls-certs\") pod \"14fee1b9-b0f8-4fa4-a1f2-61b89c4d9daf\" (UID: \"14fee1b9-b0f8-4fa4-a1f2-61b89c4d9daf\") "
Sep 11 23:28:14.928336 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:28:14.928287    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14fee1b9-b0f8-4fa4-a1f2-61b89c4d9daf-tmp-dir\") pod \"14fee1b9-b0f8-4fa4-a1f2-61b89c4d9daf\" (UID: \"14fee1b9-b0f8-4fa4-a1f2-61b89c4d9daf\") "
Sep 11 23:28:14.928388 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:28:14.928325    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14fee1b9-b0f8-4fa4-a1f2-61b89c4d9daf-home" (OuterVolumeSpecName: "home") pod "14fee1b9-b0f8-4fa4-a1f2-61b89c4d9daf" (UID: "14fee1b9-b0f8-4fa4-a1f2-61b89c4d9daf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:28:14.928493 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:28:14.928476    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n85d8\" (UniqueName: \"kubernetes.io/projected/14fee1b9-b0f8-4fa4-a1f2-61b89c4d9daf-kube-api-access-n85d8\") pod \"14fee1b9-b0f8-4fa4-a1f2-61b89c4d9daf\" (UID: \"14fee1b9-b0f8-4fa4-a1f2-61b89c4d9daf\") "
Sep 11 23:28:14.928569 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:28:14.928516    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14fee1b9-b0f8-4fa4-a1f2-61b89c4d9daf-dshm\") pod \"14fee1b9-b0f8-4fa4-a1f2-61b89c4d9daf\" (UID: \"14fee1b9-b0f8-4fa4-a1f2-61b89c4d9daf\") "
Sep 11 23:28:14.928569 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:28:14.928535    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14fee1b9-b0f8-4fa4-a1f2-61b89c4d9daf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "14fee1b9-b0f8-4fa4-a1f2-61b89c4d9daf" (UID: "14fee1b9-b0f8-4fa4-a1f2-61b89c4d9daf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:28:14.928569 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:28:14.928543    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14fee1b9-b0f8-4fa4-a1f2-61b89c4d9daf-model-cache\") pod \"14fee1b9-b0f8-4fa4-a1f2-61b89c4d9daf\" (UID: \"14fee1b9-b0f8-4fa4-a1f2-61b89c4d9daf\") "
Sep 11 23:28:14.928825 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:28:14.928801    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14fee1b9-b0f8-4fa4-a1f2-61b89c4d9daf-model-cache" (OuterVolumeSpecName: "model-cache") pod "14fee1b9-b0f8-4fa4-a1f2-61b89c4d9daf" (UID: "14fee1b9-b0f8-4fa4-a1f2-61b89c4d9daf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:28:14.929243 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:28:14.929224    2546 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14fee1b9-b0f8-4fa4-a1f2-61b89c4d9daf-model-cache\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:28:14.929363 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:28:14.929350    2546 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14fee1b9-b0f8-4fa4-a1f2-61b89c4d9daf-home\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:28:14.929457 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:28:14.929437    2546 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14fee1b9-b0f8-4fa4-a1f2-61b89c4d9daf-tmp-dir\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:28:14.930767 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:28:14.930741    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14fee1b9-b0f8-4fa4-a1f2-61b89c4d9daf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "14fee1b9-b0f8-4fa4-a1f2-61b89c4d9daf" (UID: "14fee1b9-b0f8-4fa4-a1f2-61b89c4d9daf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 23:28:14.930870 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:28:14.930812    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14fee1b9-b0f8-4fa4-a1f2-61b89c4d9daf-dshm" (OuterVolumeSpecName: "dshm") pod "14fee1b9-b0f8-4fa4-a1f2-61b89c4d9daf" (UID: "14fee1b9-b0f8-4fa4-a1f2-61b89c4d9daf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:28:14.931098 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:28:14.931072    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14fee1b9-b0f8-4fa4-a1f2-61b89c4d9daf-kube-api-access-n85d8" (OuterVolumeSpecName: "kube-api-access-n85d8") pod "14fee1b9-b0f8-4fa4-a1f2-61b89c4d9daf" (UID: "14fee1b9-b0f8-4fa4-a1f2-61b89c4d9daf"). InnerVolumeSpecName "kube-api-access-n85d8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 23:28:15.030886 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:28:15.030847    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n85d8\" (UniqueName: \"kubernetes.io/projected/14fee1b9-b0f8-4fa4-a1f2-61b89c4d9daf-kube-api-access-n85d8\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:28:15.030886 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:28:15.030882    2546 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14fee1b9-b0f8-4fa4-a1f2-61b89c4d9daf-dshm\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:28:15.030886 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:28:15.030892    2546 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14fee1b9-b0f8-4fa4-a1f2-61b89c4d9daf-tls-certs\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:28:15.200200 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:28:15.200144    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-b96fb0e1/e2e-lifecycle-model-b96fb0e1-kserve-5688546d8c-4fdqw"]
Sep 11 23:28:15.203788 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:28:15.203759    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-b96fb0e1/e2e-lifecycle-model-b96fb0e1-kserve-5688546d8c-4fdqw"]
Sep 11 23:28:16.495050 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:28:16.495016    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14fee1b9-b0f8-4fa4-a1f2-61b89c4d9daf" path="/var/lib/kubelet/pods/14fee1b9-b0f8-4fa4-a1f2-61b89c4d9daf/volumes"
Sep 11 23:28:34.483411 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:28:34.483371    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"]
Sep 11 23:29:01.612530 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:29:01.612481    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"]
Sep 11 23:29:06.472857 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:29:06.472818    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b96fb0e1-openshift-default-df6bbcc4d-shm4v"]
Sep 11 23:29:06.473434 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:29:06.473051    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-b96fb0e1-openshift-default-df6bbcc4d-shm4v" podUID="3c330e7f-955c-4e6f-b4c8-5b0dee024ba5" containerName="istio-proxy" containerID="cri-o://7985807caa2afa5f34824ac4e77fcb377d5fc7da5561325d3c7c636ec7e6c65f" gracePeriod=30
Sep 11 23:29:11.745387 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:29:11.745322    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b96fb0e1-openshift-default-df6bbcc4d-shm4v"
Sep 11 23:29:11.845936 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:29:11.845901    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c330e7f-955c-4e6f-b4c8-5b0dee024ba5-istio-data\") pod \"3c330e7f-955c-4e6f-b4c8-5b0dee024ba5\" (UID: \"3c330e7f-955c-4e6f-b4c8-5b0dee024ba5\") "
Sep 11 23:29:11.846146 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:29:11.845978    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c330e7f-955c-4e6f-b4c8-5b0dee024ba5-workload-socket\") pod \"3c330e7f-955c-4e6f-b4c8-5b0dee024ba5\" (UID: \"3c330e7f-955c-4e6f-b4c8-5b0dee024ba5\") "
Sep 11 23:29:11.846146 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:29:11.846011    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c330e7f-955c-4e6f-b4c8-5b0dee024ba5-istio-token\") pod \"3c330e7f-955c-4e6f-b4c8-5b0dee024ba5\" (UID: \"3c330e7f-955c-4e6f-b4c8-5b0dee024ba5\") "
Sep 11 23:29:11.846146 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:29:11.846043    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c330e7f-955c-4e6f-b4c8-5b0dee024ba5-workload-certs\") pod \"3c330e7f-955c-4e6f-b4c8-5b0dee024ba5\" (UID: \"3c330e7f-955c-4e6f-b4c8-5b0dee024ba5\") "
Sep 11 23:29:11.846146 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:29:11.846071    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-867j4\" (UniqueName: \"kubernetes.io/projected/3c330e7f-955c-4e6f-b4c8-5b0dee024ba5-kube-api-access-867j4\") pod \"3c330e7f-955c-4e6f-b4c8-5b0dee024ba5\" (UID: \"3c330e7f-955c-4e6f-b4c8-5b0dee024ba5\") "
Sep 11 23:29:11.846146 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:29:11.846118    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c330e7f-955c-4e6f-b4c8-5b0dee024ba5-credential-socket\") pod \"3c330e7f-955c-4e6f-b4c8-5b0dee024ba5\" (UID: \"3c330e7f-955c-4e6f-b4c8-5b0dee024ba5\") "
Sep 11 23:29:11.846534 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:29:11.846197    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c330e7f-955c-4e6f-b4c8-5b0dee024ba5-istio-envoy\") pod \"3c330e7f-955c-4e6f-b4c8-5b0dee024ba5\" (UID: \"3c330e7f-955c-4e6f-b4c8-5b0dee024ba5\") "
Sep 11 23:29:11.846534 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:29:11.846270    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c330e7f-955c-4e6f-b4c8-5b0dee024ba5-istiod-ca-cert\") pod \"3c330e7f-955c-4e6f-b4c8-5b0dee024ba5\" (UID: \"3c330e7f-955c-4e6f-b4c8-5b0dee024ba5\") "
Sep 11 23:29:11.846534 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:29:11.846306    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c330e7f-955c-4e6f-b4c8-5b0dee024ba5-istio-data" (OuterVolumeSpecName: "istio-data") pod "3c330e7f-955c-4e6f-b4c8-5b0dee024ba5" (UID: "3c330e7f-955c-4e6f-b4c8-5b0dee024ba5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:29:11.846534 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:29:11.846374    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c330e7f-955c-4e6f-b4c8-5b0dee024ba5-istio-podinfo\") pod \"3c330e7f-955c-4e6f-b4c8-5b0dee024ba5\" (UID: \"3c330e7f-955c-4e6f-b4c8-5b0dee024ba5\") "
Sep 11 23:29:11.846534 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:29:11.846464    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c330e7f-955c-4e6f-b4c8-5b0dee024ba5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3c330e7f-955c-4e6f-b4c8-5b0dee024ba5" (UID: "3c330e7f-955c-4e6f-b4c8-5b0dee024ba5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:29:11.846534 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:29:11.846477    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c330e7f-955c-4e6f-b4c8-5b0dee024ba5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3c330e7f-955c-4e6f-b4c8-5b0dee024ba5" (UID: "3c330e7f-955c-4e6f-b4c8-5b0dee024ba5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:29:11.846865 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:29:11.846636    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c330e7f-955c-4e6f-b4c8-5b0dee024ba5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3c330e7f-955c-4e6f-b4c8-5b0dee024ba5" (UID: "3c330e7f-955c-4e6f-b4c8-5b0dee024ba5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:29:11.846865 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:29:11.846711    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c330e7f-955c-4e6f-b4c8-5b0dee024ba5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3c330e7f-955c-4e6f-b4c8-5b0dee024ba5" (UID: "3c330e7f-955c-4e6f-b4c8-5b0dee024ba5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 23:29:11.846865 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:29:11.846852    2546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c330e7f-955c-4e6f-b4c8-5b0dee024ba5-istio-data\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:29:11.846865 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:29:11.846873    2546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c330e7f-955c-4e6f-b4c8-5b0dee024ba5-workload-socket\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:29:11.846865 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:29:11.846886    2546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c330e7f-955c-4e6f-b4c8-5b0dee024ba5-workload-certs\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:29:11.846865 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:29:11.846900    2546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c330e7f-955c-4e6f-b4c8-5b0dee024ba5-credential-socket\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:29:11.846865 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:29:11.846914    2546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c330e7f-955c-4e6f-b4c8-5b0dee024ba5-istiod-ca-cert\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:29:11.848682 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:29:11.848652    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c330e7f-955c-4e6f-b4c8-5b0dee024ba5-istio-token" (OuterVolumeSpecName: "istio-token") pod "3c330e7f-955c-4e6f-b4c8-5b0dee024ba5" (UID: "3c330e7f-955c-4e6f-b4c8-5b0dee024ba5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 23:29:11.848682 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:29:11.848666    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c330e7f-955c-4e6f-b4c8-5b0dee024ba5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3c330e7f-955c-4e6f-b4c8-5b0dee024ba5" (UID: "3c330e7f-955c-4e6f-b4c8-5b0dee024ba5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 23:29:11.848832 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:29:11.848680    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c330e7f-955c-4e6f-b4c8-5b0dee024ba5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3c330e7f-955c-4e6f-b4c8-5b0dee024ba5" (UID: "3c330e7f-955c-4e6f-b4c8-5b0dee024ba5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:29:11.848832 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:29:11.848687    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c330e7f-955c-4e6f-b4c8-5b0dee024ba5-kube-api-access-867j4" (OuterVolumeSpecName: "kube-api-access-867j4") pod "3c330e7f-955c-4e6f-b4c8-5b0dee024ba5" (UID: "3c330e7f-955c-4e6f-b4c8-5b0dee024ba5"). InnerVolumeSpecName "kube-api-access-867j4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 23:29:11.948246 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:29:11.948208    2546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c330e7f-955c-4e6f-b4c8-5b0dee024ba5-istio-token\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:29:11.948246 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:29:11.948239    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-867j4\" (UniqueName: \"kubernetes.io/projected/3c330e7f-955c-4e6f-b4c8-5b0dee024ba5-kube-api-access-867j4\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:29:11.948246 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:29:11.948249    2546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c330e7f-955c-4e6f-b4c8-5b0dee024ba5-istio-envoy\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:29:11.948492 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:29:11.948260    2546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c330e7f-955c-4e6f-b4c8-5b0dee024ba5-istio-podinfo\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:29:12.290741 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:29:12.290696    2546 generic.go:356] "Generic (PLEG): container finished" podID="3c330e7f-955c-4e6f-b4c8-5b0dee024ba5" containerID="7985807caa2afa5f34824ac4e77fcb377d5fc7da5561325d3c7c636ec7e6c65f" exitCode=0
Sep 11 23:29:12.290966 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:29:12.290743    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b96fb0e1-openshift-default-df6bbcc4d-shm4v" event={"ID":"3c330e7f-955c-4e6f-b4c8-5b0dee024ba5","Type":"ContainerDied","Data":"7985807caa2afa5f34824ac4e77fcb377d5fc7da5561325d3c7c636ec7e6c65f"}
Sep 11 23:29:12.290966 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:29:12.290759    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b96fb0e1-openshift-default-df6bbcc4d-shm4v"
Sep 11 23:29:12.290966 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:29:12.290808    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b96fb0e1-openshift-default-df6bbcc4d-shm4v" event={"ID":"3c330e7f-955c-4e6f-b4c8-5b0dee024ba5","Type":"ContainerDied","Data":"04fe5627a8e7a7eec4cbd2942646239ac840a587e8041802aec7338f051d1848"}
Sep 11 23:29:12.290966 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:29:12.290832    2546 scope.go:117] "RemoveContainer" containerID="7985807caa2afa5f34824ac4e77fcb377d5fc7da5561325d3c7c636ec7e6c65f"
Sep 11 23:29:12.302295 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:29:12.302266    2546 scope.go:117] "RemoveContainer" containerID="a1bf2018af1bc1ccbdac0ba0ca08734bd30b10cc2303dfcff0520ed9be0cabef"
Sep 11 23:29:12.314572 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:29:12.314542    2546 scope.go:117] "RemoveContainer" containerID="7985807caa2afa5f34824ac4e77fcb377d5fc7da5561325d3c7c636ec7e6c65f"
Sep 11 23:29:12.314896 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:29:12.314876    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7985807caa2afa5f34824ac4e77fcb377d5fc7da5561325d3c7c636ec7e6c65f\": container with ID starting with 7985807caa2afa5f34824ac4e77fcb377d5fc7da5561325d3c7c636ec7e6c65f not found: ID does not exist" containerID="7985807caa2afa5f34824ac4e77fcb377d5fc7da5561325d3c7c636ec7e6c65f"
Sep 11 23:29:12.314972 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:29:12.314904    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7985807caa2afa5f34824ac4e77fcb377d5fc7da5561325d3c7c636ec7e6c65f"} err="failed to get container status \"7985807caa2afa5f34824ac4e77fcb377d5fc7da5561325d3c7c636ec7e6c65f\": rpc error: code = NotFound desc = could not find container \"7985807caa2afa5f34824ac4e77fcb377d5fc7da5561325d3c7c636ec7e6c65f\": container with ID starting with 7985807caa2afa5f34824ac4e77fcb377d5fc7da5561325d3c7c636ec7e6c65f not found: ID does not exist"
Sep 11 23:29:12.314972 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:29:12.314922    2546 scope.go:117] "RemoveContainer" containerID="a1bf2018af1bc1ccbdac0ba0ca08734bd30b10cc2303dfcff0520ed9be0cabef"
Sep 11 23:29:12.315253 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:29:12.315229    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1bf2018af1bc1ccbdac0ba0ca08734bd30b10cc2303dfcff0520ed9be0cabef\": container with ID starting with a1bf2018af1bc1ccbdac0ba0ca08734bd30b10cc2303dfcff0520ed9be0cabef not found: ID does not exist" containerID="a1bf2018af1bc1ccbdac0ba0ca08734bd30b10cc2303dfcff0520ed9be0cabef"
Sep 11 23:29:12.315336 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:29:12.315259    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1bf2018af1bc1ccbdac0ba0ca08734bd30b10cc2303dfcff0520ed9be0cabef"} err="failed to get container status \"a1bf2018af1bc1ccbdac0ba0ca08734bd30b10cc2303dfcff0520ed9be0cabef\": rpc error: code = NotFound desc = could not find container \"a1bf2018af1bc1ccbdac0ba0ca08734bd30b10cc2303dfcff0520ed9be0cabef\": container with ID starting with a1bf2018af1bc1ccbdac0ba0ca08734bd30b10cc2303dfcff0520ed9be0cabef not found: ID does not exist"
Sep 11 23:29:12.325487 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:29:12.325450    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b96fb0e1-openshift-default-df6bbcc4d-shm4v"]
Sep 11 23:29:12.338881 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:29:12.338848    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-b96fb0e1-openshift-default-df6bbcc4d-shm4v"]
Sep 11 23:29:12.495522 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:29:12.495483    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c330e7f-955c-4e6f-b4c8-5b0dee024ba5" path="/var/lib/kubelet/pods/3c330e7f-955c-4e6f-b4c8-5b0dee024ba5/volumes"
Sep 11 23:29:16.337237 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:29:16.337199    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-12fb6b/embedding-w0-12fb6b-kserve-585bc8ccb5-q5647"]
Sep 11 23:29:16.337747 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:29:16.337680    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c330e7f-955c-4e6f-b4c8-5b0dee024ba5" containerName="istio-proxy"
Sep 11 23:29:16.337747 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:29:16.337693    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c330e7f-955c-4e6f-b4c8-5b0dee024ba5" containerName="istio-proxy"
Sep 11 23:29:16.337747 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:29:16.337700    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14fee1b9-b0f8-4fa4-a1f2-61b89c4d9daf" containerName="main"
Sep 11 23:29:16.337747 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:29:16.337706    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="14fee1b9-b0f8-4fa4-a1f2-61b89c4d9daf" containerName="main"
Sep 11 23:29:16.337747 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:29:16.337716    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c330e7f-955c-4e6f-b4c8-5b0dee024ba5" containerName="istio-proxy"
Sep 11 23:29:16.337747 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:29:16.337723    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c330e7f-955c-4e6f-b4c8-5b0dee024ba5" containerName="istio-proxy"
Sep 11 23:29:16.337747 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:29:16.337738    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ddd3b74-bca5-4776-8b7e-f12f2f2dd8c7" containerName="revoke-keys"
Sep 11 23:29:16.337747 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:29:16.337743    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ddd3b74-bca5-4776-8b7e-f12f2f2dd8c7" containerName="revoke-keys"
Sep 11 23:29:16.338231 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:29:16.337804    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="14fee1b9-b0f8-4fa4-a1f2-61b89c4d9daf" containerName="main"
Sep 11 23:29:16.338231 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:29:16.337814    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c330e7f-955c-4e6f-b4c8-5b0dee024ba5" containerName="istio-proxy"
Sep 11 23:29:16.338231 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:29:16.337823    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ddd3b74-bca5-4776-8b7e-f12f2f2dd8c7" containerName="revoke-keys"
Sep 11 23:29:16.342682 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:29:16.342654    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-12fb6b/embedding-w0-12fb6b-kserve-585bc8ccb5-q5647"
Sep 11 23:29:16.345982 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:29:16.345956    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-12fb6b\"/\"embedding-w0-12fb6b-kserve-self-signed-certs\""
Sep 11 23:29:16.355100 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:29:16.355073    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-12fb6b/embedding-w0-12fb6b-kserve-585bc8ccb5-q5647"]
Sep 11 23:29:16.389568 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:29:16.389531    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc328101-34e6-4746-9e4f-3e58116a6e70-home\") pod \"embedding-w0-12fb6b-kserve-585bc8ccb5-q5647\" (UID: \"dc328101-34e6-4746-9e4f-3e58116a6e70\") " pod="e2e-models-e2e-worker-w0-12fb6b/embedding-w0-12fb6b-kserve-585bc8ccb5-q5647"
Sep 11 23:29:16.389765 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:29:16.389577    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc328101-34e6-4746-9e4f-3e58116a6e70-tls-certs\") pod \"embedding-w0-12fb6b-kserve-585bc8ccb5-q5647\" (UID: \"dc328101-34e6-4746-9e4f-3e58116a6e70\") " pod="e2e-models-e2e-worker-w0-12fb6b/embedding-w0-12fb6b-kserve-585bc8ccb5-q5647"
Sep 11 23:29:16.389765 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:29:16.389605    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc328101-34e6-4746-9e4f-3e58116a6e70-model-cache\") pod \"embedding-w0-12fb6b-kserve-585bc8ccb5-q5647\" (UID: \"dc328101-34e6-4746-9e4f-3e58116a6e70\") " pod="e2e-models-e2e-worker-w0-12fb6b/embedding-w0-12fb6b-kserve-585bc8ccb5-q5647"
Sep 11 23:29:16.389765 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:29:16.389644    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7498\" (UniqueName: \"kubernetes.io/projected/dc328101-34e6-4746-9e4f-3e58116a6e70-kube-api-access-c7498\") pod \"embedding-w0-12fb6b-kserve-585bc8ccb5-q5647\" (UID: \"dc328101-34e6-4746-9e4f-3e58116a6e70\") " pod="e2e-models-e2e-worker-w0-12fb6b/embedding-w0-12fb6b-kserve-585bc8ccb5-q5647"
Sep 11 23:29:16.389908 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:29:16.389771    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc328101-34e6-4746-9e4f-3e58116a6e70-tmp-dir\") pod \"embedding-w0-12fb6b-kserve-585bc8ccb5-q5647\" (UID: \"dc328101-34e6-4746-9e4f-3e58116a6e70\") " pod="e2e-models-e2e-worker-w0-12fb6b/embedding-w0-12fb6b-kserve-585bc8ccb5-q5647"
Sep 11 23:29:16.389908 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:29:16.389820    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc328101-34e6-4746-9e4f-3e58116a6e70-dshm\") pod \"embedding-w0-12fb6b-kserve-585bc8ccb5-q5647\" (UID: \"dc328101-34e6-4746-9e4f-3e58116a6e70\") " pod="e2e-models-e2e-worker-w0-12fb6b/embedding-w0-12fb6b-kserve-585bc8ccb5-q5647"
Sep 11 23:29:16.490456 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:29:16.490406    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc328101-34e6-4746-9e4f-3e58116a6e70-tmp-dir\") pod \"embedding-w0-12fb6b-kserve-585bc8ccb5-q5647\" (UID: \"dc328101-34e6-4746-9e4f-3e58116a6e70\") " pod="e2e-models-e2e-worker-w0-12fb6b/embedding-w0-12fb6b-kserve-585bc8ccb5-q5647"
Sep 11 23:29:16.490686 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:29:16.490484    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc328101-34e6-4746-9e4f-3e58116a6e70-dshm\") pod \"embedding-w0-12fb6b-kserve-585bc8ccb5-q5647\" (UID: \"dc328101-34e6-4746-9e4f-3e58116a6e70\") " pod="e2e-models-e2e-worker-w0-12fb6b/embedding-w0-12fb6b-kserve-585bc8ccb5-q5647"
Sep 11 23:29:16.490686 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:29:16.490550    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc328101-34e6-4746-9e4f-3e58116a6e70-home\") pod \"embedding-w0-12fb6b-kserve-585bc8ccb5-q5647\" (UID: \"dc328101-34e6-4746-9e4f-3e58116a6e70\") " pod="e2e-models-e2e-worker-w0-12fb6b/embedding-w0-12fb6b-kserve-585bc8ccb5-q5647"
Sep 11 23:29:16.490686 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:29:16.490594    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc328101-34e6-4746-9e4f-3e58116a6e70-tls-certs\") pod \"embedding-w0-12fb6b-kserve-585bc8ccb5-q5647\" (UID: \"dc328101-34e6-4746-9e4f-3e58116a6e70\") " pod="e2e-models-e2e-worker-w0-12fb6b/embedding-w0-12fb6b-kserve-585bc8ccb5-q5647"
Sep 11 23:29:16.490686 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:29:16.490627    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc328101-34e6-4746-9e4f-3e58116a6e70-model-cache\") pod \"embedding-w0-12fb6b-kserve-585bc8ccb5-q5647\" (UID: \"dc328101-34e6-4746-9e4f-3e58116a6e70\") " pod="e2e-models-e2e-worker-w0-12fb6b/embedding-w0-12fb6b-kserve-585bc8ccb5-q5647"
Sep 11 23:29:16.490686 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:29:16.490667    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7498\" (UniqueName: \"kubernetes.io/projected/dc328101-34e6-4746-9e4f-3e58116a6e70-kube-api-access-c7498\") pod \"embedding-w0-12fb6b-kserve-585bc8ccb5-q5647\" (UID: \"dc328101-34e6-4746-9e4f-3e58116a6e70\") " pod="e2e-models-e2e-worker-w0-12fb6b/embedding-w0-12fb6b-kserve-585bc8ccb5-q5647"
Sep 11 23:29:16.490991 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:29:16.490855    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc328101-34e6-4746-9e4f-3e58116a6e70-tmp-dir\") pod \"embedding-w0-12fb6b-kserve-585bc8ccb5-q5647\" (UID: \"dc328101-34e6-4746-9e4f-3e58116a6e70\") " pod="e2e-models-e2e-worker-w0-12fb6b/embedding-w0-12fb6b-kserve-585bc8ccb5-q5647"
Sep 11 23:29:16.491191 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:29:16.491111    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc328101-34e6-4746-9e4f-3e58116a6e70-model-cache\") pod \"embedding-w0-12fb6b-kserve-585bc8ccb5-q5647\" (UID: \"dc328101-34e6-4746-9e4f-3e58116a6e70\") " pod="e2e-models-e2e-worker-w0-12fb6b/embedding-w0-12fb6b-kserve-585bc8ccb5-q5647"
Sep 11 23:29:16.491394 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:29:16.491337    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc328101-34e6-4746-9e4f-3e58116a6e70-home\") pod \"embedding-w0-12fb6b-kserve-585bc8ccb5-q5647\" (UID: \"dc328101-34e6-4746-9e4f-3e58116a6e70\") " pod="e2e-models-e2e-worker-w0-12fb6b/embedding-w0-12fb6b-kserve-585bc8ccb5-q5647"
Sep 11 23:29:16.493362 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:29:16.493340    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc328101-34e6-4746-9e4f-3e58116a6e70-dshm\") pod \"embedding-w0-12fb6b-kserve-585bc8ccb5-q5647\" (UID: \"dc328101-34e6-4746-9e4f-3e58116a6e70\") " pod="e2e-models-e2e-worker-w0-12fb6b/embedding-w0-12fb6b-kserve-585bc8ccb5-q5647"
Sep 11 23:29:16.495073 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:29:16.495049    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-12fb6b\"/\"embedding-w0-12fb6b-kserve-self-signed-certs\""
Sep 11 23:29:16.514388 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:29:16.511136    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc328101-34e6-4746-9e4f-3e58116a6e70-tls-certs\") pod \"embedding-w0-12fb6b-kserve-585bc8ccb5-q5647\" (UID: \"dc328101-34e6-4746-9e4f-3e58116a6e70\") " pod="e2e-models-e2e-worker-w0-12fb6b/embedding-w0-12fb6b-kserve-585bc8ccb5-q5647"
Sep 11 23:29:16.514388 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:29:16.514264    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7498\" (UniqueName: \"kubernetes.io/projected/dc328101-34e6-4746-9e4f-3e58116a6e70-kube-api-access-c7498\") pod \"embedding-w0-12fb6b-kserve-585bc8ccb5-q5647\" (UID: \"dc328101-34e6-4746-9e4f-3e58116a6e70\") " pod="e2e-models-e2e-worker-w0-12fb6b/embedding-w0-12fb6b-kserve-585bc8ccb5-q5647"
Sep 11 23:29:16.654029 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:29:16.653941    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-12fb6b/embedding-w0-12fb6b-kserve-585bc8ccb5-q5647"
Sep 11 23:29:16.777721 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:29:16.777687    2546 scope.go:117] "RemoveContainer" containerID="0493115a5ed3eb17b5a773cf10f5a3769f598047a3310b783a67a4dc867aee03"
Sep 11 23:29:16.939678 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:29:16.939645    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-12fb6b/embedding-w0-12fb6b-kserve-585bc8ccb5-q5647"]
Sep 11 23:29:16.940336 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:29:16.940309    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddc328101_34e6_4746_9e4f_3e58116a6e70.slice/crio-6d78bbc21f6189ee74c5f6f400ed07dafee98ac2047192dc10f2bdfdf9676243 WatchSource:0}: Error finding container 6d78bbc21f6189ee74c5f6f400ed07dafee98ac2047192dc10f2bdfdf9676243: Status 404 returned error can't find the container with id 6d78bbc21f6189ee74c5f6f400ed07dafee98ac2047192dc10f2bdfdf9676243
Sep 11 23:29:17.328209 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:29:17.328087    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-12fb6b/embedding-w0-12fb6b-kserve-585bc8ccb5-q5647" event={"ID":"dc328101-34e6-4746-9e4f-3e58116a6e70","Type":"ContainerStarted","Data":"f8b435be85ec4a7dc300a3569204be232a116a36d8bb2fd5d5542684d39259fe"}
Sep 11 23:29:17.328209 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:29:17.328131    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-12fb6b/embedding-w0-12fb6b-kserve-585bc8ccb5-q5647" event={"ID":"dc328101-34e6-4746-9e4f-3e58116a6e70","Type":"ContainerStarted","Data":"6d78bbc21f6189ee74c5f6f400ed07dafee98ac2047192dc10f2bdfdf9676243"}
Sep 11 23:29:17.388096 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:29:17.388032    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-12fb6b/embedding-w0-12fb6b-kserve-585bc8ccb5-q5647" podStartSLOduration=1.388011497 podStartE2EDuration="1.388011497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 23:29:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:29:17.385301779 +0000 UTC m=+1681.418430996" watchObservedRunningTime="2026-09-11 23:29:17.388011497 +0000 UTC m=+1681.421140716"
Sep 11 23:29:26.654775 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:29:26.654734    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-12fb6b/embedding-w0-12fb6b-kserve-585bc8ccb5-q5647"
Sep 11 23:29:26.655223 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:29:26.654789    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-12fb6b/embedding-w0-12fb6b-kserve-585bc8ccb5-q5647"
Sep 11 23:29:26.667573 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:29:26.667542    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-12fb6b/embedding-w0-12fb6b-kserve-585bc8ccb5-q5647"
Sep 11 23:29:27.412187 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:29:27.412135    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-12fb6b/embedding-w0-12fb6b-kserve-585bc8ccb5-q5647"
Sep 11 23:29:41.300992 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:29:41.300909    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"]
Sep 11 23:30:00.149602 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:00.149560    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819490-kwbtx"]
Sep 11 23:30:00.150197 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:00.150180    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c330e7f-955c-4e6f-b4c8-5b0dee024ba5" containerName="istio-proxy"
Sep 11 23:30:00.153351 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:00.153330    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-97d9dc-29819490-7b6mr"]
Sep 11 23:30:00.153488 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:00.153466    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819490-kwbtx"
Sep 11 23:30:00.156458 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:00.156432    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vrx7m\""
Sep 11 23:30:00.157067 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:00.157041    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-12fb6b-29819490-z7ztp"]
Sep 11 23:30:00.157554 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:00.157530    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-97d9dc-29819490-7b6mr"
Sep 11 23:30:00.161266 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:00.161243    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819490-kwbtx"]
Sep 11 23:30:00.161386 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:00.161370    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-12fb6b-29819490-z7ztp"
Sep 11 23:30:00.169692 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:00.169667    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819490-kwbtx"
Sep 11 23:30:00.173281 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:00.173248    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-97d9dc-29819490-7b6mr"]
Sep 11 23:30:00.174677 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:00.174653    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-12fb6b-29819490-z7ztp"]
Sep 11 23:30:00.180119 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:00.180093    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cd48eb-29819490-m2hcj"]
Sep 11 23:30:00.185022 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:00.184986    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cd48eb-29819490-m2hcj"
Sep 11 23:30:00.185362 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:00.185341    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-97d9dc-29819490-7b6mr"
Sep 11 23:30:00.193465 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:00.193022    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cd48eb-29819490-m2hcj"]
Sep 11 23:30:00.235394 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:00.235331    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-12fb6b-29819490-z7ztp"
Sep 11 23:30:00.246347 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:00.246321    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cd48eb-29819490-m2hcj"
Sep 11 23:30:00.478531 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:00.478507    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819490-kwbtx"]
Sep 11 23:30:00.479971 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:30:00.479942    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd55d452e_62be_494c_a30f_ee01a215943a.slice/crio-b7516fdfcc6c26793d9e9cad07b498e4b3d558da7b03edfd0733209dfd034cd7 WatchSource:0}: Error finding container b7516fdfcc6c26793d9e9cad07b498e4b3d558da7b03edfd0733209dfd034cd7: Status 404 returned error can't find the container with id b7516fdfcc6c26793d9e9cad07b498e4b3d558da7b03edfd0733209dfd034cd7
Sep 11 23:30:00.490273 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:30:00.490243    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1076ba3_1d97_4eda_9e02_52ef3eb30ff1.slice/crio-10be7a9f9855fb7c19d0c9e2ad1d413baf340d03b2b9a4f3317611c670cc48e3 WatchSource:0}: Error finding container 10be7a9f9855fb7c19d0c9e2ad1d413baf340d03b2b9a4f3317611c670cc48e3: Status 404 returned error can't find the container with id 10be7a9f9855fb7c19d0c9e2ad1d413baf340d03b2b9a4f3317611c670cc48e3
Sep 11 23:30:00.495677 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:00.495623    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-97d9dc-29819490-7b6mr"]
Sep 11 23:30:00.558769 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:00.558741    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-12fb6b-29819490-z7ztp"]
Sep 11 23:30:00.560576 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:30:00.560549    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29b56839_d428_44de_9434_96fc011d873b.slice/crio-7c5b58d9964e7f02fb4c01b868d3d60e3481ae07f3a6abd84f990fac45188cb2 WatchSource:0}: Error finding container 7c5b58d9964e7f02fb4c01b868d3d60e3481ae07f3a6abd84f990fac45188cb2: Status 404 returned error can't find the container with id 7c5b58d9964e7f02fb4c01b868d3d60e3481ae07f3a6abd84f990fac45188cb2
Sep 11 23:30:00.569753 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:00.569728    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cd48eb-29819490-m2hcj"]
Sep 11 23:30:00.570907 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:30:00.570880    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38d1bf05_fa75_4d8b_9f46_f843f80c0f86.slice/crio-1a22a7a06880f3018ae3fa455cc0a5a22284a90eb0b9b12d2a21e13e5852a390 WatchSource:0}: Error finding container 1a22a7a06880f3018ae3fa455cc0a5a22284a90eb0b9b12d2a21e13e5852a390: Status 404 returned error can't find the container with id 1a22a7a06880f3018ae3fa455cc0a5a22284a90eb0b9b12d2a21e13e5852a390
Sep 11 23:30:00.645604 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:00.645570    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cd48eb-29819490-m2hcj" event={"ID":"38d1bf05-fa75-4d8b-9f46-f843f80c0f86","Type":"ContainerStarted","Data":"1a22a7a06880f3018ae3fa455cc0a5a22284a90eb0b9b12d2a21e13e5852a390"}
Sep 11 23:30:00.646882 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:00.646856    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-12fb6b-29819490-z7ztp" event={"ID":"29b56839-d428-44de-9434-96fc011d873b","Type":"ContainerStarted","Data":"7c5b58d9964e7f02fb4c01b868d3d60e3481ae07f3a6abd84f990fac45188cb2"}
Sep 11 23:30:00.648074 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:00.648046    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-97d9dc-29819490-7b6mr" event={"ID":"b1076ba3-1d97-4eda-9e02-52ef3eb30ff1","Type":"ContainerStarted","Data":"10be7a9f9855fb7c19d0c9e2ad1d413baf340d03b2b9a4f3317611c670cc48e3"}
Sep 11 23:30:00.649250 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:00.649230    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819490-kwbtx" event={"ID":"d55d452e-62be-494c-a30f-ee01a215943a","Type":"ContainerStarted","Data":"b7516fdfcc6c26793d9e9cad07b498e4b3d558da7b03edfd0733209dfd034cd7"}
Sep 11 23:30:01.658014 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:01.657973    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-12fb6b-29819490-z7ztp" event={"ID":"29b56839-d428-44de-9434-96fc011d873b","Type":"ContainerStarted","Data":"5eb31700a5b6a4306eefb1b5d145ec9fd4d3823315e3ebe80f44a3ba66be7060"}
Sep 11 23:30:01.659713 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:01.659688    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-97d9dc-29819490-7b6mr" event={"ID":"b1076ba3-1d97-4eda-9e02-52ef3eb30ff1","Type":"ContainerStarted","Data":"7aa01b141c38d4cb2f50a806742c15a8b50fe63de303b8cea0bf09893b578803"}
Sep 11 23:30:01.661235 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:01.661202    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819490-kwbtx" event={"ID":"d55d452e-62be-494c-a30f-ee01a215943a","Type":"ContainerStarted","Data":"01a1669f545fd335cf3586bf29f6bd5577c8635988db2af54be80a9c1ea82e1f"}
Sep 11 23:30:01.662818 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:01.662792    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cd48eb-29819490-m2hcj" event={"ID":"38d1bf05-fa75-4d8b-9f46-f843f80c0f86","Type":"ContainerStarted","Data":"3067ffd280000032781c905c3b4337da13c075689a673e68ad39d0da26fac3bf"}
Sep 11 23:30:01.676858 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:01.676809    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-12fb6b-29819490-z7ztp" podStartSLOduration=1.67679483 podStartE2EDuration="1.67679483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:30:01.674659398 +0000 UTC m=+1725.707788617" watchObservedRunningTime="2026-09-11 23:30:01.67679483 +0000 UTC m=+1725.709924046"
Sep 11 23:30:01.694965 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:01.694906    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819490-kwbtx" podStartSLOduration=1.694888041 podStartE2EDuration="1.694888041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:30:01.691510356 +0000 UTC m=+1725.724639574" watchObservedRunningTime="2026-09-11 23:30:01.694888041 +0000 UTC m=+1725.728017259"
Sep 11 23:30:01.710128 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:01.710065    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cd48eb-29819490-m2hcj" podStartSLOduration=1.710048668 podStartE2EDuration="1.710048668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:30:01.708072745 +0000 UTC m=+1725.741201977" watchObservedRunningTime="2026-09-11 23:30:01.710048668 +0000 UTC m=+1725.743177905"
Sep 11 23:30:01.733904 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:01.729998    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-97d9dc-29819490-7b6mr" podStartSLOduration=1.729972796 podStartE2EDuration="1.729972796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:30:01.726865568 +0000 UTC m=+1725.759994788" watchObservedRunningTime="2026-09-11 23:30:01.729972796 +0000 UTC m=+1725.763102013"
Sep 11 23:30:02.677257 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:02.677214    2546 generic.go:356] "Generic (PLEG): container finished" podID="d55d452e-62be-494c-a30f-ee01a215943a" containerID="01a1669f545fd335cf3586bf29f6bd5577c8635988db2af54be80a9c1ea82e1f" exitCode=0
Sep 11 23:30:02.677740 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:02.677297    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819490-kwbtx" event={"ID":"d55d452e-62be-494c-a30f-ee01a215943a","Type":"ContainerDied","Data":"01a1669f545fd335cf3586bf29f6bd5577c8635988db2af54be80a9c1ea82e1f"}
Sep 11 23:30:02.678928 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:02.678898    2546 generic.go:356] "Generic (PLEG): container finished" podID="38d1bf05-fa75-4d8b-9f46-f843f80c0f86" containerID="3067ffd280000032781c905c3b4337da13c075689a673e68ad39d0da26fac3bf" exitCode=0
Sep 11 23:30:02.679076 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:02.678987    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cd48eb-29819490-m2hcj" event={"ID":"38d1bf05-fa75-4d8b-9f46-f843f80c0f86","Type":"ContainerDied","Data":"3067ffd280000032781c905c3b4337da13c075689a673e68ad39d0da26fac3bf"}
Sep 11 23:30:02.680582 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:02.680554    2546 generic.go:356] "Generic (PLEG): container finished" podID="29b56839-d428-44de-9434-96fc011d873b" containerID="5eb31700a5b6a4306eefb1b5d145ec9fd4d3823315e3ebe80f44a3ba66be7060" exitCode=0
Sep 11 23:30:02.680707 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:02.680629    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-12fb6b-29819490-z7ztp" event={"ID":"29b56839-d428-44de-9434-96fc011d873b","Type":"ContainerDied","Data":"5eb31700a5b6a4306eefb1b5d145ec9fd4d3823315e3ebe80f44a3ba66be7060"}
Sep 11 23:30:02.682118 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:02.682097    2546 generic.go:356] "Generic (PLEG): container finished" podID="b1076ba3-1d97-4eda-9e02-52ef3eb30ff1" containerID="7aa01b141c38d4cb2f50a806742c15a8b50fe63de303b8cea0bf09893b578803" exitCode=0
Sep 11 23:30:02.682244 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:02.682192    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-97d9dc-29819490-7b6mr" event={"ID":"b1076ba3-1d97-4eda-9e02-52ef3eb30ff1","Type":"ContainerDied","Data":"7aa01b141c38d4cb2f50a806742c15a8b50fe63de303b8cea0bf09893b578803"}
Sep 11 23:30:02.839715 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:02.839653    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw"]
Sep 11 23:30:02.845066 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:02.845000    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw"
Sep 11 23:30:02.849606 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:02.849576    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-451902-openshift-default-dockercfg-vj4jm\""
Sep 11 23:30:02.859465 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:02.859437    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw"]
Sep 11 23:30:03.013266 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:03.013138    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e0419a9-c88a-4f69-bd07-751d5406fd0e-istiod-ca-cert\") pod \"e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw\" (UID: \"6e0419a9-c88a-4f69-bd07-751d5406fd0e\") " pod="openshift-ingress/e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw"
Sep 11 23:30:03.013266 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:03.013228    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dv967\" (UniqueName: \"kubernetes.io/projected/6e0419a9-c88a-4f69-bd07-751d5406fd0e-kube-api-access-dv967\") pod \"e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw\" (UID: \"6e0419a9-c88a-4f69-bd07-751d5406fd0e\") " pod="openshift-ingress/e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw"
Sep 11 23:30:03.013497 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:03.013288    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e0419a9-c88a-4f69-bd07-751d5406fd0e-istio-envoy\") pod \"e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw\" (UID: \"6e0419a9-c88a-4f69-bd07-751d5406fd0e\") " pod="openshift-ingress/e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw"
Sep 11 23:30:03.013497 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:03.013327    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e0419a9-c88a-4f69-bd07-751d5406fd0e-istio-podinfo\") pod \"e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw\" (UID: \"6e0419a9-c88a-4f69-bd07-751d5406fd0e\") " pod="openshift-ingress/e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw"
Sep 11 23:30:03.013497 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:03.013375    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e0419a9-c88a-4f69-bd07-751d5406fd0e-istio-token\") pod \"e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw\" (UID: \"6e0419a9-c88a-4f69-bd07-751d5406fd0e\") " pod="openshift-ingress/e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw"
Sep 11 23:30:03.013497 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:03.013403    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e0419a9-c88a-4f69-bd07-751d5406fd0e-workload-certs\") pod \"e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw\" (UID: \"6e0419a9-c88a-4f69-bd07-751d5406fd0e\") " pod="openshift-ingress/e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw"
Sep 11 23:30:03.013497 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:03.013463    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e0419a9-c88a-4f69-bd07-751d5406fd0e-workload-socket\") pod \"e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw\" (UID: \"6e0419a9-c88a-4f69-bd07-751d5406fd0e\") " pod="openshift-ingress/e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw"
Sep 11 23:30:03.013731 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:03.013563    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e0419a9-c88a-4f69-bd07-751d5406fd0e-istio-data\") pod \"e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw\" (UID: \"6e0419a9-c88a-4f69-bd07-751d5406fd0e\") " pod="openshift-ingress/e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw"
Sep 11 23:30:03.013731 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:03.013645    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e0419a9-c88a-4f69-bd07-751d5406fd0e-credential-socket\") pod \"e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw\" (UID: \"6e0419a9-c88a-4f69-bd07-751d5406fd0e\") " pod="openshift-ingress/e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw"
Sep 11 23:30:03.114805 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:03.114758    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e0419a9-c88a-4f69-bd07-751d5406fd0e-istio-data\") pod \"e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw\" (UID: \"6e0419a9-c88a-4f69-bd07-751d5406fd0e\") " pod="openshift-ingress/e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw"
Sep 11 23:30:03.115009 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:03.114826    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e0419a9-c88a-4f69-bd07-751d5406fd0e-credential-socket\") pod \"e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw\" (UID: \"6e0419a9-c88a-4f69-bd07-751d5406fd0e\") " pod="openshift-ingress/e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw"
Sep 11 23:30:03.115009 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:03.114865    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e0419a9-c88a-4f69-bd07-751d5406fd0e-istiod-ca-cert\") pod \"e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw\" (UID: \"6e0419a9-c88a-4f69-bd07-751d5406fd0e\") " pod="openshift-ingress/e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw"
Sep 11 23:30:03.115009 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:03.114894    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dv967\" (UniqueName: \"kubernetes.io/projected/6e0419a9-c88a-4f69-bd07-751d5406fd0e-kube-api-access-dv967\") pod \"e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw\" (UID: \"6e0419a9-c88a-4f69-bd07-751d5406fd0e\") " pod="openshift-ingress/e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw"
Sep 11 23:30:03.115009 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:03.114930    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e0419a9-c88a-4f69-bd07-751d5406fd0e-istio-envoy\") pod \"e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw\" (UID: \"6e0419a9-c88a-4f69-bd07-751d5406fd0e\") " pod="openshift-ingress/e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw"
Sep 11 23:30:03.115009 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:03.114975    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e0419a9-c88a-4f69-bd07-751d5406fd0e-istio-podinfo\") pod \"e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw\" (UID: \"6e0419a9-c88a-4f69-bd07-751d5406fd0e\") " pod="openshift-ingress/e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw"
Sep 11 23:30:03.115328 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:03.115021    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e0419a9-c88a-4f69-bd07-751d5406fd0e-istio-token\") pod \"e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw\" (UID: \"6e0419a9-c88a-4f69-bd07-751d5406fd0e\") " pod="openshift-ingress/e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw"
Sep 11 23:30:03.115328 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:03.115048    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e0419a9-c88a-4f69-bd07-751d5406fd0e-workload-certs\") pod \"e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw\" (UID: \"6e0419a9-c88a-4f69-bd07-751d5406fd0e\") " pod="openshift-ingress/e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw"
Sep 11 23:30:03.115328 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:03.115128    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e0419a9-c88a-4f69-bd07-751d5406fd0e-workload-socket\") pod \"e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw\" (UID: \"6e0419a9-c88a-4f69-bd07-751d5406fd0e\") " pod="openshift-ingress/e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw"
Sep 11 23:30:03.115328 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:03.115264    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e0419a9-c88a-4f69-bd07-751d5406fd0e-istio-data\") pod \"e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw\" (UID: \"6e0419a9-c88a-4f69-bd07-751d5406fd0e\") " pod="openshift-ingress/e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw"
Sep 11 23:30:03.115601 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:03.115560    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e0419a9-c88a-4f69-bd07-751d5406fd0e-workload-socket\") pod \"e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw\" (UID: \"6e0419a9-c88a-4f69-bd07-751d5406fd0e\") " pod="openshift-ingress/e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw"
Sep 11 23:30:03.115715 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:03.115683    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e0419a9-c88a-4f69-bd07-751d5406fd0e-workload-certs\") pod \"e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw\" (UID: \"6e0419a9-c88a-4f69-bd07-751d5406fd0e\") " pod="openshift-ingress/e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw"
Sep 11 23:30:03.115715 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:03.115714    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e0419a9-c88a-4f69-bd07-751d5406fd0e-credential-socket\") pod \"e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw\" (UID: \"6e0419a9-c88a-4f69-bd07-751d5406fd0e\") " pod="openshift-ingress/e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw"
Sep 11 23:30:03.115715 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:03.115831    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e0419a9-c88a-4f69-bd07-751d5406fd0e-istiod-ca-cert\") pod \"e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw\" (UID: \"6e0419a9-c88a-4f69-bd07-751d5406fd0e\") " pod="openshift-ingress/e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw"
Sep 11 23:30:03.117591 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:03.117564    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e0419a9-c88a-4f69-bd07-751d5406fd0e-istio-envoy\") pod \"e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw\" (UID: \"6e0419a9-c88a-4f69-bd07-751d5406fd0e\") " pod="openshift-ingress/e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw"
Sep 11 23:30:03.117702 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:03.117670    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e0419a9-c88a-4f69-bd07-751d5406fd0e-istio-podinfo\") pod \"e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw\" (UID: \"6e0419a9-c88a-4f69-bd07-751d5406fd0e\") " pod="openshift-ingress/e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw"
Sep 11 23:30:03.134308 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:03.134270    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dv967\" (UniqueName: \"kubernetes.io/projected/6e0419a9-c88a-4f69-bd07-751d5406fd0e-kube-api-access-dv967\") pod \"e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw\" (UID: \"6e0419a9-c88a-4f69-bd07-751d5406fd0e\") " pod="openshift-ingress/e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw"
Sep 11 23:30:03.137097 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:03.137071    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e0419a9-c88a-4f69-bd07-751d5406fd0e-istio-token\") pod \"e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw\" (UID: \"6e0419a9-c88a-4f69-bd07-751d5406fd0e\") " pod="openshift-ingress/e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw"
Sep 11 23:30:03.157995 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:03.157967    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw"
Sep 11 23:30:03.450037 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:03.450007    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw"]
Sep 11 23:30:03.451902 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:30:03.451875    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e0419a9_c88a_4f69_bd07_751d5406fd0e.slice/crio-7d7ff9c2b4d67de04b2bd1cdf8dedf440e61f2bc688edaed112f896cb7d2f61c WatchSource:0}: Error finding container 7d7ff9c2b4d67de04b2bd1cdf8dedf440e61f2bc688edaed112f896cb7d2f61c: Status 404 returned error can't find the container with id 7d7ff9c2b4d67de04b2bd1cdf8dedf440e61f2bc688edaed112f896cb7d2f61c
Sep 11 23:30:03.454235 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:03.454202    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:30:03.454307 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:03.454273    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:30:03.454307 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:03.454302    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:30:03.692608 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:03.692559    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw" event={"ID":"6e0419a9-c88a-4f69-bd07-751d5406fd0e","Type":"ContainerStarted","Data":"cfcb19e22a3bcd059b4ec46499e729ef235b12aaa2b5b0ee8e3b3722d0290412"}
Sep 11 23:30:03.693082 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:03.692629    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw" event={"ID":"6e0419a9-c88a-4f69-bd07-751d5406fd0e","Type":"ContainerStarted","Data":"7d7ff9c2b4d67de04b2bd1cdf8dedf440e61f2bc688edaed112f896cb7d2f61c"}
Sep 11 23:30:03.718702 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:03.718556    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw" podStartSLOduration=1.718530142 podStartE2EDuration="1.718530142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 23:30:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:30:03.714606043 +0000 UTC m=+1727.747735342" watchObservedRunningTime="2026-09-11 23:30:03.718530142 +0000 UTC m=+1727.751659361"
Sep 11 23:30:03.912650 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:03.912599    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-97d9dc-29819490-7b6mr"
Sep 11 23:30:03.950869 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:03.950841    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-12fb6b-29819490-z7ztp"
Sep 11 23:30:03.954988 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:03.954962    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cd48eb-29819490-m2hcj"
Sep 11 23:30:03.959212 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:03.959188    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819490-kwbtx"
Sep 11 23:30:04.158206 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:04.158133    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw"
Sep 11 23:30:04.163428 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:04.163402    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw"
Sep 11 23:30:04.701065 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:04.701031    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819490-kwbtx"
Sep 11 23:30:04.701065 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:04.701040    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819490-kwbtx" event={"ID":"d55d452e-62be-494c-a30f-ee01a215943a","Type":"ContainerDied","Data":"b7516fdfcc6c26793d9e9cad07b498e4b3d558da7b03edfd0733209dfd034cd7"}
Sep 11 23:30:04.701560 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:04.701104    2546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b7516fdfcc6c26793d9e9cad07b498e4b3d558da7b03edfd0733209dfd034cd7"
Sep 11 23:30:04.702518 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:04.702501    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cd48eb-29819490-m2hcj"
Sep 11 23:30:04.702625 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:04.702554    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cd48eb-29819490-m2hcj" event={"ID":"38d1bf05-fa75-4d8b-9f46-f843f80c0f86","Type":"ContainerDied","Data":"1a22a7a06880f3018ae3fa455cc0a5a22284a90eb0b9b12d2a21e13e5852a390"}
Sep 11 23:30:04.702625 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:04.702603    2546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a22a7a06880f3018ae3fa455cc0a5a22284a90eb0b9b12d2a21e13e5852a390"
Sep 11 23:30:04.704701 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:04.704296    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-12fb6b-29819490-z7ztp"
Sep 11 23:30:04.704701 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:04.704301    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-12fb6b-29819490-z7ztp" event={"ID":"29b56839-d428-44de-9434-96fc011d873b","Type":"ContainerDied","Data":"7c5b58d9964e7f02fb4c01b868d3d60e3481ae07f3a6abd84f990fac45188cb2"}
Sep 11 23:30:04.704701 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:04.704428    2546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c5b58d9964e7f02fb4c01b868d3d60e3481ae07f3a6abd84f990fac45188cb2"
Sep 11 23:30:04.705944 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:04.705918    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-97d9dc-29819490-7b6mr" event={"ID":"b1076ba3-1d97-4eda-9e02-52ef3eb30ff1","Type":"ContainerDied","Data":"10be7a9f9855fb7c19d0c9e2ad1d413baf340d03b2b9a4f3317611c670cc48e3"}
Sep 11 23:30:04.706070 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:04.705963    2546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10be7a9f9855fb7c19d0c9e2ad1d413baf340d03b2b9a4f3317611c670cc48e3"
Sep 11 23:30:04.706070 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:04.705984    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-97d9dc-29819490-7b6mr"
Sep 11 23:30:04.706406 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:04.706385    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw"
Sep 11 23:30:04.707440 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:04.707399    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw"
Sep 11 23:30:12.475364 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:12.475327    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-451902-d8d4b8fdb-pwgs4"]
Sep 11 23:30:12.482029 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:12.476110    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d55d452e-62be-494c-a30f-ee01a215943a" containerName="cleanup"
Sep 11 23:30:12.482029 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:12.476140    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="d55d452e-62be-494c-a30f-ee01a215943a" containerName="cleanup"
Sep 11 23:30:12.482029 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:12.476178    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29b56839-d428-44de-9434-96fc011d873b" containerName="cleanup"
Sep 11 23:30:12.482029 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:12.476187    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="29b56839-d428-44de-9434-96fc011d873b" containerName="cleanup"
Sep 11 23:30:12.482029 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:12.476224    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38d1bf05-fa75-4d8b-9f46-f843f80c0f86" containerName="cleanup"
Sep 11 23:30:12.482029 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:12.476232    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="38d1bf05-fa75-4d8b-9f46-f843f80c0f86" containerName="cleanup"
Sep 11 23:30:12.482029 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:12.476262    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1076ba3-1d97-4eda-9e02-52ef3eb30ff1" containerName="cleanup"
Sep 11 23:30:12.482029 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:12.476270    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1076ba3-1d97-4eda-9e02-52ef3eb30ff1" containerName="cleanup"
Sep 11 23:30:12.482029 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:12.476430    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="38d1bf05-fa75-4d8b-9f46-f843f80c0f86" containerName="cleanup"
Sep 11 23:30:12.482029 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:12.476448    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="29b56839-d428-44de-9434-96fc011d873b" containerName="cleanup"
Sep 11 23:30:12.482029 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:12.476467    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="d55d452e-62be-494c-a30f-ee01a215943a" containerName="cleanup"
Sep 11 23:30:12.482029 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:12.476488    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1076ba3-1d97-4eda-9e02-52ef3eb30ff1" containerName="cleanup"
Sep 11 23:30:12.484373 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:12.484348    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-451902-d8d4b8fdb-pwgs4"
Sep 11 23:30:12.492203 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:12.492176    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-451902-dockercfg-9rg9d\""
Sep 11 23:30:12.514748 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:12.494678    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-451902\""
Sep 11 23:30:12.525051 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:12.524925    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-451902-d8d4b8fdb-pwgs4"]
Sep 11 23:30:12.531704 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:12.530169    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rx7w\" (UniqueName: \"kubernetes.io/projected/728bdfa7-251f-4bd8-a725-6b19a7aa7373-kube-api-access-8rx7w\") pod \"payload-pre-processing-e2e-api-a-451902-d8d4b8fdb-pwgs4\" (UID: \"728bdfa7-251f-4bd8-a725-6b19a7aa7373\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-451902-d8d4b8fdb-pwgs4"
Sep 11 23:30:12.531986 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:12.531966    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/728bdfa7-251f-4bd8-a725-6b19a7aa7373-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-451902-d8d4b8fdb-pwgs4\" (UID: \"728bdfa7-251f-4bd8-a725-6b19a7aa7373\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-451902-d8d4b8fdb-pwgs4"
Sep 11 23:30:12.633140 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:12.633105    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rx7w\" (UniqueName: \"kubernetes.io/projected/728bdfa7-251f-4bd8-a725-6b19a7aa7373-kube-api-access-8rx7w\") pod \"payload-pre-processing-e2e-api-a-451902-d8d4b8fdb-pwgs4\" (UID: \"728bdfa7-251f-4bd8-a725-6b19a7aa7373\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-451902-d8d4b8fdb-pwgs4"
Sep 11 23:30:12.633320 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:12.633192    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/728bdfa7-251f-4bd8-a725-6b19a7aa7373-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-451902-d8d4b8fdb-pwgs4\" (UID: \"728bdfa7-251f-4bd8-a725-6b19a7aa7373\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-451902-d8d4b8fdb-pwgs4"
Sep 11 23:30:12.633846 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:12.633828    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/728bdfa7-251f-4bd8-a725-6b19a7aa7373-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-451902-d8d4b8fdb-pwgs4\" (UID: \"728bdfa7-251f-4bd8-a725-6b19a7aa7373\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-451902-d8d4b8fdb-pwgs4"
Sep 11 23:30:12.646903 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:12.646879    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rx7w\" (UniqueName: \"kubernetes.io/projected/728bdfa7-251f-4bd8-a725-6b19a7aa7373-kube-api-access-8rx7w\") pod \"payload-pre-processing-e2e-api-a-451902-d8d4b8fdb-pwgs4\" (UID: \"728bdfa7-251f-4bd8-a725-6b19a7aa7373\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-451902-d8d4b8fdb-pwgs4"
Sep 11 23:30:12.835380 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:12.835278    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-451902-d8d4b8fdb-pwgs4"
Sep 11 23:30:13.119974 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:13.119928    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-451902-d8d4b8fdb-pwgs4"]
Sep 11 23:30:13.120213 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:30:13.120140    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod728bdfa7_251f_4bd8_a725_6b19a7aa7373.slice/crio-732700dd8bff3df59678907ce968274acf1b1284f4683109633d73640e838197 WatchSource:0}: Error finding container 732700dd8bff3df59678907ce968274acf1b1284f4683109633d73640e838197: Status 404 returned error can't find the container with id 732700dd8bff3df59678907ce968274acf1b1284f4683109633d73640e838197
Sep 11 23:30:13.770592 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:13.770547    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-451902-d8d4b8fdb-pwgs4" event={"ID":"728bdfa7-251f-4bd8-a725-6b19a7aa7373","Type":"ContainerStarted","Data":"9c039e1804c2503eb01146e349a150e1a5af76ba24eb60d7d2a01ce61ec8b966"}
Sep 11 23:30:13.770953 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:13.770745    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-451902-d8d4b8fdb-pwgs4"
Sep 11 23:30:13.770953 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:13.770776    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-451902-d8d4b8fdb-pwgs4" event={"ID":"728bdfa7-251f-4bd8-a725-6b19a7aa7373","Type":"ContainerStarted","Data":"732700dd8bff3df59678907ce968274acf1b1284f4683109633d73640e838197"}
Sep 11 23:30:13.798706 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:13.798638    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-451902-d8d4b8fdb-pwgs4" podStartSLOduration=1.798613612 podStartE2EDuration="1.798613612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 23:30:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:30:13.794784946 +0000 UTC m=+1737.827914155" watchObservedRunningTime="2026-09-11 23:30:13.798613612 +0000 UTC m=+1737.831742829"
Sep 11 23:30:23.896786 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:23.896746    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"]
Sep 11 23:30:24.780362 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:24.780333    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-451902-d8d4b8fdb-pwgs4"
Sep 11 23:30:31.374657 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:31.374620    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-12fb6b-rjfjk"]
Sep 11 23:30:31.379045 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:31.379024    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-12fb6b-rjfjk"
Sep 11 23:30:31.382621 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:31.382596    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 23:30:31.382740 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:31.382609    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vrx7m\""
Sep 11 23:30:31.389472 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:31.389444    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-12fb6b-rjfjk"]
Sep 11 23:30:31.465900 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:31.465860    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/667435a0-dc03-4efe-8edd-d743b5a2efb0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-12fb6b-rjfjk\" (UID: \"667435a0-dc03-4efe-8edd-d743b5a2efb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-12fb6b-rjfjk"
Sep 11 23:30:31.566995 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:31.566954    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/667435a0-dc03-4efe-8edd-d743b5a2efb0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-12fb6b-rjfjk\" (UID: \"667435a0-dc03-4efe-8edd-d743b5a2efb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-12fb6b-rjfjk"
Sep 11 23:30:31.567617 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:31.567589    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/667435a0-dc03-4efe-8edd-d743b5a2efb0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-12fb6b-rjfjk\" (UID: \"667435a0-dc03-4efe-8edd-d743b5a2efb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-12fb6b-rjfjk"
Sep 11 23:30:31.692300 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:31.692263    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-12fb6b-rjfjk"
Sep 11 23:30:31.988256 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:31.988225    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-12fb6b-rjfjk"]
Sep 11 23:30:31.988745 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:30:31.988718    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod667435a0_dc03_4efe_8edd_d743b5a2efb0.slice/crio-2eae70494076decaacfb7b78127346186e247179ad4402a62f2972d31fc9577d WatchSource:0}: Error finding container 2eae70494076decaacfb7b78127346186e247179ad4402a62f2972d31fc9577d: Status 404 returned error can't find the container with id 2eae70494076decaacfb7b78127346186e247179ad4402a62f2972d31fc9577d
Sep 11 23:30:32.911164 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:32.911108    2546 generic.go:356] "Generic (PLEG): container finished" podID="667435a0-dc03-4efe-8edd-d743b5a2efb0" containerID="afa50b0afaf4647971b34dc1b281837dbe3f24c5b03ec0cf932508393b684264" exitCode=0
Sep 11 23:30:32.911638 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:32.911205    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-12fb6b-rjfjk" event={"ID":"667435a0-dc03-4efe-8edd-d743b5a2efb0","Type":"ContainerDied","Data":"afa50b0afaf4647971b34dc1b281837dbe3f24c5b03ec0cf932508393b684264"}
Sep 11 23:30:32.911638 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:32.911274    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-12fb6b-rjfjk" event={"ID":"667435a0-dc03-4efe-8edd-d743b5a2efb0","Type":"ContainerStarted","Data":"2eae70494076decaacfb7b78127346186e247179ad4402a62f2972d31fc9577d"}
Sep 11 23:30:34.081526 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:34.081492    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-12fb6b-rjfjk"
Sep 11 23:30:34.198216 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:34.198109    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/667435a0-dc03-4efe-8edd-d743b5a2efb0-maas-api-service-ca\") pod \"667435a0-dc03-4efe-8edd-d743b5a2efb0\" (UID: \"667435a0-dc03-4efe-8edd-d743b5a2efb0\") "
Sep 11 23:30:34.198548 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:34.198517    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/667435a0-dc03-4efe-8edd-d743b5a2efb0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "667435a0-dc03-4efe-8edd-d743b5a2efb0" (UID: "667435a0-dc03-4efe-8edd-d743b5a2efb0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 23:30:34.299174 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:34.299127    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/667435a0-dc03-4efe-8edd-d743b5a2efb0-maas-api-service-ca\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:30:34.929108 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:34.929064    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-12fb6b-rjfjk" event={"ID":"667435a0-dc03-4efe-8edd-d743b5a2efb0","Type":"ContainerDied","Data":"2eae70494076decaacfb7b78127346186e247179ad4402a62f2972d31fc9577d"}
Sep 11 23:30:34.931360 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:34.929124    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-12fb6b-rjfjk"
Sep 11 23:30:34.931360 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:34.929134    2546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2eae70494076decaacfb7b78127346186e247179ad4402a62f2972d31fc9577d"
Sep 11 23:30:34.949676 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:34.949644    2546 patch_prober.go:28] interesting pod/e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": read tcp 10.132.0.2:41738->10.132.0.85:15021: read: connection reset by peer" start-of-body=
Sep 11 23:30:34.949811 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:34.949756    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw" podUID="6e0419a9-c88a-4f69-bd07-751d5406fd0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": read tcp 10.132.0.2:41738->10.132.0.85:15021: read: connection reset by peer"
Sep 11 23:30:35.938506 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:35.938474    2546 generic.go:356] "Generic (PLEG): container finished" podID="6e0419a9-c88a-4f69-bd07-751d5406fd0e" containerID="cfcb19e22a3bcd059b4ec46499e729ef235b12aaa2b5b0ee8e3b3722d0290412" exitCode=0
Sep 11 23:30:35.939033 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:35.938542    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw" event={"ID":"6e0419a9-c88a-4f69-bd07-751d5406fd0e","Type":"ContainerDied","Data":"cfcb19e22a3bcd059b4ec46499e729ef235b12aaa2b5b0ee8e3b3722d0290412"}
Sep 11 23:30:35.939233 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:35.939215    2546 scope.go:117] "RemoveContainer" containerID="cfcb19e22a3bcd059b4ec46499e729ef235b12aaa2b5b0ee8e3b3722d0290412"
Sep 11 23:30:35.941091 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:35.941057    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:30:35.941215 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:35.941181    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:30:35.941291 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:35.941236    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:30:36.236893 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:36.236808    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-12fb6b/distinct-2-w0-12fb6b-kserve-56c4887795-bf8sz"]
Sep 11 23:30:36.249027 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:36.248985    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-12fb6b/embedding-w0-12fb6b-kserve-585bc8ccb5-q5647"]
Sep 11 23:30:36.262423 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:36.262374    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-12fb6b/premium-w0-12fb6b-kserve-557db4c777-5zbbn"]
Sep 11 23:30:36.275564 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:36.275527    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-12fb6b/simulator-w0-12fb6b-kserve-9749df7b4-859hf"]
Sep 11 23:30:36.422382 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:36.422350    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8"]
Sep 11 23:30:36.422845 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:36.422832    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="667435a0-dc03-4efe-8edd-d743b5a2efb0" containerName="revoke-keys"
Sep 11 23:30:36.422890 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:36.422848    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="667435a0-dc03-4efe-8edd-d743b5a2efb0" containerName="revoke-keys"
Sep 11 23:30:36.422963 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:36.422955    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="667435a0-dc03-4efe-8edd-d743b5a2efb0" containerName="revoke-keys"
Sep 11 23:30:36.426849 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:36.426823    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8"
Sep 11 23:30:36.429852 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:36.429824    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-d1b012-openshift-default-dockercfg-dbjhc\""
Sep 11 23:30:36.441358 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:36.441327    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8"]
Sep 11 23:30:36.522874 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:36.522779    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/181669c0-2cfb-463e-9dd2-3c4c874b8442-workload-certs\") pod \"e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8\" (UID: \"181669c0-2cfb-463e-9dd2-3c4c874b8442\") " pod="openshift-ingress/e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8"
Sep 11 23:30:36.522874 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:36.522842    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/181669c0-2cfb-463e-9dd2-3c4c874b8442-istio-envoy\") pod \"e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8\" (UID: \"181669c0-2cfb-463e-9dd2-3c4c874b8442\") " pod="openshift-ingress/e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8"
Sep 11 23:30:36.523108 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:36.522884    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/181669c0-2cfb-463e-9dd2-3c4c874b8442-istio-data\") pod \"e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8\" (UID: \"181669c0-2cfb-463e-9dd2-3c4c874b8442\") " pod="openshift-ingress/e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8"
Sep 11 23:30:36.523108 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:36.522951    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzl72\" (UniqueName: \"kubernetes.io/projected/181669c0-2cfb-463e-9dd2-3c4c874b8442-kube-api-access-xzl72\") pod \"e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8\" (UID: \"181669c0-2cfb-463e-9dd2-3c4c874b8442\") " pod="openshift-ingress/e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8"
Sep 11 23:30:36.523108 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:36.522996    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/181669c0-2cfb-463e-9dd2-3c4c874b8442-istio-token\") pod \"e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8\" (UID: \"181669c0-2cfb-463e-9dd2-3c4c874b8442\") " pod="openshift-ingress/e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8"
Sep 11 23:30:36.523108 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:36.523023    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/181669c0-2cfb-463e-9dd2-3c4c874b8442-istio-podinfo\") pod \"e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8\" (UID: \"181669c0-2cfb-463e-9dd2-3c4c874b8442\") " pod="openshift-ingress/e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8"
Sep 11 23:30:36.523375 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:36.523110    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/181669c0-2cfb-463e-9dd2-3c4c874b8442-credential-socket\") pod \"e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8\" (UID: \"181669c0-2cfb-463e-9dd2-3c4c874b8442\") " pod="openshift-ingress/e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8"
Sep 11 23:30:36.523375 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:36.523216    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/181669c0-2cfb-463e-9dd2-3c4c874b8442-workload-socket\") pod \"e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8\" (UID: \"181669c0-2cfb-463e-9dd2-3c4c874b8442\") " pod="openshift-ingress/e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8"
Sep 11 23:30:36.523375 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:36.523244    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/181669c0-2cfb-463e-9dd2-3c4c874b8442-istiod-ca-cert\") pod \"e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8\" (UID: \"181669c0-2cfb-463e-9dd2-3c4c874b8442\") " pod="openshift-ingress/e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8"
Sep 11 23:30:36.623987 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:36.623949    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/181669c0-2cfb-463e-9dd2-3c4c874b8442-credential-socket\") pod \"e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8\" (UID: \"181669c0-2cfb-463e-9dd2-3c4c874b8442\") " pod="openshift-ingress/e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8"
Sep 11 23:30:36.624382 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:36.624348    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/181669c0-2cfb-463e-9dd2-3c4c874b8442-workload-socket\") pod \"e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8\" (UID: \"181669c0-2cfb-463e-9dd2-3c4c874b8442\") " pod="openshift-ingress/e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8"
Sep 11 23:30:36.624537 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:36.624396    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/181669c0-2cfb-463e-9dd2-3c4c874b8442-istiod-ca-cert\") pod \"e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8\" (UID: \"181669c0-2cfb-463e-9dd2-3c4c874b8442\") " pod="openshift-ingress/e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8"
Sep 11 23:30:36.624537 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:36.624460    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/181669c0-2cfb-463e-9dd2-3c4c874b8442-workload-certs\") pod \"e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8\" (UID: \"181669c0-2cfb-463e-9dd2-3c4c874b8442\") " pod="openshift-ingress/e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8"
Sep 11 23:30:36.624675 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:36.624555    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/181669c0-2cfb-463e-9dd2-3c4c874b8442-istio-envoy\") pod \"e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8\" (UID: \"181669c0-2cfb-463e-9dd2-3c4c874b8442\") " pod="openshift-ingress/e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8"
Sep 11 23:30:36.624675 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:36.624633    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/181669c0-2cfb-463e-9dd2-3c4c874b8442-istio-data\") pod \"e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8\" (UID: \"181669c0-2cfb-463e-9dd2-3c4c874b8442\") " pod="openshift-ingress/e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8"
Sep 11 23:30:36.624800 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:36.624690    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzl72\" (UniqueName: \"kubernetes.io/projected/181669c0-2cfb-463e-9dd2-3c4c874b8442-kube-api-access-xzl72\") pod \"e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8\" (UID: \"181669c0-2cfb-463e-9dd2-3c4c874b8442\") " pod="openshift-ingress/e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8"
Sep 11 23:30:36.624800 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:36.624703    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/181669c0-2cfb-463e-9dd2-3c4c874b8442-workload-socket\") pod \"e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8\" (UID: \"181669c0-2cfb-463e-9dd2-3c4c874b8442\") " pod="openshift-ingress/e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8"
Sep 11 23:30:36.624800 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:36.624724    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/181669c0-2cfb-463e-9dd2-3c4c874b8442-istio-token\") pod \"e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8\" (UID: \"181669c0-2cfb-463e-9dd2-3c4c874b8442\") " pod="openshift-ingress/e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8"
Sep 11 23:30:36.624800 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:36.624708    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/181669c0-2cfb-463e-9dd2-3c4c874b8442-credential-socket\") pod \"e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8\" (UID: \"181669c0-2cfb-463e-9dd2-3c4c874b8442\") " pod="openshift-ingress/e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8"
Sep 11 23:30:36.624800 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:36.624766    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/181669c0-2cfb-463e-9dd2-3c4c874b8442-istio-podinfo\") pod \"e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8\" (UID: \"181669c0-2cfb-463e-9dd2-3c4c874b8442\") " pod="openshift-ingress/e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8"
Sep 11 23:30:36.625085 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:36.624990    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/181669c0-2cfb-463e-9dd2-3c4c874b8442-workload-certs\") pod \"e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8\" (UID: \"181669c0-2cfb-463e-9dd2-3c4c874b8442\") " pod="openshift-ingress/e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8"
Sep 11 23:30:36.625494 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:36.625464    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/181669c0-2cfb-463e-9dd2-3c4c874b8442-istiod-ca-cert\") pod \"e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8\" (UID: \"181669c0-2cfb-463e-9dd2-3c4c874b8442\") " pod="openshift-ingress/e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8"
Sep 11 23:30:36.625649 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:36.625626    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/181669c0-2cfb-463e-9dd2-3c4c874b8442-istio-data\") pod \"e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8\" (UID: \"181669c0-2cfb-463e-9dd2-3c4c874b8442\") " pod="openshift-ingress/e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8"
Sep 11 23:30:36.628552 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:36.628504    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/181669c0-2cfb-463e-9dd2-3c4c874b8442-istio-envoy\") pod \"e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8\" (UID: \"181669c0-2cfb-463e-9dd2-3c4c874b8442\") " pod="openshift-ingress/e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8"
Sep 11 23:30:36.629475 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:36.629445    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/181669c0-2cfb-463e-9dd2-3c4c874b8442-istio-podinfo\") pod \"e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8\" (UID: \"181669c0-2cfb-463e-9dd2-3c4c874b8442\") " pod="openshift-ingress/e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8"
Sep 11 23:30:36.643771 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:36.643743    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzl72\" (UniqueName: \"kubernetes.io/projected/181669c0-2cfb-463e-9dd2-3c4c874b8442-kube-api-access-xzl72\") pod \"e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8\" (UID: \"181669c0-2cfb-463e-9dd2-3c4c874b8442\") " pod="openshift-ingress/e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8"
Sep 11 23:30:36.644619 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:36.644600    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/181669c0-2cfb-463e-9dd2-3c4c874b8442-istio-token\") pod \"e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8\" (UID: \"181669c0-2cfb-463e-9dd2-3c4c874b8442\") " pod="openshift-ingress/e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8"
Sep 11 23:30:36.742218 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:36.742148    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8"
Sep 11 23:30:36.947766 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:36.947726    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw" event={"ID":"6e0419a9-c88a-4f69-bd07-751d5406fd0e","Type":"ContainerStarted","Data":"83c2605234559cac0dd49cd5f60cab1a058eeca48d4e5d97141a988ac00adbdc"}
Sep 11 23:30:37.031991 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:30:37.031957    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod181669c0_2cfb_463e_9dd2_3c4c874b8442.slice/crio-3230fdaf1734f5cf2774703ff1f04dfb425429bc5570c6ed9fe08bf6f884ea8e WatchSource:0}: Error finding container 3230fdaf1734f5cf2774703ff1f04dfb425429bc5570c6ed9fe08bf6f884ea8e: Status 404 returned error can't find the container with id 3230fdaf1734f5cf2774703ff1f04dfb425429bc5570c6ed9fe08bf6f884ea8e
Sep 11 23:30:37.032345 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:37.032319    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8"]
Sep 11 23:30:37.034537 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:37.034502    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:30:37.034667 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:37.034581    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:30:37.034667 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:37.034629    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:30:37.158336 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:37.158304    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw"
Sep 11 23:30:37.158537 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:37.158510    2546 patch_prober.go:28] interesting pod/e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Sep 11 23:30:37.158594 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:37.158574    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw" podUID="6e0419a9-c88a-4f69-bd07-751d5406fd0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Sep 11 23:30:37.959256 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:37.959213    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8" event={"ID":"181669c0-2cfb-463e-9dd2-3c4c874b8442","Type":"ContainerStarted","Data":"c9b554fd3b7cc40398cff34d03ae89630868a2922b8f87f6b027516fd5ca09c5"}
Sep 11 23:30:37.959748 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:37.959269    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8" event={"ID":"181669c0-2cfb-463e-9dd2-3c4c874b8442","Type":"ContainerStarted","Data":"3230fdaf1734f5cf2774703ff1f04dfb425429bc5570c6ed9fe08bf6f884ea8e"}
Sep 11 23:30:37.987169 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:37.987086    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8" podStartSLOduration=1.987070575 podStartE2EDuration="1.987070575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 23:30:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:30:37.982405916 +0000 UTC m=+1762.015535134" watchObservedRunningTime="2026-09-11 23:30:37.987070575 +0000 UTC m=+1762.020199792"
Sep 11 23:30:38.158981 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:38.158939    2546 patch_prober.go:28] interesting pod/e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Sep 11 23:30:38.159218 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:38.158997    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw" podUID="6e0419a9-c88a-4f69-bd07-751d5406fd0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Sep 11 23:30:38.742778 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:38.742727    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8"
Sep 11 23:30:38.752741 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:38.752694    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8"
Sep 11 23:30:38.967650 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:38.967614    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8"
Sep 11 23:30:38.968983 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:38.968959    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8"
Sep 11 23:30:39.158345 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:39.158317    2546 patch_prober.go:28] interesting pod/e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Sep 11 23:30:39.158512 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:39.158365    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw" podUID="6e0419a9-c88a-4f69-bd07-751d5406fd0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Sep 11 23:30:40.158583 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:40.158539    2546 patch_prober.go:28] interesting pod/e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Sep 11 23:30:40.158992 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:40.158606    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw" podUID="6e0419a9-c88a-4f69-bd07-751d5406fd0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Sep 11 23:30:41.158498 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:41.158460    2546 patch_prober.go:28] interesting pod/e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Sep 11 23:30:41.158690 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:41.158527    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw" podUID="6e0419a9-c88a-4f69-bd07-751d5406fd0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Sep 11 23:30:42.158383 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:42.158344    2546 patch_prober.go:28] interesting pod/e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Sep 11 23:30:42.158575 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:42.158405    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw" podUID="6e0419a9-c88a-4f69-bd07-751d5406fd0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Sep 11 23:30:43.163000 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:43.162967    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw"
Sep 11 23:30:43.163444 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:43.163423    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw"
Sep 11 23:30:43.164320 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:43.164293    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw"
Sep 11 23:30:43.821085 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:43.821050    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d1b012-59676779d6-48drk"]
Sep 11 23:30:43.825450 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:43.825422    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d1b012-59676779d6-48drk"
Sep 11 23:30:43.829285 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:43.829255    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-d1b012\""
Sep 11 23:30:43.829441 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:43.829374    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-d1b012-dockercfg-9hzm8\""
Sep 11 23:30:43.843764 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:43.843731    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d1b012-59676779d6-48drk"]
Sep 11 23:30:43.911658 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:43.911611    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tktbg\" (UniqueName: \"kubernetes.io/projected/05ebbb64-3875-4dfa-af7f-7f68dfded62a-kube-api-access-tktbg\") pod \"payload-pre-processing-e2e-api-b-d1b012-59676779d6-48drk\" (UID: \"05ebbb64-3875-4dfa-af7f-7f68dfded62a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d1b012-59676779d6-48drk"
Sep 11 23:30:43.911877 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:43.911725    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05ebbb64-3875-4dfa-af7f-7f68dfded62a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d1b012-59676779d6-48drk\" (UID: \"05ebbb64-3875-4dfa-af7f-7f68dfded62a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d1b012-59676779d6-48drk"
Sep 11 23:30:44.012670 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:44.012630    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tktbg\" (UniqueName: \"kubernetes.io/projected/05ebbb64-3875-4dfa-af7f-7f68dfded62a-kube-api-access-tktbg\") pod \"payload-pre-processing-e2e-api-b-d1b012-59676779d6-48drk\" (UID: \"05ebbb64-3875-4dfa-af7f-7f68dfded62a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d1b012-59676779d6-48drk"
Sep 11 23:30:44.012985 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:44.012967    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05ebbb64-3875-4dfa-af7f-7f68dfded62a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d1b012-59676779d6-48drk\" (UID: \"05ebbb64-3875-4dfa-af7f-7f68dfded62a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d1b012-59676779d6-48drk"
Sep 11 23:30:44.013963 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:44.013936    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05ebbb64-3875-4dfa-af7f-7f68dfded62a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d1b012-59676779d6-48drk\" (UID: \"05ebbb64-3875-4dfa-af7f-7f68dfded62a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d1b012-59676779d6-48drk"
Sep 11 23:30:44.026334 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:44.026306    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tktbg\" (UniqueName: \"kubernetes.io/projected/05ebbb64-3875-4dfa-af7f-7f68dfded62a-kube-api-access-tktbg\") pod \"payload-pre-processing-e2e-api-b-d1b012-59676779d6-48drk\" (UID: \"05ebbb64-3875-4dfa-af7f-7f68dfded62a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d1b012-59676779d6-48drk"
Sep 11 23:30:44.139002 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:44.138898    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d1b012-59676779d6-48drk"
Sep 11 23:30:44.417867 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:44.417838    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d1b012-59676779d6-48drk"]
Sep 11 23:30:44.418761 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:30:44.418738    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05ebbb64_3875_4dfa_af7f_7f68dfded62a.slice/crio-650af064d45b4f2e517ab0df357406afff7fa00f27f62c581e8b3f12b302d97f WatchSource:0}: Error finding container 650af064d45b4f2e517ab0df357406afff7fa00f27f62c581e8b3f12b302d97f: Status 404 returned error can't find the container with id 650af064d45b4f2e517ab0df357406afff7fa00f27f62c581e8b3f12b302d97f
Sep 11 23:30:45.027711 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:45.027671    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d1b012-59676779d6-48drk" event={"ID":"05ebbb64-3875-4dfa-af7f-7f68dfded62a","Type":"ContainerStarted","Data":"7bb8bbead719dd86d6f37347f22bf15ca3c7bfada9d3a01fc0380d1322a817e9"}
Sep 11 23:30:45.027930 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:45.027770    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d1b012-59676779d6-48drk" event={"ID":"05ebbb64-3875-4dfa-af7f-7f68dfded62a","Type":"ContainerStarted","Data":"650af064d45b4f2e517ab0df357406afff7fa00f27f62c581e8b3f12b302d97f"}
Sep 11 23:30:45.028203 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:45.028178    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d1b012-59676779d6-48drk"
Sep 11 23:30:45.047969 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:45.047913    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d1b012-59676779d6-48drk" podStartSLOduration=2.047895827 podStartE2EDuration="2.047895827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 23:30:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:30:45.046305814 +0000 UTC m=+1769.079435033" watchObservedRunningTime="2026-09-11 23:30:45.047895827 +0000 UTC m=+1769.081025093"
Sep 11 23:30:51.260826 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:51.260777    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-12fb6b/distinct-2-w0-12fb6b-kserve-56c4887795-bf8sz" podUID="9da6fe34-d6b1-4b92-a067-e2677f71bca9" containerName="main" containerID="cri-o://328ba10a98d3d824ec19b37471e0a837842a2a0f8bd38a173fad6257b723a171" gracePeriod=45
Sep 11 23:30:51.273898 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:51.273859    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-12fb6b/embedding-w0-12fb6b-kserve-585bc8ccb5-q5647" podUID="dc328101-34e6-4746-9e4f-3e58116a6e70" containerName="main" containerID="cri-o://f8b435be85ec4a7dc300a3569204be232a116a36d8bb2fd5d5542684d39259fe" gracePeriod=45
Sep 11 23:30:51.283802 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:51.283766    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-12fb6b/premium-w0-12fb6b-kserve-557db4c777-5zbbn" podUID="74942f9b-139d-4bee-9380-c251ac6dc246" containerName="main" containerID="cri-o://f842f454d6c900755eaf2f8137c9d8810ad3a6e983d8a1c9e7171fbfc8c7f708" gracePeriod=45
Sep 11 23:30:51.298722 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:51.298679    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-12fb6b/simulator-w0-12fb6b-kserve-9749df7b4-859hf" podUID="2d4d86af-7f04-4bcf-ad33-fb1fed658519" containerName="main" containerID="cri-o://d87a308915173a6be103b44404ac88d92021424ebc3386d922e48d83f1985a23" gracePeriod=45
Sep 11 23:30:51.573409 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:51.573376    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-12fb6b/distinct-2-w0-12fb6b-kserve-56c4887795-bf8sz"
Sep 11 23:30:51.638444 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:51.638403    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-12fb6b/premium-w0-12fb6b-kserve-557db4c777-5zbbn"
Sep 11 23:30:51.659761 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:51.659649    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-12fb6b/embedding-w0-12fb6b-kserve-585bc8ccb5-q5647"
Sep 11 23:30:51.661949 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:51.661916    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-12fb6b/simulator-w0-12fb6b-kserve-9749df7b4-859hf"
Sep 11 23:30:51.703926 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:51.703843    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74942f9b-139d-4bee-9380-c251ac6dc246-tmp-dir\") pod \"74942f9b-139d-4bee-9380-c251ac6dc246\" (UID: \"74942f9b-139d-4bee-9380-c251ac6dc246\") "
Sep 11 23:30:51.704194 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:51.704129    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9da6fe34-d6b1-4b92-a067-e2677f71bca9-home\") pod \"9da6fe34-d6b1-4b92-a067-e2677f71bca9\" (UID: \"9da6fe34-d6b1-4b92-a067-e2677f71bca9\") "
Sep 11 23:30:51.704194 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:51.704195    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qql8n\" (UniqueName: \"kubernetes.io/projected/9da6fe34-d6b1-4b92-a067-e2677f71bca9-kube-api-access-qql8n\") pod \"9da6fe34-d6b1-4b92-a067-e2677f71bca9\" (UID: \"9da6fe34-d6b1-4b92-a067-e2677f71bca9\") "
Sep 11 23:30:51.704391 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:51.704254    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9da6fe34-d6b1-4b92-a067-e2677f71bca9-dshm\") pod \"9da6fe34-d6b1-4b92-a067-e2677f71bca9\" (UID: \"9da6fe34-d6b1-4b92-a067-e2677f71bca9\") "
Sep 11 23:30:51.704391 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:51.704293    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74942f9b-139d-4bee-9380-c251ac6dc246-home\") pod \"74942f9b-139d-4bee-9380-c251ac6dc246\" (UID: \"74942f9b-139d-4bee-9380-c251ac6dc246\") "
Sep 11 23:30:51.704391 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:51.704312    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9da6fe34-d6b1-4b92-a067-e2677f71bca9-model-cache\") pod \"9da6fe34-d6b1-4b92-a067-e2677f71bca9\" (UID: \"9da6fe34-d6b1-4b92-a067-e2677f71bca9\") "
Sep 11 23:30:51.704391 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:51.704327    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9da6fe34-d6b1-4b92-a067-e2677f71bca9-tls-certs\") pod \"9da6fe34-d6b1-4b92-a067-e2677f71bca9\" (UID: \"9da6fe34-d6b1-4b92-a067-e2677f71bca9\") "
Sep 11 23:30:51.704391 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:51.704385    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9da6fe34-d6b1-4b92-a067-e2677f71bca9-tmp-dir\") pod \"9da6fe34-d6b1-4b92-a067-e2677f71bca9\" (UID: \"9da6fe34-d6b1-4b92-a067-e2677f71bca9\") "
Sep 11 23:30:51.704776 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:51.704410    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74942f9b-139d-4bee-9380-c251ac6dc246-dshm\") pod \"74942f9b-139d-4bee-9380-c251ac6dc246\" (UID: \"74942f9b-139d-4bee-9380-c251ac6dc246\") "
Sep 11 23:30:51.704776 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:51.704437    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jflrm\" (UniqueName: \"kubernetes.io/projected/74942f9b-139d-4bee-9380-c251ac6dc246-kube-api-access-jflrm\") pod \"74942f9b-139d-4bee-9380-c251ac6dc246\" (UID: \"74942f9b-139d-4bee-9380-c251ac6dc246\") "
Sep 11 23:30:51.704776 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:51.704465    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74942f9b-139d-4bee-9380-c251ac6dc246-model-cache\") pod \"74942f9b-139d-4bee-9380-c251ac6dc246\" (UID: \"74942f9b-139d-4bee-9380-c251ac6dc246\") "
Sep 11 23:30:51.704776 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:51.704512    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74942f9b-139d-4bee-9380-c251ac6dc246-tls-certs\") pod \"74942f9b-139d-4bee-9380-c251ac6dc246\" (UID: \"74942f9b-139d-4bee-9380-c251ac6dc246\") "
Sep 11 23:30:51.705709 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:51.705679    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9da6fe34-d6b1-4b92-a067-e2677f71bca9-model-cache" (OuterVolumeSpecName: "model-cache") pod "9da6fe34-d6b1-4b92-a067-e2677f71bca9" (UID: "9da6fe34-d6b1-4b92-a067-e2677f71bca9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:30:51.705885 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:51.705863    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74942f9b-139d-4bee-9380-c251ac6dc246-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "74942f9b-139d-4bee-9380-c251ac6dc246" (UID: "74942f9b-139d-4bee-9380-c251ac6dc246"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:30:51.706072 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:51.706030    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9da6fe34-d6b1-4b92-a067-e2677f71bca9-home" (OuterVolumeSpecName: "home") pod "9da6fe34-d6b1-4b92-a067-e2677f71bca9" (UID: "9da6fe34-d6b1-4b92-a067-e2677f71bca9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:30:51.709364 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:51.709324    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9da6fe34-d6b1-4b92-a067-e2677f71bca9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9da6fe34-d6b1-4b92-a067-e2677f71bca9" (UID: "9da6fe34-d6b1-4b92-a067-e2677f71bca9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:30:51.709883 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:51.709399    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74942f9b-139d-4bee-9380-c251ac6dc246-home" (OuterVolumeSpecName: "home") pod "74942f9b-139d-4bee-9380-c251ac6dc246" (UID: "74942f9b-139d-4bee-9380-c251ac6dc246"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:30:51.709883 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:51.709413    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74942f9b-139d-4bee-9380-c251ac6dc246-model-cache" (OuterVolumeSpecName: "model-cache") pod "74942f9b-139d-4bee-9380-c251ac6dc246" (UID: "74942f9b-139d-4bee-9380-c251ac6dc246"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:30:51.709883 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:51.709784    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9da6fe34-d6b1-4b92-a067-e2677f71bca9-kube-api-access-qql8n" (OuterVolumeSpecName: "kube-api-access-qql8n") pod "9da6fe34-d6b1-4b92-a067-e2677f71bca9" (UID: "9da6fe34-d6b1-4b92-a067-e2677f71bca9"). InnerVolumeSpecName "kube-api-access-qql8n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 23:30:51.710880 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:51.710850    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74942f9b-139d-4bee-9380-c251ac6dc246-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "74942f9b-139d-4bee-9380-c251ac6dc246" (UID: "74942f9b-139d-4bee-9380-c251ac6dc246"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 23:30:51.712483 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:51.712444    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74942f9b-139d-4bee-9380-c251ac6dc246-dshm" (OuterVolumeSpecName: "dshm") pod "74942f9b-139d-4bee-9380-c251ac6dc246" (UID: "74942f9b-139d-4bee-9380-c251ac6dc246"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:30:51.713097 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:51.713068    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9da6fe34-d6b1-4b92-a067-e2677f71bca9-dshm" (OuterVolumeSpecName: "dshm") pod "9da6fe34-d6b1-4b92-a067-e2677f71bca9" (UID: "9da6fe34-d6b1-4b92-a067-e2677f71bca9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:30:51.713602 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:51.713571    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74942f9b-139d-4bee-9380-c251ac6dc246-kube-api-access-jflrm" (OuterVolumeSpecName: "kube-api-access-jflrm") pod "74942f9b-139d-4bee-9380-c251ac6dc246" (UID: "74942f9b-139d-4bee-9380-c251ac6dc246"). InnerVolumeSpecName "kube-api-access-jflrm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 23:30:51.713827 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:51.713797    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9da6fe34-d6b1-4b92-a067-e2677f71bca9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9da6fe34-d6b1-4b92-a067-e2677f71bca9" (UID: "9da6fe34-d6b1-4b92-a067-e2677f71bca9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 23:30:51.806285 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:51.806139    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d4d86af-7f04-4bcf-ad33-fb1fed658519-tmp-dir\") pod \"2d4d86af-7f04-4bcf-ad33-fb1fed658519\" (UID: \"2d4d86af-7f04-4bcf-ad33-fb1fed658519\") "
Sep 11 23:30:51.806285 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:51.806234    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d4d86af-7f04-4bcf-ad33-fb1fed658519-home\") pod \"2d4d86af-7f04-4bcf-ad33-fb1fed658519\" (UID: \"2d4d86af-7f04-4bcf-ad33-fb1fed658519\") "
Sep 11 23:30:51.806285 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:51.806262    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc328101-34e6-4746-9e4f-3e58116a6e70-dshm\") pod \"dc328101-34e6-4746-9e4f-3e58116a6e70\" (UID: \"dc328101-34e6-4746-9e4f-3e58116a6e70\") "
Sep 11 23:30:51.806583 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:51.806342    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d4d86af-7f04-4bcf-ad33-fb1fed658519-dshm\") pod \"2d4d86af-7f04-4bcf-ad33-fb1fed658519\" (UID: \"2d4d86af-7f04-4bcf-ad33-fb1fed658519\") "
Sep 11 23:30:51.806583 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:51.806378    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d4d86af-7f04-4bcf-ad33-fb1fed658519-tls-certs\") pod \"2d4d86af-7f04-4bcf-ad33-fb1fed658519\" (UID: \"2d4d86af-7f04-4bcf-ad33-fb1fed658519\") "
Sep 11 23:30:51.806583 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:51.806422    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d4d86af-7f04-4bcf-ad33-fb1fed658519-model-cache\") pod \"2d4d86af-7f04-4bcf-ad33-fb1fed658519\" (UID: \"2d4d86af-7f04-4bcf-ad33-fb1fed658519\") "
Sep 11 23:30:51.806583 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:51.806450    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc328101-34e6-4746-9e4f-3e58116a6e70-model-cache\") pod \"dc328101-34e6-4746-9e4f-3e58116a6e70\" (UID: \"dc328101-34e6-4746-9e4f-3e58116a6e70\") "
Sep 11 23:30:51.806583 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:51.806486    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c7498\" (UniqueName: \"kubernetes.io/projected/dc328101-34e6-4746-9e4f-3e58116a6e70-kube-api-access-c7498\") pod \"dc328101-34e6-4746-9e4f-3e58116a6e70\" (UID: \"dc328101-34e6-4746-9e4f-3e58116a6e70\") "
Sep 11 23:30:51.806583 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:51.806580    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc328101-34e6-4746-9e4f-3e58116a6e70-tmp-dir\") pod \"dc328101-34e6-4746-9e4f-3e58116a6e70\" (UID: \"dc328101-34e6-4746-9e4f-3e58116a6e70\") "
Sep 11 23:30:51.806902 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:51.806484    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d4d86af-7f04-4bcf-ad33-fb1fed658519-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2d4d86af-7f04-4bcf-ad33-fb1fed658519" (UID: "2d4d86af-7f04-4bcf-ad33-fb1fed658519"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:30:51.806902 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:51.806506    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d4d86af-7f04-4bcf-ad33-fb1fed658519-home" (OuterVolumeSpecName: "home") pod "2d4d86af-7f04-4bcf-ad33-fb1fed658519" (UID: "2d4d86af-7f04-4bcf-ad33-fb1fed658519"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:30:51.806902 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:51.806633    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc328101-34e6-4746-9e4f-3e58116a6e70-home\") pod \"dc328101-34e6-4746-9e4f-3e58116a6e70\" (UID: \"dc328101-34e6-4746-9e4f-3e58116a6e70\") "
Sep 11 23:30:51.806902 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:51.806682    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d4d86af-7f04-4bcf-ad33-fb1fed658519-model-cache" (OuterVolumeSpecName: "model-cache") pod "2d4d86af-7f04-4bcf-ad33-fb1fed658519" (UID: "2d4d86af-7f04-4bcf-ad33-fb1fed658519"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:30:51.806902 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:51.806712    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc328101-34e6-4746-9e4f-3e58116a6e70-tls-certs\") pod \"dc328101-34e6-4746-9e4f-3e58116a6e70\" (UID: \"dc328101-34e6-4746-9e4f-3e58116a6e70\") "
Sep 11 23:30:51.806902 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:51.806762    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wbxsk\" (UniqueName: \"kubernetes.io/projected/2d4d86af-7f04-4bcf-ad33-fb1fed658519-kube-api-access-wbxsk\") pod \"2d4d86af-7f04-4bcf-ad33-fb1fed658519\" (UID: \"2d4d86af-7f04-4bcf-ad33-fb1fed658519\") "
Sep 11 23:30:51.806902 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:51.806808    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc328101-34e6-4746-9e4f-3e58116a6e70-home" (OuterVolumeSpecName: "home") pod "dc328101-34e6-4746-9e4f-3e58116a6e70" (UID: "dc328101-34e6-4746-9e4f-3e58116a6e70"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:30:51.807469 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:51.806942    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc328101-34e6-4746-9e4f-3e58116a6e70-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dc328101-34e6-4746-9e4f-3e58116a6e70" (UID: "dc328101-34e6-4746-9e4f-3e58116a6e70"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:30:51.807585 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:51.807559    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc328101-34e6-4746-9e4f-3e58116a6e70-model-cache" (OuterVolumeSpecName: "model-cache") pod "dc328101-34e6-4746-9e4f-3e58116a6e70" (UID: "dc328101-34e6-4746-9e4f-3e58116a6e70"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:30:51.807655 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:51.807621    2546 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d4d86af-7f04-4bcf-ad33-fb1fed658519-model-cache\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:30:51.807711 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:51.807668    2546 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74942f9b-139d-4bee-9380-c251ac6dc246-tmp-dir\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:30:51.807711 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:51.807683    2546 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9da6fe34-d6b1-4b92-a067-e2677f71bca9-home\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:30:51.807711 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:51.807698    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qql8n\" (UniqueName: \"kubernetes.io/projected/9da6fe34-d6b1-4b92-a067-e2677f71bca9-kube-api-access-qql8n\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:30:51.807867 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:51.807734    2546 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9da6fe34-d6b1-4b92-a067-e2677f71bca9-dshm\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:30:51.807867 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:51.807750    2546 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc328101-34e6-4746-9e4f-3e58116a6e70-tmp-dir\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:30:51.807867 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:51.807764    2546 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74942f9b-139d-4bee-9380-c251ac6dc246-home\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:30:51.807867 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:51.807776    2546 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9da6fe34-d6b1-4b92-a067-e2677f71bca9-model-cache\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:30:51.807867 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:51.807789    2546 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9da6fe34-d6b1-4b92-a067-e2677f71bca9-tls-certs\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:30:51.807867 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:51.807824    2546 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc328101-34e6-4746-9e4f-3e58116a6e70-home\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:30:51.807867 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:51.807837    2546 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9da6fe34-d6b1-4b92-a067-e2677f71bca9-tmp-dir\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:30:51.807867 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:51.807849    2546 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74942f9b-139d-4bee-9380-c251ac6dc246-dshm\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:30:51.807867 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:51.807863    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jflrm\" (UniqueName: \"kubernetes.io/projected/74942f9b-139d-4bee-9380-c251ac6dc246-kube-api-access-jflrm\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:30:51.808353 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:51.807899    2546 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74942f9b-139d-4bee-9380-c251ac6dc246-model-cache\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:30:51.808353 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:51.807914    2546 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d4d86af-7f04-4bcf-ad33-fb1fed658519-tmp-dir\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:30:51.808353 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:51.807926    2546 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d4d86af-7f04-4bcf-ad33-fb1fed658519-home\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:30:51.808353 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:51.807939    2546 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74942f9b-139d-4bee-9380-c251ac6dc246-tls-certs\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:30:51.811276 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:51.811093    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc328101-34e6-4746-9e4f-3e58116a6e70-dshm" (OuterVolumeSpecName: "dshm") pod "dc328101-34e6-4746-9e4f-3e58116a6e70" (UID: "dc328101-34e6-4746-9e4f-3e58116a6e70"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:30:51.811276 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:51.811097    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc328101-34e6-4746-9e4f-3e58116a6e70-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dc328101-34e6-4746-9e4f-3e58116a6e70" (UID: "dc328101-34e6-4746-9e4f-3e58116a6e70"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 23:30:51.811276 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:51.811186    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d4d86af-7f04-4bcf-ad33-fb1fed658519-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2d4d86af-7f04-4bcf-ad33-fb1fed658519" (UID: "2d4d86af-7f04-4bcf-ad33-fb1fed658519"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 23:30:51.811276 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:51.811218    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d4d86af-7f04-4bcf-ad33-fb1fed658519-dshm" (OuterVolumeSpecName: "dshm") pod "2d4d86af-7f04-4bcf-ad33-fb1fed658519" (UID: "2d4d86af-7f04-4bcf-ad33-fb1fed658519"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:30:51.811603 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:51.811579    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d4d86af-7f04-4bcf-ad33-fb1fed658519-kube-api-access-wbxsk" (OuterVolumeSpecName: "kube-api-access-wbxsk") pod "2d4d86af-7f04-4bcf-ad33-fb1fed658519" (UID: "2d4d86af-7f04-4bcf-ad33-fb1fed658519"). InnerVolumeSpecName "kube-api-access-wbxsk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 23:30:51.813905 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:51.813866    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc328101-34e6-4746-9e4f-3e58116a6e70-kube-api-access-c7498" (OuterVolumeSpecName: "kube-api-access-c7498") pod "dc328101-34e6-4746-9e4f-3e58116a6e70" (UID: "dc328101-34e6-4746-9e4f-3e58116a6e70"). InnerVolumeSpecName "kube-api-access-c7498". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 23:30:51.909518 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:51.909471    2546 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc328101-34e6-4746-9e4f-3e58116a6e70-dshm\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:30:51.909518 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:51.909507    2546 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d4d86af-7f04-4bcf-ad33-fb1fed658519-dshm\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:30:51.909518 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:51.909516    2546 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d4d86af-7f04-4bcf-ad33-fb1fed658519-tls-certs\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:30:51.909772 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:51.909526    2546 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc328101-34e6-4746-9e4f-3e58116a6e70-model-cache\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:30:51.909772 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:51.909546    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7498\" (UniqueName: \"kubernetes.io/projected/dc328101-34e6-4746-9e4f-3e58116a6e70-kube-api-access-c7498\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:30:51.909772 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:51.909555    2546 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc328101-34e6-4746-9e4f-3e58116a6e70-tls-certs\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:30:51.909772 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:51.909564    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbxsk\" (UniqueName: \"kubernetes.io/projected/2d4d86af-7f04-4bcf-ad33-fb1fed658519-kube-api-access-wbxsk\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:30:52.081467 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:52.081360    2546 generic.go:356] "Generic (PLEG): container finished" podID="2d4d86af-7f04-4bcf-ad33-fb1fed658519" containerID="d87a308915173a6be103b44404ac88d92021424ebc3386d922e48d83f1985a23" exitCode=0
Sep 11 23:30:52.081467 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:52.081401    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-12fb6b/simulator-w0-12fb6b-kserve-9749df7b4-859hf"
Sep 11 23:30:52.081467 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:52.081455    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-12fb6b/simulator-w0-12fb6b-kserve-9749df7b4-859hf" event={"ID":"2d4d86af-7f04-4bcf-ad33-fb1fed658519","Type":"ContainerDied","Data":"d87a308915173a6be103b44404ac88d92021424ebc3386d922e48d83f1985a23"}
Sep 11 23:30:52.081737 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:52.081495    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-12fb6b/simulator-w0-12fb6b-kserve-9749df7b4-859hf" event={"ID":"2d4d86af-7f04-4bcf-ad33-fb1fed658519","Type":"ContainerDied","Data":"0c7da0ebdc99a47fe9dccd1eca947e860c6e740b24b22c24d62be12a64eae768"}
Sep 11 23:30:52.081737 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:52.081517    2546 scope.go:117] "RemoveContainer" containerID="d87a308915173a6be103b44404ac88d92021424ebc3386d922e48d83f1985a23"
Sep 11 23:30:52.083047 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:52.083028    2546 generic.go:356] "Generic (PLEG): container finished" podID="9da6fe34-d6b1-4b92-a067-e2677f71bca9" containerID="328ba10a98d3d824ec19b37471e0a837842a2a0f8bd38a173fad6257b723a171" exitCode=0
Sep 11 23:30:52.083197 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:52.083060    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-12fb6b/distinct-2-w0-12fb6b-kserve-56c4887795-bf8sz" event={"ID":"9da6fe34-d6b1-4b92-a067-e2677f71bca9","Type":"ContainerDied","Data":"328ba10a98d3d824ec19b37471e0a837842a2a0f8bd38a173fad6257b723a171"}
Sep 11 23:30:52.083197 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:52.083107    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-12fb6b/distinct-2-w0-12fb6b-kserve-56c4887795-bf8sz"
Sep 11 23:30:52.083197 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:52.083117    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-12fb6b/distinct-2-w0-12fb6b-kserve-56c4887795-bf8sz" event={"ID":"9da6fe34-d6b1-4b92-a067-e2677f71bca9","Type":"ContainerDied","Data":"3853204736b987171cba6e482e492fefedd3961087dfc5f7e136a964d516d994"}
Sep 11 23:30:52.084786 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:52.084767    2546 generic.go:356] "Generic (PLEG): container finished" podID="dc328101-34e6-4746-9e4f-3e58116a6e70" containerID="f8b435be85ec4a7dc300a3569204be232a116a36d8bb2fd5d5542684d39259fe" exitCode=0
Sep 11 23:30:52.084898 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:52.084819    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-12fb6b/embedding-w0-12fb6b-kserve-585bc8ccb5-q5647"
Sep 11 23:30:52.084961 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:52.084827    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-12fb6b/embedding-w0-12fb6b-kserve-585bc8ccb5-q5647" event={"ID":"dc328101-34e6-4746-9e4f-3e58116a6e70","Type":"ContainerDied","Data":"f8b435be85ec4a7dc300a3569204be232a116a36d8bb2fd5d5542684d39259fe"}
Sep 11 23:30:52.085031 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:52.084962    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-12fb6b/embedding-w0-12fb6b-kserve-585bc8ccb5-q5647" event={"ID":"dc328101-34e6-4746-9e4f-3e58116a6e70","Type":"ContainerDied","Data":"6d78bbc21f6189ee74c5f6f400ed07dafee98ac2047192dc10f2bdfdf9676243"}
Sep 11 23:30:52.088735 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:52.088706    2546 generic.go:356] "Generic (PLEG): container finished" podID="74942f9b-139d-4bee-9380-c251ac6dc246" containerID="f842f454d6c900755eaf2f8137c9d8810ad3a6e983d8a1c9e7171fbfc8c7f708" exitCode=0
Sep 11 23:30:52.088829 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:52.088758    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-12fb6b/premium-w0-12fb6b-kserve-557db4c777-5zbbn"
Sep 11 23:30:52.088871 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:52.088839    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-12fb6b/premium-w0-12fb6b-kserve-557db4c777-5zbbn" event={"ID":"74942f9b-139d-4bee-9380-c251ac6dc246","Type":"ContainerDied","Data":"f842f454d6c900755eaf2f8137c9d8810ad3a6e983d8a1c9e7171fbfc8c7f708"}
Sep 11 23:30:52.088871 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:52.088862    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-12fb6b/premium-w0-12fb6b-kserve-557db4c777-5zbbn" event={"ID":"74942f9b-139d-4bee-9380-c251ac6dc246","Type":"ContainerDied","Data":"a7843d3ae9110528901299eb03474f7309982c4fff978f1b60fa7375264de9f8"}
Sep 11 23:30:52.094203 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:52.094113    2546 scope.go:117] "RemoveContainer" containerID="d87a308915173a6be103b44404ac88d92021424ebc3386d922e48d83f1985a23"
Sep 11 23:30:52.094472 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:30:52.094453    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d87a308915173a6be103b44404ac88d92021424ebc3386d922e48d83f1985a23\": container with ID starting with d87a308915173a6be103b44404ac88d92021424ebc3386d922e48d83f1985a23 not found: ID does not exist" containerID="d87a308915173a6be103b44404ac88d92021424ebc3386d922e48d83f1985a23"
Sep 11 23:30:52.094562 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:52.094496    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d87a308915173a6be103b44404ac88d92021424ebc3386d922e48d83f1985a23"} err="failed to get container status \"d87a308915173a6be103b44404ac88d92021424ebc3386d922e48d83f1985a23\": rpc error: code = NotFound desc = could not find container \"d87a308915173a6be103b44404ac88d92021424ebc3386d922e48d83f1985a23\": container with ID starting with d87a308915173a6be103b44404ac88d92021424ebc3386d922e48d83f1985a23 not found: ID does not exist"
Sep 11 23:30:52.094562 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:52.094518    2546 scope.go:117] "RemoveContainer" containerID="328ba10a98d3d824ec19b37471e0a837842a2a0f8bd38a173fad6257b723a171"
Sep 11 23:30:52.104835 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:52.104817    2546 scope.go:117] "RemoveContainer" containerID="328ba10a98d3d824ec19b37471e0a837842a2a0f8bd38a173fad6257b723a171"
Sep 11 23:30:52.105130 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:30:52.105112    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"328ba10a98d3d824ec19b37471e0a837842a2a0f8bd38a173fad6257b723a171\": container with ID starting with 328ba10a98d3d824ec19b37471e0a837842a2a0f8bd38a173fad6257b723a171 not found: ID does not exist" containerID="328ba10a98d3d824ec19b37471e0a837842a2a0f8bd38a173fad6257b723a171"
Sep 11 23:30:52.105209 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:52.105136    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"328ba10a98d3d824ec19b37471e0a837842a2a0f8bd38a173fad6257b723a171"} err="failed to get container status \"328ba10a98d3d824ec19b37471e0a837842a2a0f8bd38a173fad6257b723a171\": rpc error: code = NotFound desc = could not find container \"328ba10a98d3d824ec19b37471e0a837842a2a0f8bd38a173fad6257b723a171\": container with ID starting with 328ba10a98d3d824ec19b37471e0a837842a2a0f8bd38a173fad6257b723a171 not found: ID does not exist"
Sep 11 23:30:52.105209 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:52.105167    2546 scope.go:117] "RemoveContainer" containerID="f8b435be85ec4a7dc300a3569204be232a116a36d8bb2fd5d5542684d39259fe"
Sep 11 23:30:52.113433 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:52.113408    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-12fb6b/simulator-w0-12fb6b-kserve-9749df7b4-859hf"]
Sep 11 23:30:52.115444 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:52.115399    2546 scope.go:117] "RemoveContainer" containerID="f8b435be85ec4a7dc300a3569204be232a116a36d8bb2fd5d5542684d39259fe"
Sep 11 23:30:52.116104 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:30:52.116078    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8b435be85ec4a7dc300a3569204be232a116a36d8bb2fd5d5542684d39259fe\": container with ID starting with f8b435be85ec4a7dc300a3569204be232a116a36d8bb2fd5d5542684d39259fe not found: ID does not exist" containerID="f8b435be85ec4a7dc300a3569204be232a116a36d8bb2fd5d5542684d39259fe"
Sep 11 23:30:52.116247 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:52.116111    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8b435be85ec4a7dc300a3569204be232a116a36d8bb2fd5d5542684d39259fe"} err="failed to get container status \"f8b435be85ec4a7dc300a3569204be232a116a36d8bb2fd5d5542684d39259fe\": rpc error: code = NotFound desc = could not find container \"f8b435be85ec4a7dc300a3569204be232a116a36d8bb2fd5d5542684d39259fe\": container with ID starting with f8b435be85ec4a7dc300a3569204be232a116a36d8bb2fd5d5542684d39259fe not found: ID does not exist"
Sep 11 23:30:52.116247 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:52.116131    2546 scope.go:117] "RemoveContainer" containerID="f842f454d6c900755eaf2f8137c9d8810ad3a6e983d8a1c9e7171fbfc8c7f708"
Sep 11 23:30:52.119305 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:52.119283    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-12fb6b/simulator-w0-12fb6b-kserve-9749df7b4-859hf"]
Sep 11 23:30:52.125592 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:52.125574    2546 scope.go:117] "RemoveContainer" containerID="f842f454d6c900755eaf2f8137c9d8810ad3a6e983d8a1c9e7171fbfc8c7f708"
Sep 11 23:30:52.125946 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:30:52.125925    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f842f454d6c900755eaf2f8137c9d8810ad3a6e983d8a1c9e7171fbfc8c7f708\": container with ID starting with f842f454d6c900755eaf2f8137c9d8810ad3a6e983d8a1c9e7171fbfc8c7f708 not found: ID does not exist" containerID="f842f454d6c900755eaf2f8137c9d8810ad3a6e983d8a1c9e7171fbfc8c7f708"
Sep 11 23:30:52.126004 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:52.125953    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f842f454d6c900755eaf2f8137c9d8810ad3a6e983d8a1c9e7171fbfc8c7f708"} err="failed to get container status \"f842f454d6c900755eaf2f8137c9d8810ad3a6e983d8a1c9e7171fbfc8c7f708\": rpc error: code = NotFound desc = could not find container \"f842f454d6c900755eaf2f8137c9d8810ad3a6e983d8a1c9e7171fbfc8c7f708\": container with ID starting with f842f454d6c900755eaf2f8137c9d8810ad3a6e983d8a1c9e7171fbfc8c7f708 not found: ID does not exist"
Sep 11 23:30:52.134942 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:52.134919    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-12fb6b/premium-w0-12fb6b-kserve-557db4c777-5zbbn"]
Sep 11 23:30:52.137980 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:52.137956    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-12fb6b/premium-w0-12fb6b-kserve-557db4c777-5zbbn"]
Sep 11 23:30:52.152859 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:52.152835    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-12fb6b/distinct-2-w0-12fb6b-kserve-56c4887795-bf8sz"]
Sep 11 23:30:52.155719 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:52.155701    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-12fb6b/distinct-2-w0-12fb6b-kserve-56c4887795-bf8sz"]
Sep 11 23:30:52.166356 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:52.166337    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-12fb6b/embedding-w0-12fb6b-kserve-585bc8ccb5-q5647"]
Sep 11 23:30:52.172562 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:52.172542    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-12fb6b/embedding-w0-12fb6b-kserve-585bc8ccb5-q5647"]
Sep 11 23:30:52.495751 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:52.495720    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d4d86af-7f04-4bcf-ad33-fb1fed658519" path="/var/lib/kubelet/pods/2d4d86af-7f04-4bcf-ad33-fb1fed658519/volumes"
Sep 11 23:30:52.496146 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:52.496132    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74942f9b-139d-4bee-9380-c251ac6dc246" path="/var/lib/kubelet/pods/74942f9b-139d-4bee-9380-c251ac6dc246/volumes"
Sep 11 23:30:52.496518 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:52.496505    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9da6fe34-d6b1-4b92-a067-e2677f71bca9" path="/var/lib/kubelet/pods/9da6fe34-d6b1-4b92-a067-e2677f71bca9/volumes"
Sep 11 23:30:52.497179 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:52.497147    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc328101-34e6-4746-9e4f-3e58116a6e70" path="/var/lib/kubelet/pods/dc328101-34e6-4746-9e4f-3e58116a6e70/volumes"
Sep 11 23:30:52.570928 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:52.570898    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-12fb6b-29819490-z7ztp"]
Sep 11 23:30:52.576436 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:52.576410    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-12fb6b-29819490-z7ztp"]
Sep 11 23:30:52.794774 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:52.794695    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-12fb6b-rjfjk"]
Sep 11 23:30:52.802108 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:52.802079    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-12fb6b-rjfjk"]
Sep 11 23:30:54.501707 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:54.501673    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29b56839-d428-44de-9434-96fc011d873b" path="/var/lib/kubelet/pods/29b56839-d428-44de-9434-96fc011d873b/volumes"
Sep 11 23:30:54.502082 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:54.501970    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="667435a0-dc03-4efe-8edd-d743b5a2efb0" path="/var/lib/kubelet/pods/667435a0-dc03-4efe-8edd-d743b5a2efb0/volumes"
Sep 11 23:30:57.044267 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:30:57.044238    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d1b012-59676779d6-48drk"
Sep 11 23:31:00.079463 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:00.079431    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp"]
Sep 11 23:31:00.079869 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:00.079683    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp" podUID="48592704-31d9-4e6f-bad1-ac862a6dfa72" containerName="istio-proxy" containerID="cri-o://554d358cebbba92d7feceaf69fe2551433117579b9ebcb71c2eafd353fb28af4" gracePeriod=30
Sep 11 23:31:01.158581 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:01.158542    2546 generic.go:356] "Generic (PLEG): container finished" podID="a5a83b12-239d-4342-9f10-dc236f6fbb2b" containerID="489dfc2189f4ab2947303b73fff8bbe6dd1bdc12afe712592429797c4478aa27" exitCode=137
Sep 11 23:31:01.158992 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:01.158620    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c56f6" event={"ID":"a5a83b12-239d-4342-9f10-dc236f6fbb2b","Type":"ContainerDied","Data":"489dfc2189f4ab2947303b73fff8bbe6dd1bdc12afe712592429797c4478aa27"}
Sep 11 23:31:01.159041 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:01.159011    2546 scope.go:117] "RemoveContainer" containerID="489dfc2189f4ab2947303b73fff8bbe6dd1bdc12afe712592429797c4478aa27"
Sep 11 23:31:02.172031 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:02.171997    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c56f6" event={"ID":"a5a83b12-239d-4342-9f10-dc236f6fbb2b","Type":"ContainerStarted","Data":"30c1843721e461c23442e28b54983c16e083b8003fb63b3dc37bfdf1388c0229"}
Sep 11 23:31:02.172496 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:02.172451    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c56f6"
Sep 11 23:31:02.506964 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:02.506889    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-451902-gdv8f"]
Sep 11 23:31:02.507383 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:02.507369    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74942f9b-139d-4bee-9380-c251ac6dc246" containerName="main"
Sep 11 23:31:02.507433 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:02.507386    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="74942f9b-139d-4bee-9380-c251ac6dc246" containerName="main"
Sep 11 23:31:02.507433 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:02.507398    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d4d86af-7f04-4bcf-ad33-fb1fed658519" containerName="main"
Sep 11 23:31:02.507433 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:02.507404    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d4d86af-7f04-4bcf-ad33-fb1fed658519" containerName="main"
Sep 11 23:31:02.507433 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:02.507420    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc328101-34e6-4746-9e4f-3e58116a6e70" containerName="main"
Sep 11 23:31:02.507433 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:02.507426    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc328101-34e6-4746-9e4f-3e58116a6e70" containerName="main"
Sep 11 23:31:02.507586 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:02.507449    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9da6fe34-d6b1-4b92-a067-e2677f71bca9" containerName="main"
Sep 11 23:31:02.507586 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:02.507457    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="9da6fe34-d6b1-4b92-a067-e2677f71bca9" containerName="main"
Sep 11 23:31:02.507586 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:02.507530    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc328101-34e6-4746-9e4f-3e58116a6e70" containerName="main"
Sep 11 23:31:02.507586 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:02.507541    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="74942f9b-139d-4bee-9380-c251ac6dc246" containerName="main"
Sep 11 23:31:02.507586 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:02.507547    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="9da6fe34-d6b1-4b92-a067-e2677f71bca9" containerName="main"
Sep 11 23:31:02.507586 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:02.507552    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d4d86af-7f04-4bcf-ad33-fb1fed658519" containerName="main"
Sep 11 23:31:02.510848 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:02.510833    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-451902-gdv8f"
Sep 11 23:31:02.514016 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:02.513993    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vrx7m\""
Sep 11 23:31:02.514255 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:02.514233    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 23:31:02.519102 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:02.519044    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-451902-gdv8f"]
Sep 11 23:31:02.625440 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:02.625405    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa1be752-ebc7-4fe2-ad40-5d0734a20d69-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-451902-gdv8f\" (UID: \"fa1be752-ebc7-4fe2-ad40-5d0734a20d69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-451902-gdv8f"
Sep 11 23:31:02.726256 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:02.726220    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa1be752-ebc7-4fe2-ad40-5d0734a20d69-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-451902-gdv8f\" (UID: \"fa1be752-ebc7-4fe2-ad40-5d0734a20d69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-451902-gdv8f"
Sep 11 23:31:02.726810 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:02.726789    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa1be752-ebc7-4fe2-ad40-5d0734a20d69-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-451902-gdv8f\" (UID: \"fa1be752-ebc7-4fe2-ad40-5d0734a20d69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-451902-gdv8f"
Sep 11 23:31:02.823399 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:02.823328    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-451902-gdv8f"
Sep 11 23:31:03.112756 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:03.112726    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-451902-gdv8f"]
Sep 11 23:31:03.112928 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:31:03.112904    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa1be752_ebc7_4fe2_ad40_5d0734a20d69.slice/crio-df79fc98fb7fc440a2807f876286d579fa8b82e20561a1a6558af313145030ac WatchSource:0}: Error finding container df79fc98fb7fc440a2807f876286d579fa8b82e20561a1a6558af313145030ac: Status 404 returned error can't find the container with id df79fc98fb7fc440a2807f876286d579fa8b82e20561a1a6558af313145030ac
Sep 11 23:31:03.181185 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:03.181141    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-451902-gdv8f" event={"ID":"fa1be752-ebc7-4fe2-ad40-5d0734a20d69","Type":"ContainerStarted","Data":"df79fc98fb7fc440a2807f876286d579fa8b82e20561a1a6558af313145030ac"}
Sep 11 23:31:04.190373 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:04.190339    2546 generic.go:356] "Generic (PLEG): container finished" podID="fa1be752-ebc7-4fe2-ad40-5d0734a20d69" containerID="f55206e553cbef925be9237271e7c65ff353cb8cbaa70ebcf65db14b26db9f57" exitCode=0
Sep 11 23:31:04.190772 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:04.190385    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-451902-gdv8f" event={"ID":"fa1be752-ebc7-4fe2-ad40-5d0734a20d69","Type":"ContainerDied","Data":"f55206e553cbef925be9237271e7c65ff353cb8cbaa70ebcf65db14b26db9f57"}
Sep 11 23:31:05.200667 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:05.200635    2546 generic.go:356] "Generic (PLEG): container finished" podID="48592704-31d9-4e6f-bad1-ac862a6dfa72" containerID="554d358cebbba92d7feceaf69fe2551433117579b9ebcb71c2eafd353fb28af4" exitCode=0
Sep 11 23:31:05.201026 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:05.200718    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp" event={"ID":"48592704-31d9-4e6f-bad1-ac862a6dfa72","Type":"ContainerDied","Data":"554d358cebbba92d7feceaf69fe2551433117579b9ebcb71c2eafd353fb28af4"}
Sep 11 23:31:05.201026 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:05.200782    2546 scope.go:117] "RemoveContainer" containerID="037fb0ae3d3ce58a7da3acd0f1d575db3e324cf70b9536808ea6eeff2f2e363e"
Sep 11 23:31:05.383110 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:05.383080    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp"
Sep 11 23:31:05.385315 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:05.385293    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-451902-gdv8f"
Sep 11 23:31:05.556394 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:05.556344    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48592704-31d9-4e6f-bad1-ac862a6dfa72-istiod-ca-cert\") pod \"48592704-31d9-4e6f-bad1-ac862a6dfa72\" (UID: \"48592704-31d9-4e6f-bad1-ac862a6dfa72\") "
Sep 11 23:31:05.556570 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:05.556431    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48592704-31d9-4e6f-bad1-ac862a6dfa72-credential-socket\") pod \"48592704-31d9-4e6f-bad1-ac862a6dfa72\" (UID: \"48592704-31d9-4e6f-bad1-ac862a6dfa72\") "
Sep 11 23:31:05.556570 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:05.556487    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t6hq2\" (UniqueName: \"kubernetes.io/projected/48592704-31d9-4e6f-bad1-ac862a6dfa72-kube-api-access-t6hq2\") pod \"48592704-31d9-4e6f-bad1-ac862a6dfa72\" (UID: \"48592704-31d9-4e6f-bad1-ac862a6dfa72\") "
Sep 11 23:31:05.556570 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:05.556539    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48592704-31d9-4e6f-bad1-ac862a6dfa72-istio-envoy\") pod \"48592704-31d9-4e6f-bad1-ac862a6dfa72\" (UID: \"48592704-31d9-4e6f-bad1-ac862a6dfa72\") "
Sep 11 23:31:05.556730 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:05.556601    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48592704-31d9-4e6f-bad1-ac862a6dfa72-istio-podinfo\") pod \"48592704-31d9-4e6f-bad1-ac862a6dfa72\" (UID: \"48592704-31d9-4e6f-bad1-ac862a6dfa72\") "
Sep 11 23:31:05.556730 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:05.556622    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48592704-31d9-4e6f-bad1-ac862a6dfa72-istio-token\") pod \"48592704-31d9-4e6f-bad1-ac862a6dfa72\" (UID: \"48592704-31d9-4e6f-bad1-ac862a6dfa72\") "
Sep 11 23:31:05.556730 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:05.556646    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48592704-31d9-4e6f-bad1-ac862a6dfa72-workload-certs\") pod \"48592704-31d9-4e6f-bad1-ac862a6dfa72\" (UID: \"48592704-31d9-4e6f-bad1-ac862a6dfa72\") "
Sep 11 23:31:05.556868 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:05.556753    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48592704-31d9-4e6f-bad1-ac862a6dfa72-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "48592704-31d9-4e6f-bad1-ac862a6dfa72" (UID: "48592704-31d9-4e6f-bad1-ac862a6dfa72"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:31:05.556868 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:05.556803    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48592704-31d9-4e6f-bad1-ac862a6dfa72-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "48592704-31d9-4e6f-bad1-ac862a6dfa72" (UID: "48592704-31d9-4e6f-bad1-ac862a6dfa72"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 23:31:05.556988 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:05.556891    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48592704-31d9-4e6f-bad1-ac862a6dfa72-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "48592704-31d9-4e6f-bad1-ac862a6dfa72" (UID: "48592704-31d9-4e6f-bad1-ac862a6dfa72"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:31:05.556988 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:05.556814    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48592704-31d9-4e6f-bad1-ac862a6dfa72-workload-socket\") pod \"48592704-31d9-4e6f-bad1-ac862a6dfa72\" (UID: \"48592704-31d9-4e6f-bad1-ac862a6dfa72\") "
Sep 11 23:31:05.557110 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:05.556991    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa1be752-ebc7-4fe2-ad40-5d0734a20d69-maas-api-service-ca\") pod \"fa1be752-ebc7-4fe2-ad40-5d0734a20d69\" (UID: \"fa1be752-ebc7-4fe2-ad40-5d0734a20d69\") "
Sep 11 23:31:05.557110 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:05.557026    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48592704-31d9-4e6f-bad1-ac862a6dfa72-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "48592704-31d9-4e6f-bad1-ac862a6dfa72" (UID: "48592704-31d9-4e6f-bad1-ac862a6dfa72"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:31:05.557110 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:05.557065    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48592704-31d9-4e6f-bad1-ac862a6dfa72-istio-data\") pod \"48592704-31d9-4e6f-bad1-ac862a6dfa72\" (UID: \"48592704-31d9-4e6f-bad1-ac862a6dfa72\") "
Sep 11 23:31:05.557361 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:05.557339    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48592704-31d9-4e6f-bad1-ac862a6dfa72-istio-data" (OuterVolumeSpecName: "istio-data") pod "48592704-31d9-4e6f-bad1-ac862a6dfa72" (UID: "48592704-31d9-4e6f-bad1-ac862a6dfa72"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:31:05.557434 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:05.557387    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa1be752-ebc7-4fe2-ad40-5d0734a20d69-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fa1be752-ebc7-4fe2-ad40-5d0734a20d69" (UID: "fa1be752-ebc7-4fe2-ad40-5d0734a20d69"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 23:31:05.557787 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:05.557722    2546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48592704-31d9-4e6f-bad1-ac862a6dfa72-credential-socket\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:31:05.557787 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:05.557748    2546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48592704-31d9-4e6f-bad1-ac862a6dfa72-workload-certs\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:31:05.557787 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:05.557766    2546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48592704-31d9-4e6f-bad1-ac862a6dfa72-workload-socket\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:31:05.557787 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:05.557781    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa1be752-ebc7-4fe2-ad40-5d0734a20d69-maas-api-service-ca\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:31:05.558050 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:05.557795    2546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48592704-31d9-4e6f-bad1-ac862a6dfa72-istio-data\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:31:05.558050 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:05.557809    2546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48592704-31d9-4e6f-bad1-ac862a6dfa72-istiod-ca-cert\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:31:05.559124 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:05.559091    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48592704-31d9-4e6f-bad1-ac862a6dfa72-kube-api-access-t6hq2" (OuterVolumeSpecName: "kube-api-access-t6hq2") pod "48592704-31d9-4e6f-bad1-ac862a6dfa72" (UID: "48592704-31d9-4e6f-bad1-ac862a6dfa72"). InnerVolumeSpecName "kube-api-access-t6hq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 23:31:05.559246 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:05.559124    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48592704-31d9-4e6f-bad1-ac862a6dfa72-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "48592704-31d9-4e6f-bad1-ac862a6dfa72" (UID: "48592704-31d9-4e6f-bad1-ac862a6dfa72"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:31:05.559246 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:05.559138    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48592704-31d9-4e6f-bad1-ac862a6dfa72-istio-token" (OuterVolumeSpecName: "istio-token") pod "48592704-31d9-4e6f-bad1-ac862a6dfa72" (UID: "48592704-31d9-4e6f-bad1-ac862a6dfa72"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 23:31:05.559395 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:05.559377    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/48592704-31d9-4e6f-bad1-ac862a6dfa72-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "48592704-31d9-4e6f-bad1-ac862a6dfa72" (UID: "48592704-31d9-4e6f-bad1-ac862a6dfa72"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 23:31:05.658622 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:05.658583    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6hq2\" (UniqueName: \"kubernetes.io/projected/48592704-31d9-4e6f-bad1-ac862a6dfa72-kube-api-access-t6hq2\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:31:05.658622 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:05.658611    2546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48592704-31d9-4e6f-bad1-ac862a6dfa72-istio-envoy\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:31:05.658622 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:05.658621    2546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48592704-31d9-4e6f-bad1-ac862a6dfa72-istio-podinfo\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:31:05.658899 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:05.658632    2546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48592704-31d9-4e6f-bad1-ac862a6dfa72-istio-token\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:31:06.208992 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:06.208961    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp"
Sep 11 23:31:06.208992 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:06.208975    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp" event={"ID":"48592704-31d9-4e6f-bad1-ac862a6dfa72","Type":"ContainerDied","Data":"69e2b66a45ce76d3c745c018d4efec12145cb5e4a3a45246511f7736cbdb684a"}
Sep 11 23:31:06.209551 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:06.209041    2546 scope.go:117] "RemoveContainer" containerID="554d358cebbba92d7feceaf69fe2551433117579b9ebcb71c2eafd353fb28af4"
Sep 11 23:31:06.210687 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:06.210658    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-451902-gdv8f"
Sep 11 23:31:06.210770 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:06.210657    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-451902-gdv8f" event={"ID":"fa1be752-ebc7-4fe2-ad40-5d0734a20d69","Type":"ContainerDied","Data":"df79fc98fb7fc440a2807f876286d579fa8b82e20561a1a6558af313145030ac"}
Sep 11 23:31:06.210820 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:06.210804    2546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df79fc98fb7fc440a2807f876286d579fa8b82e20561a1a6558af313145030ac"
Sep 11 23:31:06.249941 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:06.249907    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp"]
Sep 11 23:31:06.272575 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:06.272545    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-12fb6b-openshift-default-6b9b8c9494-xrbgp"]
Sep 11 23:31:06.496225 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:06.496112    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48592704-31d9-4e6f-bad1-ac862a6dfa72" path="/var/lib/kubelet/pods/48592704-31d9-4e6f-bad1-ac862a6dfa72/volumes"
Sep 11 23:31:13.184020 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:13.183935    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c56f6"
Sep 11 23:31:16.644818 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:16.644780    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7244_6b0a8af3-932c-4091-b7a0-180fe1751a3a/ovn-acl-logging/0.log"
Sep 11 23:31:16.646584 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:16.646562    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7244_6b0a8af3-932c-4091-b7a0-180fe1751a3a/ovn-acl-logging/0.log"
Sep 11 23:31:16.823854 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:16.823813    2546 scope.go:117] "RemoveContainer" containerID="87d03b6d0a71f7544c3365b189b5d64b03dfa36bacb025da177a6043b3e52600"
Sep 11 23:31:18.511218 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:18.511182    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-451902-d8d4b8fdb-pwgs4"]
Sep 11 23:31:18.511719 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:18.511446    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-451902-d8d4b8fdb-pwgs4" podUID="728bdfa7-251f-4bd8-a725-6b19a7aa7373" containerName="payload-pre-processing" containerID="cri-o://9c039e1804c2503eb01146e349a150e1a5af76ba24eb60d7d2a01ce61ec8b966" gracePeriod=30
Sep 11 23:31:18.735920 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:18.733884    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-451902-gdv8f"]
Sep 11 23:31:18.747583 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:18.747482    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-451902-gdv8f"]
Sep 11 23:31:18.827476 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:18.827447    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-451902-d8d4b8fdb-pwgs4"
Sep 11 23:31:18.911267 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:18.911230    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8rx7w\" (UniqueName: \"kubernetes.io/projected/728bdfa7-251f-4bd8-a725-6b19a7aa7373-kube-api-access-8rx7w\") pod \"728bdfa7-251f-4bd8-a725-6b19a7aa7373\" (UID: \"728bdfa7-251f-4bd8-a725-6b19a7aa7373\") "
Sep 11 23:31:18.911267 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:18.911274    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/728bdfa7-251f-4bd8-a725-6b19a7aa7373-plugins-config-volume\") pod \"728bdfa7-251f-4bd8-a725-6b19a7aa7373\" (UID: \"728bdfa7-251f-4bd8-a725-6b19a7aa7373\") "
Sep 11 23:31:18.911723 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:18.911691    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/728bdfa7-251f-4bd8-a725-6b19a7aa7373-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "728bdfa7-251f-4bd8-a725-6b19a7aa7373" (UID: "728bdfa7-251f-4bd8-a725-6b19a7aa7373"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 23:31:18.913999 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:18.913956    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/728bdfa7-251f-4bd8-a725-6b19a7aa7373-kube-api-access-8rx7w" (OuterVolumeSpecName: "kube-api-access-8rx7w") pod "728bdfa7-251f-4bd8-a725-6b19a7aa7373" (UID: "728bdfa7-251f-4bd8-a725-6b19a7aa7373"). InnerVolumeSpecName "kube-api-access-8rx7w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 23:31:19.012890 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:19.012853    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rx7w\" (UniqueName: \"kubernetes.io/projected/728bdfa7-251f-4bd8-a725-6b19a7aa7373-kube-api-access-8rx7w\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:31:19.012890 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:19.012886    2546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/728bdfa7-251f-4bd8-a725-6b19a7aa7373-plugins-config-volume\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:31:19.312641 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:19.312606    2546 generic.go:356] "Generic (PLEG): container finished" podID="728bdfa7-251f-4bd8-a725-6b19a7aa7373" containerID="9c039e1804c2503eb01146e349a150e1a5af76ba24eb60d7d2a01ce61ec8b966" exitCode=0
Sep 11 23:31:19.312822 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:19.312653    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-451902-d8d4b8fdb-pwgs4"
Sep 11 23:31:19.312822 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:19.312652    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-451902-d8d4b8fdb-pwgs4" event={"ID":"728bdfa7-251f-4bd8-a725-6b19a7aa7373","Type":"ContainerDied","Data":"9c039e1804c2503eb01146e349a150e1a5af76ba24eb60d7d2a01ce61ec8b966"}
Sep 11 23:31:19.312822 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:19.312793    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-451902-d8d4b8fdb-pwgs4" event={"ID":"728bdfa7-251f-4bd8-a725-6b19a7aa7373","Type":"ContainerDied","Data":"732700dd8bff3df59678907ce968274acf1b1284f4683109633d73640e838197"}
Sep 11 23:31:19.312822 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:19.312821    2546 scope.go:117] "RemoveContainer" containerID="9c039e1804c2503eb01146e349a150e1a5af76ba24eb60d7d2a01ce61ec8b966"
Sep 11 23:31:19.323988 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:19.323960    2546 scope.go:117] "RemoveContainer" containerID="9c039e1804c2503eb01146e349a150e1a5af76ba24eb60d7d2a01ce61ec8b966"
Sep 11 23:31:19.324372 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:31:19.324352    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c039e1804c2503eb01146e349a150e1a5af76ba24eb60d7d2a01ce61ec8b966\": container with ID starting with 9c039e1804c2503eb01146e349a150e1a5af76ba24eb60d7d2a01ce61ec8b966 not found: ID does not exist" containerID="9c039e1804c2503eb01146e349a150e1a5af76ba24eb60d7d2a01ce61ec8b966"
Sep 11 23:31:19.324448 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:19.324383    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c039e1804c2503eb01146e349a150e1a5af76ba24eb60d7d2a01ce61ec8b966"} err="failed to get container status \"9c039e1804c2503eb01146e349a150e1a5af76ba24eb60d7d2a01ce61ec8b966\": rpc error: code = NotFound desc = could not find container \"9c039e1804c2503eb01146e349a150e1a5af76ba24eb60d7d2a01ce61ec8b966\": container with ID starting with 9c039e1804c2503eb01146e349a150e1a5af76ba24eb60d7d2a01ce61ec8b966 not found: ID does not exist"
Sep 11 23:31:19.375907 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:19.375866    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-451902-d8d4b8fdb-pwgs4"]
Sep 11 23:31:19.392658 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:19.392625    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-451902-d8d4b8fdb-pwgs4"]
Sep 11 23:31:20.495173 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:20.495134    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="728bdfa7-251f-4bd8-a725-6b19a7aa7373" path="/var/lib/kubelet/pods/728bdfa7-251f-4bd8-a725-6b19a7aa7373/volumes"
Sep 11 23:31:20.495625 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:20.495503    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa1be752-ebc7-4fe2-ad40-5d0734a20d69" path="/var/lib/kubelet/pods/fa1be752-ebc7-4fe2-ad40-5d0734a20d69/volumes"
Sep 11 23:31:25.809191 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:25.809134    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw"]
Sep 11 23:31:25.809605 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:25.809370    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw" podUID="6e0419a9-c88a-4f69-bd07-751d5406fd0e" containerName="istio-proxy" containerID="cri-o://83c2605234559cac0dd49cd5f60cab1a058eeca48d4e5d97141a988ac00adbdc" gracePeriod=30
Sep 11 23:31:26.155392 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:26.155359    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d1b012-wr45p"]
Sep 11 23:31:26.155887 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:26.155868    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa1be752-ebc7-4fe2-ad40-5d0734a20d69" containerName="revoke-keys"
Sep 11 23:31:26.155980 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:26.155891    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa1be752-ebc7-4fe2-ad40-5d0734a20d69" containerName="revoke-keys"
Sep 11 23:31:26.155980 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:26.155910    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48592704-31d9-4e6f-bad1-ac862a6dfa72" containerName="istio-proxy"
Sep 11 23:31:26.155980 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:26.155918    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="48592704-31d9-4e6f-bad1-ac862a6dfa72" containerName="istio-proxy"
Sep 11 23:31:26.155980 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:26.155961    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="728bdfa7-251f-4bd8-a725-6b19a7aa7373" containerName="payload-pre-processing"
Sep 11 23:31:26.155980 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:26.155970    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="728bdfa7-251f-4bd8-a725-6b19a7aa7373" containerName="payload-pre-processing"
Sep 11 23:31:26.156265 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:26.156092    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa1be752-ebc7-4fe2-ad40-5d0734a20d69" containerName="revoke-keys"
Sep 11 23:31:26.156265 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:26.156106    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="48592704-31d9-4e6f-bad1-ac862a6dfa72" containerName="istio-proxy"
Sep 11 23:31:26.156265 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:26.156116    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="728bdfa7-251f-4bd8-a725-6b19a7aa7373" containerName="payload-pre-processing"
Sep 11 23:31:26.170928 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:26.170892    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d1b012-wr45p"
Sep 11 23:31:26.175882 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:26.175780    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vrx7m\""
Sep 11 23:31:26.176908 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:26.176835    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 23:31:26.179283 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:26.179253    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d1b012-wr45p"]
Sep 11 23:31:26.290912 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:26.290871    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5503de09-30bb-4b85-b1af-e22f2be25072-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d1b012-wr45p\" (UID: \"5503de09-30bb-4b85-b1af-e22f2be25072\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d1b012-wr45p"
Sep 11 23:31:26.391850 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:26.391803    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5503de09-30bb-4b85-b1af-e22f2be25072-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d1b012-wr45p\" (UID: \"5503de09-30bb-4b85-b1af-e22f2be25072\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d1b012-wr45p"
Sep 11 23:31:26.392682 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:26.392653    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5503de09-30bb-4b85-b1af-e22f2be25072-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d1b012-wr45p\" (UID: \"5503de09-30bb-4b85-b1af-e22f2be25072\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d1b012-wr45p"
Sep 11 23:31:26.488266 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:26.488137    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d1b012-wr45p"
Sep 11 23:31:26.769042 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:26.769008    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d1b012-wr45p"]
Sep 11 23:31:26.770362 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:31:26.770330    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5503de09_30bb_4b85_b1af_e22f2be25072.slice/crio-fb9d4a9b09520a9cbf405020720cce5430f284f26339a4d643202163f38d75f6 WatchSource:0}: Error finding container fb9d4a9b09520a9cbf405020720cce5430f284f26339a4d643202163f38d75f6: Status 404 returned error can't find the container with id fb9d4a9b09520a9cbf405020720cce5430f284f26339a4d643202163f38d75f6
Sep 11 23:31:27.371553 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:27.371463    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d1b012-wr45p" event={"ID":"5503de09-30bb-4b85-b1af-e22f2be25072","Type":"ContainerStarted","Data":"2bb70bee19c1f18b364a2d0064b66381a7c017d1932c852bb41c11e000bd6b09"}
Sep 11 23:31:27.371553 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:27.371520    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d1b012-wr45p" event={"ID":"5503de09-30bb-4b85-b1af-e22f2be25072","Type":"ContainerStarted","Data":"fb9d4a9b09520a9cbf405020720cce5430f284f26339a4d643202163f38d75f6"}
Sep 11 23:31:27.393298 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:27.393225    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d1b012-wr45p" podStartSLOduration=1.393206375 podStartE2EDuration="1.393206375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 23:31:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:31:27.390069648 +0000 UTC m=+1811.423198865" watchObservedRunningTime="2026-09-11 23:31:27.393206375 +0000 UTC m=+1811.426335591"
Sep 11 23:31:28.165695 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:28.165647    2546 patch_prober.go:28] interesting pod/e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 11 23:31:28.165871 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:28.165728    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw" podUID="6e0419a9-c88a-4f69-bd07-751d5406fd0e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 23:31:28.381214 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:28.381173    2546 generic.go:356] "Generic (PLEG): container finished" podID="5503de09-30bb-4b85-b1af-e22f2be25072" containerID="2bb70bee19c1f18b364a2d0064b66381a7c017d1932c852bb41c11e000bd6b09" exitCode=0
Sep 11 23:31:28.381624 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:28.381217    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d1b012-wr45p" event={"ID":"5503de09-30bb-4b85-b1af-e22f2be25072","Type":"ContainerDied","Data":"2bb70bee19c1f18b364a2d0064b66381a7c017d1932c852bb41c11e000bd6b09"}
Sep 11 23:31:29.544993 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:29.544957    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d1b012-wr45p"
Sep 11 23:31:29.624292 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:29.624253    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5503de09-30bb-4b85-b1af-e22f2be25072-maas-api-service-ca\") pod \"5503de09-30bb-4b85-b1af-e22f2be25072\" (UID: \"5503de09-30bb-4b85-b1af-e22f2be25072\") "
Sep 11 23:31:29.624637 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:29.624615    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5503de09-30bb-4b85-b1af-e22f2be25072-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5503de09-30bb-4b85-b1af-e22f2be25072" (UID: "5503de09-30bb-4b85-b1af-e22f2be25072"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 23:31:29.725511 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:29.725473    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5503de09-30bb-4b85-b1af-e22f2be25072-maas-api-service-ca\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:31:30.091780 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:30.091689    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"]
Sep 11 23:31:30.396652 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:30.396564    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d1b012-wr45p" event={"ID":"5503de09-30bb-4b85-b1af-e22f2be25072","Type":"ContainerDied","Data":"fb9d4a9b09520a9cbf405020720cce5430f284f26339a4d643202163f38d75f6"}
Sep 11 23:31:30.396652 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:30.396613    2546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb9d4a9b09520a9cbf405020720cce5430f284f26339a4d643202163f38d75f6"
Sep 11 23:31:30.396652 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:30.396580    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d1b012-wr45p"
Sep 11 23:31:31.076599 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:31.076519    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw"
Sep 11 23:31:31.243591 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:31.243550    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e0419a9-c88a-4f69-bd07-751d5406fd0e-credential-socket\") pod \"6e0419a9-c88a-4f69-bd07-751d5406fd0e\" (UID: \"6e0419a9-c88a-4f69-bd07-751d5406fd0e\") "
Sep 11 23:31:31.243801 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:31.243622    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e0419a9-c88a-4f69-bd07-751d5406fd0e-istio-envoy\") pod \"6e0419a9-c88a-4f69-bd07-751d5406fd0e\" (UID: \"6e0419a9-c88a-4f69-bd07-751d5406fd0e\") "
Sep 11 23:31:31.243801 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:31.243646    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e0419a9-c88a-4f69-bd07-751d5406fd0e-istiod-ca-cert\") pod \"6e0419a9-c88a-4f69-bd07-751d5406fd0e\" (UID: \"6e0419a9-c88a-4f69-bd07-751d5406fd0e\") "
Sep 11 23:31:31.243801 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:31.243700    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e0419a9-c88a-4f69-bd07-751d5406fd0e-istio-token\") pod \"6e0419a9-c88a-4f69-bd07-751d5406fd0e\" (UID: \"6e0419a9-c88a-4f69-bd07-751d5406fd0e\") "
Sep 11 23:31:31.243801 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:31.243724    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e0419a9-c88a-4f69-bd07-751d5406fd0e-workload-socket\") pod \"6e0419a9-c88a-4f69-bd07-751d5406fd0e\" (UID: \"6e0419a9-c88a-4f69-bd07-751d5406fd0e\") "
Sep 11 23:31:31.244022 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:31.243840    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dv967\" (UniqueName: \"kubernetes.io/projected/6e0419a9-c88a-4f69-bd07-751d5406fd0e-kube-api-access-dv967\") pod \"6e0419a9-c88a-4f69-bd07-751d5406fd0e\" (UID: \"6e0419a9-c88a-4f69-bd07-751d5406fd0e\") "
Sep 11 23:31:31.244022 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:31.243915    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e0419a9-c88a-4f69-bd07-751d5406fd0e-istio-data\") pod \"6e0419a9-c88a-4f69-bd07-751d5406fd0e\" (UID: \"6e0419a9-c88a-4f69-bd07-751d5406fd0e\") "
Sep 11 23:31:31.244022 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:31.243918    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e0419a9-c88a-4f69-bd07-751d5406fd0e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6e0419a9-c88a-4f69-bd07-751d5406fd0e" (UID: "6e0419a9-c88a-4f69-bd07-751d5406fd0e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:31:31.244022 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:31.243942    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e0419a9-c88a-4f69-bd07-751d5406fd0e-workload-certs\") pod \"6e0419a9-c88a-4f69-bd07-751d5406fd0e\" (UID: \"6e0419a9-c88a-4f69-bd07-751d5406fd0e\") "
Sep 11 23:31:31.244022 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:31.244014    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e0419a9-c88a-4f69-bd07-751d5406fd0e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6e0419a9-c88a-4f69-bd07-751d5406fd0e" (UID: "6e0419a9-c88a-4f69-bd07-751d5406fd0e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:31:31.244356 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:31.244016    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e0419a9-c88a-4f69-bd07-751d5406fd0e-istio-podinfo\") pod \"6e0419a9-c88a-4f69-bd07-751d5406fd0e\" (UID: \"6e0419a9-c88a-4f69-bd07-751d5406fd0e\") "
Sep 11 23:31:31.244356 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:31.244084    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e0419a9-c88a-4f69-bd07-751d5406fd0e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6e0419a9-c88a-4f69-bd07-751d5406fd0e" (UID: "6e0419a9-c88a-4f69-bd07-751d5406fd0e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 23:31:31.244356 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:31.244312    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e0419a9-c88a-4f69-bd07-751d5406fd0e-istio-data" (OuterVolumeSpecName: "istio-data") pod "6e0419a9-c88a-4f69-bd07-751d5406fd0e" (UID: "6e0419a9-c88a-4f69-bd07-751d5406fd0e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:31:31.244542 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:31.244372    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e0419a9-c88a-4f69-bd07-751d5406fd0e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6e0419a9-c88a-4f69-bd07-751d5406fd0e" (UID: "6e0419a9-c88a-4f69-bd07-751d5406fd0e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:31:31.244661 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:31.244577    2546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e0419a9-c88a-4f69-bd07-751d5406fd0e-workload-socket\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:31:31.244661 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:31.244597    2546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e0419a9-c88a-4f69-bd07-751d5406fd0e-istio-data\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:31:31.244661 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:31.244610    2546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e0419a9-c88a-4f69-bd07-751d5406fd0e-workload-certs\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:31:31.244661 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:31.244624    2546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e0419a9-c88a-4f69-bd07-751d5406fd0e-credential-socket\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:31:31.244661 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:31.244639    2546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e0419a9-c88a-4f69-bd07-751d5406fd0e-istiod-ca-cert\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:31:31.246267 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:31.246235    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e0419a9-c88a-4f69-bd07-751d5406fd0e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6e0419a9-c88a-4f69-bd07-751d5406fd0e" (UID: "6e0419a9-c88a-4f69-bd07-751d5406fd0e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:31:31.246401 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:31.246277    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6e0419a9-c88a-4f69-bd07-751d5406fd0e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6e0419a9-c88a-4f69-bd07-751d5406fd0e" (UID: "6e0419a9-c88a-4f69-bd07-751d5406fd0e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 23:31:31.246401 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:31.246319    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e0419a9-c88a-4f69-bd07-751d5406fd0e-istio-token" (OuterVolumeSpecName: "istio-token") pod "6e0419a9-c88a-4f69-bd07-751d5406fd0e" (UID: "6e0419a9-c88a-4f69-bd07-751d5406fd0e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 23:31:31.246401 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:31.246367    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e0419a9-c88a-4f69-bd07-751d5406fd0e-kube-api-access-dv967" (OuterVolumeSpecName: "kube-api-access-dv967") pod "6e0419a9-c88a-4f69-bd07-751d5406fd0e" (UID: "6e0419a9-c88a-4f69-bd07-751d5406fd0e"). InnerVolumeSpecName "kube-api-access-dv967". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 23:31:31.345887 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:31.345779    2546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e0419a9-c88a-4f69-bd07-751d5406fd0e-istio-podinfo\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:31:31.345887 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:31.345819    2546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e0419a9-c88a-4f69-bd07-751d5406fd0e-istio-envoy\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:31:31.345887 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:31.345830    2546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e0419a9-c88a-4f69-bd07-751d5406fd0e-istio-token\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:31:31.345887 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:31.345839    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dv967\" (UniqueName: \"kubernetes.io/projected/6e0419a9-c88a-4f69-bd07-751d5406fd0e-kube-api-access-dv967\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:31:31.405104 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:31.405065    2546 generic.go:356] "Generic (PLEG): container finished" podID="6e0419a9-c88a-4f69-bd07-751d5406fd0e" containerID="83c2605234559cac0dd49cd5f60cab1a058eeca48d4e5d97141a988ac00adbdc" exitCode=0
Sep 11 23:31:31.405316 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:31.405126    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw"
Sep 11 23:31:31.405316 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:31.405142    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw" event={"ID":"6e0419a9-c88a-4f69-bd07-751d5406fd0e","Type":"ContainerDied","Data":"83c2605234559cac0dd49cd5f60cab1a058eeca48d4e5d97141a988ac00adbdc"}
Sep 11 23:31:31.405316 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:31.405233    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw" event={"ID":"6e0419a9-c88a-4f69-bd07-751d5406fd0e","Type":"ContainerDied","Data":"7d7ff9c2b4d67de04b2bd1cdf8dedf440e61f2bc688edaed112f896cb7d2f61c"}
Sep 11 23:31:31.405316 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:31.405266    2546 scope.go:117] "RemoveContainer" containerID="83c2605234559cac0dd49cd5f60cab1a058eeca48d4e5d97141a988ac00adbdc"
Sep 11 23:31:31.418722 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:31.418580    2546 scope.go:117] "RemoveContainer" containerID="cfcb19e22a3bcd059b4ec46499e729ef235b12aaa2b5b0ee8e3b3722d0290412"
Sep 11 23:31:31.429831 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:31.429534    2546 scope.go:117] "RemoveContainer" containerID="83c2605234559cac0dd49cd5f60cab1a058eeca48d4e5d97141a988ac00adbdc"
Sep 11 23:31:31.430001 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:31:31.429968    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83c2605234559cac0dd49cd5f60cab1a058eeca48d4e5d97141a988ac00adbdc\": container with ID starting with 83c2605234559cac0dd49cd5f60cab1a058eeca48d4e5d97141a988ac00adbdc not found: ID does not exist" containerID="83c2605234559cac0dd49cd5f60cab1a058eeca48d4e5d97141a988ac00adbdc"
Sep 11 23:31:31.430074 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:31.430006    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83c2605234559cac0dd49cd5f60cab1a058eeca48d4e5d97141a988ac00adbdc"} err="failed to get container status \"83c2605234559cac0dd49cd5f60cab1a058eeca48d4e5d97141a988ac00adbdc\": rpc error: code = NotFound desc = could not find container \"83c2605234559cac0dd49cd5f60cab1a058eeca48d4e5d97141a988ac00adbdc\": container with ID starting with 83c2605234559cac0dd49cd5f60cab1a058eeca48d4e5d97141a988ac00adbdc not found: ID does not exist"
Sep 11 23:31:31.430074 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:31.430028    2546 scope.go:117] "RemoveContainer" containerID="cfcb19e22a3bcd059b4ec46499e729ef235b12aaa2b5b0ee8e3b3722d0290412"
Sep 11 23:31:31.430574 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:31:31.430364    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfcb19e22a3bcd059b4ec46499e729ef235b12aaa2b5b0ee8e3b3722d0290412\": container with ID starting with cfcb19e22a3bcd059b4ec46499e729ef235b12aaa2b5b0ee8e3b3722d0290412 not found: ID does not exist" containerID="cfcb19e22a3bcd059b4ec46499e729ef235b12aaa2b5b0ee8e3b3722d0290412"
Sep 11 23:31:31.430574 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:31.430386    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfcb19e22a3bcd059b4ec46499e729ef235b12aaa2b5b0ee8e3b3722d0290412"} err="failed to get container status \"cfcb19e22a3bcd059b4ec46499e729ef235b12aaa2b5b0ee8e3b3722d0290412\": rpc error: code = NotFound desc = could not find container \"cfcb19e22a3bcd059b4ec46499e729ef235b12aaa2b5b0ee8e3b3722d0290412\": container with ID starting with cfcb19e22a3bcd059b4ec46499e729ef235b12aaa2b5b0ee8e3b3722d0290412 not found: ID does not exist"
Sep 11 23:31:31.433192 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:31.433148    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw"]
Sep 11 23:31:31.448122 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:31.448090    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-451902-openshift-default-c6cc6d66b-b5hbw"]
Sep 11 23:31:32.498372 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:32.496750    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e0419a9-c88a-4f69-bd07-751d5406fd0e" path="/var/lib/kubelet/pods/6e0419a9-c88a-4f69-bd07-751d5406fd0e/volumes"
Sep 11 23:31:40.009385 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:40.009338    2546 patch_prober.go:28] interesting pod/e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 11 23:31:40.009793 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:40.009428    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8" podUID="181669c0-2cfb-463e-9dd2-3c4c874b8442" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 11 23:31:41.493485 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:41.493451    2546 generic.go:356] "Generic (PLEG): container finished" podID="181669c0-2cfb-463e-9dd2-3c4c874b8442" containerID="c9b554fd3b7cc40398cff34d03ae89630868a2922b8f87f6b027516fd5ca09c5" exitCode=0
Sep 11 23:31:41.494035 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:41.493530    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8" event={"ID":"181669c0-2cfb-463e-9dd2-3c4c874b8442","Type":"ContainerDied","Data":"c9b554fd3b7cc40398cff34d03ae89630868a2922b8f87f6b027516fd5ca09c5"}
Sep 11 23:31:41.494117 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:41.494097    2546 scope.go:117] "RemoveContainer" containerID="c9b554fd3b7cc40398cff34d03ae89630868a2922b8f87f6b027516fd5ca09c5"
Sep 11 23:31:41.495555 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:41.495529    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:31:41.495624 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:41.495600    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:31:41.495662 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:41.495649    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:31:41.916719 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:41.916683    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d1b012-59676779d6-48drk"]
Sep 11 23:31:41.917237 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:41.917204    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d1b012-59676779d6-48drk" podUID="05ebbb64-3875-4dfa-af7f-7f68dfded62a" containerName="payload-pre-processing" containerID="cri-o://7bb8bbead719dd86d6f37347f22bf15ca3c7bfada9d3a01fc0380d1322a817e9" gracePeriod=30
Sep 11 23:31:42.137050 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:42.137012    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d1b012-wr45p"]
Sep 11 23:31:42.142974 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:42.142933    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d1b012-wr45p"]
Sep 11 23:31:42.222139 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:42.222110    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d1b012-59676779d6-48drk"
Sep 11 23:31:42.265179 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:42.262934    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05ebbb64-3875-4dfa-af7f-7f68dfded62a-plugins-config-volume\") pod \"05ebbb64-3875-4dfa-af7f-7f68dfded62a\" (UID: \"05ebbb64-3875-4dfa-af7f-7f68dfded62a\") "
Sep 11 23:31:42.265179 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:42.263522    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tktbg\" (UniqueName: \"kubernetes.io/projected/05ebbb64-3875-4dfa-af7f-7f68dfded62a-kube-api-access-tktbg\") pod \"05ebbb64-3875-4dfa-af7f-7f68dfded62a\" (UID: \"05ebbb64-3875-4dfa-af7f-7f68dfded62a\") "
Sep 11 23:31:42.265179 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:42.263371    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05ebbb64-3875-4dfa-af7f-7f68dfded62a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "05ebbb64-3875-4dfa-af7f-7f68dfded62a" (UID: "05ebbb64-3875-4dfa-af7f-7f68dfded62a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 23:31:42.275694 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:42.275652    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05ebbb64-3875-4dfa-af7f-7f68dfded62a-kube-api-access-tktbg" (OuterVolumeSpecName: "kube-api-access-tktbg") pod "05ebbb64-3875-4dfa-af7f-7f68dfded62a" (UID: "05ebbb64-3875-4dfa-af7f-7f68dfded62a"). InnerVolumeSpecName "kube-api-access-tktbg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 23:31:42.364664 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:42.364625    2546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05ebbb64-3875-4dfa-af7f-7f68dfded62a-plugins-config-volume\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:31:42.364664 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:42.364666    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tktbg\" (UniqueName: \"kubernetes.io/projected/05ebbb64-3875-4dfa-af7f-7f68dfded62a-kube-api-access-tktbg\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:31:42.496276 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:42.496191    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5503de09-30bb-4b85-b1af-e22f2be25072" path="/var/lib/kubelet/pods/5503de09-30bb-4b85-b1af-e22f2be25072/volumes"
Sep 11 23:31:42.502698 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:42.502660    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8" event={"ID":"181669c0-2cfb-463e-9dd2-3c4c874b8442","Type":"ContainerStarted","Data":"17dfb0feef6c5f72f15ac1966f4a4634777d371e33af0c149b36b30448938a86"}
Sep 11 23:31:42.504325 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:42.504280    2546 generic.go:356] "Generic (PLEG): container finished" podID="05ebbb64-3875-4dfa-af7f-7f68dfded62a" containerID="7bb8bbead719dd86d6f37347f22bf15ca3c7bfada9d3a01fc0380d1322a817e9" exitCode=0
Sep 11 23:31:42.504474 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:42.504332    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d1b012-59676779d6-48drk"
Sep 11 23:31:42.504474 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:42.504402    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d1b012-59676779d6-48drk" event={"ID":"05ebbb64-3875-4dfa-af7f-7f68dfded62a","Type":"ContainerDied","Data":"7bb8bbead719dd86d6f37347f22bf15ca3c7bfada9d3a01fc0380d1322a817e9"}
Sep 11 23:31:42.504474 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:42.504455    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d1b012-59676779d6-48drk" event={"ID":"05ebbb64-3875-4dfa-af7f-7f68dfded62a","Type":"ContainerDied","Data":"650af064d45b4f2e517ab0df357406afff7fa00f27f62c581e8b3f12b302d97f"}
Sep 11 23:31:42.504637 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:42.504480    2546 scope.go:117] "RemoveContainer" containerID="7bb8bbead719dd86d6f37347f22bf15ca3c7bfada9d3a01fc0380d1322a817e9"
Sep 11 23:31:42.514989 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:42.514969    2546 scope.go:117] "RemoveContainer" containerID="7bb8bbead719dd86d6f37347f22bf15ca3c7bfada9d3a01fc0380d1322a817e9"
Sep 11 23:31:42.515369 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:31:42.515346    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bb8bbead719dd86d6f37347f22bf15ca3c7bfada9d3a01fc0380d1322a817e9\": container with ID starting with 7bb8bbead719dd86d6f37347f22bf15ca3c7bfada9d3a01fc0380d1322a817e9 not found: ID does not exist" containerID="7bb8bbead719dd86d6f37347f22bf15ca3c7bfada9d3a01fc0380d1322a817e9"
Sep 11 23:31:42.515431 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:42.515378    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bb8bbead719dd86d6f37347f22bf15ca3c7bfada9d3a01fc0380d1322a817e9"} err="failed to get container status \"7bb8bbead719dd86d6f37347f22bf15ca3c7bfada9d3a01fc0380d1322a817e9\": rpc error: code = NotFound desc = could not find container \"7bb8bbead719dd86d6f37347f22bf15ca3c7bfada9d3a01fc0380d1322a817e9\": container with ID starting with 7bb8bbead719dd86d6f37347f22bf15ca3c7bfada9d3a01fc0380d1322a817e9 not found: ID does not exist"
Sep 11 23:31:42.551929 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:42.551868    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d1b012-59676779d6-48drk"]
Sep 11 23:31:42.561458 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:42.561407    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d1b012-59676779d6-48drk"]
Sep 11 23:31:42.743170 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:42.743095    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8"
Sep 11 23:31:42.743412 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:42.743273    2546 patch_prober.go:28] interesting pod/e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Sep 11 23:31:42.743412 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:42.743326    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8" podUID="181669c0-2cfb-463e-9dd2-3c4c874b8442" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Sep 11 23:31:43.743064 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:43.743021    2546 patch_prober.go:28] interesting pod/e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Sep 11 23:31:43.743775 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:43.743731    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8" podUID="181669c0-2cfb-463e-9dd2-3c4c874b8442" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Sep 11 23:31:44.495349 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:44.495311    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05ebbb64-3875-4dfa-af7f-7f68dfded62a" path="/var/lib/kubelet/pods/05ebbb64-3875-4dfa-af7f-7f68dfded62a/volumes"
Sep 11 23:31:44.743281 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:44.743240    2546 patch_prober.go:28] interesting pod/e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Sep 11 23:31:44.743927 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:44.743888    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8" podUID="181669c0-2cfb-463e-9dd2-3c4c874b8442" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Sep 11 23:31:45.743017 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:45.742970    2546 patch_prober.go:28] interesting pod/e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Sep 11 23:31:45.743355 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:45.743318    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8" podUID="181669c0-2cfb-463e-9dd2-3c4c874b8442" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Sep 11 23:31:46.743662 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:46.743616    2546 patch_prober.go:28] interesting pod/e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Sep 11 23:31:46.744222 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:46.743697    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8" podUID="181669c0-2cfb-463e-9dd2-3c4c874b8442" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Sep 11 23:31:47.743402 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:47.743358    2546 patch_prober.go:28] interesting pod/e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Sep 11 23:31:47.743604 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:47.743450    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8" podUID="181669c0-2cfb-463e-9dd2-3c4c874b8442" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Sep 11 23:31:48.747039 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:48.747002    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8"
Sep 11 23:31:48.747463 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:48.747386    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8"
Sep 11 23:31:48.748035 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:48.748016    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8"
Sep 11 23:31:49.011184 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:49.011055    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8"]
Sep 11 23:31:49.343800 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:49.343682    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cd48eb-lqpls"]
Sep 11 23:31:49.344734 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:49.344697    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48592704-31d9-4e6f-bad1-ac862a6dfa72" containerName="istio-proxy"
Sep 11 23:31:49.344734 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:49.344733    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="48592704-31d9-4e6f-bad1-ac862a6dfa72" containerName="istio-proxy"
Sep 11 23:31:49.344963 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:49.344749    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e0419a9-c88a-4f69-bd07-751d5406fd0e" containerName="istio-proxy"
Sep 11 23:31:49.344963 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:49.344758    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e0419a9-c88a-4f69-bd07-751d5406fd0e" containerName="istio-proxy"
Sep 11 23:31:49.344963 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:49.344780    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5503de09-30bb-4b85-b1af-e22f2be25072" containerName="revoke-keys"
Sep 11 23:31:49.344963 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:49.344789    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="5503de09-30bb-4b85-b1af-e22f2be25072" containerName="revoke-keys"
Sep 11 23:31:49.344963 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:49.344800    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e0419a9-c88a-4f69-bd07-751d5406fd0e" containerName="istio-proxy"
Sep 11 23:31:49.344963 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:49.344807    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e0419a9-c88a-4f69-bd07-751d5406fd0e" containerName="istio-proxy"
Sep 11 23:31:49.344963 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:49.344820    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05ebbb64-3875-4dfa-af7f-7f68dfded62a" containerName="payload-pre-processing"
Sep 11 23:31:49.344963 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:49.344829    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="05ebbb64-3875-4dfa-af7f-7f68dfded62a" containerName="payload-pre-processing"
Sep 11 23:31:49.344963 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:49.344940    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="48592704-31d9-4e6f-bad1-ac862a6dfa72" containerName="istio-proxy"
Sep 11 23:31:49.344963 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:49.344954    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="5503de09-30bb-4b85-b1af-e22f2be25072" containerName="revoke-keys"
Sep 11 23:31:49.344963 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:49.344962    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e0419a9-c88a-4f69-bd07-751d5406fd0e" containerName="istio-proxy"
Sep 11 23:31:49.344963 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:49.344971    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e0419a9-c88a-4f69-bd07-751d5406fd0e" containerName="istio-proxy"
Sep 11 23:31:49.345634 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:49.344983    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="05ebbb64-3875-4dfa-af7f-7f68dfded62a" containerName="payload-pre-processing"
Sep 11 23:31:49.348793 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:49.348766    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cd48eb-lqpls"
Sep 11 23:31:49.352348 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:49.352320    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 23:31:49.352511 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:49.352397    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vrx7m\""
Sep 11 23:31:49.358386 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:49.357870    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cd48eb-lqpls"]
Sep 11 23:31:49.456599 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:49.456564    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a09c628-3668-4da5-a208-5e80fe0d146e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-cd48eb-lqpls\" (UID: \"9a09c628-3668-4da5-a208-5e80fe0d146e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cd48eb-lqpls"
Sep 11 23:31:49.557842 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:49.557805    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a09c628-3668-4da5-a208-5e80fe0d146e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-cd48eb-lqpls\" (UID: \"9a09c628-3668-4da5-a208-5e80fe0d146e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cd48eb-lqpls"
Sep 11 23:31:49.562195 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:49.559053    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a09c628-3668-4da5-a208-5e80fe0d146e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-cd48eb-lqpls\" (UID: \"9a09c628-3668-4da5-a208-5e80fe0d146e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cd48eb-lqpls"
Sep 11 23:31:49.573893 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:49.573860    2546 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8" secret="" err="secret \"e2e-api-b-d1b012-openshift-default-dockercfg-dbjhc\" not found"
Sep 11 23:31:49.662594 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:49.662558    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cd48eb-lqpls"
Sep 11 23:31:49.941143 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:49.941111    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cd48eb-lqpls"]
Sep 11 23:31:49.942311 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:31:49.942263    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a09c628_3668_4da5_a208_5e80fe0d146e.slice/crio-0f569eec71cb77ac6dc18062fbefd310adc4bdd954ecc678c632de9adc74f7bc WatchSource:0}: Error finding container 0f569eec71cb77ac6dc18062fbefd310adc4bdd954ecc678c632de9adc74f7bc: Status 404 returned error can't find the container with id 0f569eec71cb77ac6dc18062fbefd310adc4bdd954ecc678c632de9adc74f7bc
Sep 11 23:31:50.581800 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:50.581706    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cd48eb-lqpls" event={"ID":"9a09c628-3668-4da5-a208-5e80fe0d146e","Type":"ContainerStarted","Data":"d0fc31256b89259adfbcc8f693c1974724634d5fe2ebaf0d6d337e42adc2bfa9"}
Sep 11 23:31:50.581800 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:50.581759    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cd48eb-lqpls" event={"ID":"9a09c628-3668-4da5-a208-5e80fe0d146e","Type":"ContainerStarted","Data":"0f569eec71cb77ac6dc18062fbefd310adc4bdd954ecc678c632de9adc74f7bc"}
Sep 11 23:31:50.582058 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:50.581821    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8" podUID="181669c0-2cfb-463e-9dd2-3c4c874b8442" containerName="istio-proxy" containerID="cri-o://17dfb0feef6c5f72f15ac1966f4a4634777d371e33af0c149b36b30448938a86" gracePeriod=30
Sep 11 23:31:50.601636 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:50.601581    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cd48eb-lqpls" podStartSLOduration=1.6015646989999999 podStartE2EDuration="1.601564699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 23:31:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:31:50.598836875 +0000 UTC m=+1834.631966092" watchObservedRunningTime="2026-09-11 23:31:50.601564699 +0000 UTC m=+1834.634693924"
Sep 11 23:31:51.591472 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:51.591433    2546 generic.go:356] "Generic (PLEG): container finished" podID="9a09c628-3668-4da5-a208-5e80fe0d146e" containerID="d0fc31256b89259adfbcc8f693c1974724634d5fe2ebaf0d6d337e42adc2bfa9" exitCode=0
Sep 11 23:31:51.591885 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:51.591519    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cd48eb-lqpls" event={"ID":"9a09c628-3668-4da5-a208-5e80fe0d146e","Type":"ContainerDied","Data":"d0fc31256b89259adfbcc8f693c1974724634d5fe2ebaf0d6d337e42adc2bfa9"}
Sep 11 23:31:52.752238 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:52.752203    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cd48eb-lqpls"
Sep 11 23:31:52.891815 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:52.891713    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a09c628-3668-4da5-a208-5e80fe0d146e-maas-api-service-ca\") pod \"9a09c628-3668-4da5-a208-5e80fe0d146e\" (UID: \"9a09c628-3668-4da5-a208-5e80fe0d146e\") "
Sep 11 23:31:52.892118 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:52.892090    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a09c628-3668-4da5-a208-5e80fe0d146e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9a09c628-3668-4da5-a208-5e80fe0d146e" (UID: "9a09c628-3668-4da5-a208-5e80fe0d146e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 23:31:52.992980 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:52.992938    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a09c628-3668-4da5-a208-5e80fe0d146e-maas-api-service-ca\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:31:53.606876 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:53.606846    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cd48eb-lqpls"
Sep 11 23:31:53.607065 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:53.606879    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cd48eb-lqpls" event={"ID":"9a09c628-3668-4da5-a208-5e80fe0d146e","Type":"ContainerDied","Data":"0f569eec71cb77ac6dc18062fbefd310adc4bdd954ecc678c632de9adc74f7bc"}
Sep 11 23:31:53.607065 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:53.606927    2546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f569eec71cb77ac6dc18062fbefd310adc4bdd954ecc678c632de9adc74f7bc"
Sep 11 23:31:55.850191 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:55.850089    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8"
Sep 11 23:31:55.925620 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:55.925576    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/181669c0-2cfb-463e-9dd2-3c4c874b8442-istio-data\") pod \"181669c0-2cfb-463e-9dd2-3c4c874b8442\" (UID: \"181669c0-2cfb-463e-9dd2-3c4c874b8442\") "
Sep 11 23:31:55.925784 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:55.925655    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/181669c0-2cfb-463e-9dd2-3c4c874b8442-istio-token\") pod \"181669c0-2cfb-463e-9dd2-3c4c874b8442\" (UID: \"181669c0-2cfb-463e-9dd2-3c4c874b8442\") "
Sep 11 23:31:55.925784 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:55.925701    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/181669c0-2cfb-463e-9dd2-3c4c874b8442-istio-podinfo\") pod \"181669c0-2cfb-463e-9dd2-3c4c874b8442\" (UID: \"181669c0-2cfb-463e-9dd2-3c4c874b8442\") "
Sep 11 23:31:55.925784 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:55.925747    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/181669c0-2cfb-463e-9dd2-3c4c874b8442-istio-envoy\") pod \"181669c0-2cfb-463e-9dd2-3c4c874b8442\" (UID: \"181669c0-2cfb-463e-9dd2-3c4c874b8442\") "
Sep 11 23:31:55.925784 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:55.925775    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/181669c0-2cfb-463e-9dd2-3c4c874b8442-workload-socket\") pod \"181669c0-2cfb-463e-9dd2-3c4c874b8442\" (UID: \"181669c0-2cfb-463e-9dd2-3c4c874b8442\") "
Sep 11 23:31:55.926020 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:55.925823    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/181669c0-2cfb-463e-9dd2-3c4c874b8442-credential-socket\") pod \"181669c0-2cfb-463e-9dd2-3c4c874b8442\" (UID: \"181669c0-2cfb-463e-9dd2-3c4c874b8442\") "
Sep 11 23:31:55.926020 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:55.925866    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/181669c0-2cfb-463e-9dd2-3c4c874b8442-workload-certs\") pod \"181669c0-2cfb-463e-9dd2-3c4c874b8442\" (UID: \"181669c0-2cfb-463e-9dd2-3c4c874b8442\") "
Sep 11 23:31:55.926020 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:55.925895    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/181669c0-2cfb-463e-9dd2-3c4c874b8442-istiod-ca-cert\") pod \"181669c0-2cfb-463e-9dd2-3c4c874b8442\" (UID: \"181669c0-2cfb-463e-9dd2-3c4c874b8442\") "
Sep 11 23:31:55.926020 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:55.925929    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/181669c0-2cfb-463e-9dd2-3c4c874b8442-istio-data" (OuterVolumeSpecName: "istio-data") pod "181669c0-2cfb-463e-9dd2-3c4c874b8442" (UID: "181669c0-2cfb-463e-9dd2-3c4c874b8442"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:31:55.926020 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:55.926003    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xzl72\" (UniqueName: \"kubernetes.io/projected/181669c0-2cfb-463e-9dd2-3c4c874b8442-kube-api-access-xzl72\") pod \"181669c0-2cfb-463e-9dd2-3c4c874b8442\" (UID: \"181669c0-2cfb-463e-9dd2-3c4c874b8442\") "
Sep 11 23:31:55.926333 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:55.926134    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/181669c0-2cfb-463e-9dd2-3c4c874b8442-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "181669c0-2cfb-463e-9dd2-3c4c874b8442" (UID: "181669c0-2cfb-463e-9dd2-3c4c874b8442"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:31:55.926333 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:55.926317    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/181669c0-2cfb-463e-9dd2-3c4c874b8442-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "181669c0-2cfb-463e-9dd2-3c4c874b8442" (UID: "181669c0-2cfb-463e-9dd2-3c4c874b8442"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:31:55.926517 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:55.926491    2546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/181669c0-2cfb-463e-9dd2-3c4c874b8442-istio-data\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:31:55.926687 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:55.926669    2546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/181669c0-2cfb-463e-9dd2-3c4c874b8442-credential-socket\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:31:55.926988 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:55.926966    2546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/181669c0-2cfb-463e-9dd2-3c4c874b8442-workload-certs\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:31:55.926988 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:55.926505    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/181669c0-2cfb-463e-9dd2-3c4c874b8442-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "181669c0-2cfb-463e-9dd2-3c4c874b8442" (UID: "181669c0-2cfb-463e-9dd2-3c4c874b8442"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:31:55.927172 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:55.926653    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/181669c0-2cfb-463e-9dd2-3c4c874b8442-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "181669c0-2cfb-463e-9dd2-3c4c874b8442" (UID: "181669c0-2cfb-463e-9dd2-3c4c874b8442"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 23:31:55.928368 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:55.928344    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/181669c0-2cfb-463e-9dd2-3c4c874b8442-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "181669c0-2cfb-463e-9dd2-3c4c874b8442" (UID: "181669c0-2cfb-463e-9dd2-3c4c874b8442"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 23:31:55.928478 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:55.928410    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/181669c0-2cfb-463e-9dd2-3c4c874b8442-kube-api-access-xzl72" (OuterVolumeSpecName: "kube-api-access-xzl72") pod "181669c0-2cfb-463e-9dd2-3c4c874b8442" (UID: "181669c0-2cfb-463e-9dd2-3c4c874b8442"). InnerVolumeSpecName "kube-api-access-xzl72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 23:31:55.928657 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:55.928637    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/181669c0-2cfb-463e-9dd2-3c4c874b8442-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "181669c0-2cfb-463e-9dd2-3c4c874b8442" (UID: "181669c0-2cfb-463e-9dd2-3c4c874b8442"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:31:55.928700 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:55.928677    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/181669c0-2cfb-463e-9dd2-3c4c874b8442-istio-token" (OuterVolumeSpecName: "istio-token") pod "181669c0-2cfb-463e-9dd2-3c4c874b8442" (UID: "181669c0-2cfb-463e-9dd2-3c4c874b8442"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 23:31:56.028467 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:56.028427    2546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/181669c0-2cfb-463e-9dd2-3c4c874b8442-istiod-ca-cert\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:31:56.028467 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:56.028462    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzl72\" (UniqueName: \"kubernetes.io/projected/181669c0-2cfb-463e-9dd2-3c4c874b8442-kube-api-access-xzl72\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:31:56.028467 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:56.028473    2546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/181669c0-2cfb-463e-9dd2-3c4c874b8442-istio-token\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:31:56.028721 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:56.028482    2546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/181669c0-2cfb-463e-9dd2-3c4c874b8442-istio-podinfo\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:31:56.028721 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:56.028490    2546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/181669c0-2cfb-463e-9dd2-3c4c874b8442-istio-envoy\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:31:56.028721 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:56.028498    2546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/181669c0-2cfb-463e-9dd2-3c4c874b8442-workload-socket\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:31:56.629670 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:56.629637    2546 generic.go:356] "Generic (PLEG): container finished" podID="181669c0-2cfb-463e-9dd2-3c4c874b8442" containerID="17dfb0feef6c5f72f15ac1966f4a4634777d371e33af0c149b36b30448938a86" exitCode=0
Sep 11 23:31:56.629865 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:56.629696    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8" event={"ID":"181669c0-2cfb-463e-9dd2-3c4c874b8442","Type":"ContainerDied","Data":"17dfb0feef6c5f72f15ac1966f4a4634777d371e33af0c149b36b30448938a86"}
Sep 11 23:31:56.629865 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:56.629706    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8"
Sep 11 23:31:56.629865 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:56.629741    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8" event={"ID":"181669c0-2cfb-463e-9dd2-3c4c874b8442","Type":"ContainerDied","Data":"3230fdaf1734f5cf2774703ff1f04dfb425429bc5570c6ed9fe08bf6f884ea8e"}
Sep 11 23:31:56.629865 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:56.629763    2546 scope.go:117] "RemoveContainer" containerID="17dfb0feef6c5f72f15ac1966f4a4634777d371e33af0c149b36b30448938a86"
Sep 11 23:31:56.640451 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:56.640425    2546 scope.go:117] "RemoveContainer" containerID="c9b554fd3b7cc40398cff34d03ae89630868a2922b8f87f6b027516fd5ca09c5"
Sep 11 23:31:56.652342 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:56.652317    2546 scope.go:117] "RemoveContainer" containerID="17dfb0feef6c5f72f15ac1966f4a4634777d371e33af0c149b36b30448938a86"
Sep 11 23:31:56.652748 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:31:56.652722    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17dfb0feef6c5f72f15ac1966f4a4634777d371e33af0c149b36b30448938a86\": container with ID starting with 17dfb0feef6c5f72f15ac1966f4a4634777d371e33af0c149b36b30448938a86 not found: ID does not exist" containerID="17dfb0feef6c5f72f15ac1966f4a4634777d371e33af0c149b36b30448938a86"
Sep 11 23:31:56.652845 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:56.652757    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17dfb0feef6c5f72f15ac1966f4a4634777d371e33af0c149b36b30448938a86"} err="failed to get container status \"17dfb0feef6c5f72f15ac1966f4a4634777d371e33af0c149b36b30448938a86\": rpc error: code = NotFound desc = could not find container \"17dfb0feef6c5f72f15ac1966f4a4634777d371e33af0c149b36b30448938a86\": container with ID starting with 17dfb0feef6c5f72f15ac1966f4a4634777d371e33af0c149b36b30448938a86 not found: ID does not exist"
Sep 11 23:31:56.652845 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:56.652776    2546 scope.go:117] "RemoveContainer" containerID="c9b554fd3b7cc40398cff34d03ae89630868a2922b8f87f6b027516fd5ca09c5"
Sep 11 23:31:56.652947 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:56.652899    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8"]
Sep 11 23:31:56.653095 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:31:56.653077    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9b554fd3b7cc40398cff34d03ae89630868a2922b8f87f6b027516fd5ca09c5\": container with ID starting with c9b554fd3b7cc40398cff34d03ae89630868a2922b8f87f6b027516fd5ca09c5 not found: ID does not exist" containerID="c9b554fd3b7cc40398cff34d03ae89630868a2922b8f87f6b027516fd5ca09c5"
Sep 11 23:31:56.653143 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:56.653100    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9b554fd3b7cc40398cff34d03ae89630868a2922b8f87f6b027516fd5ca09c5"} err="failed to get container status \"c9b554fd3b7cc40398cff34d03ae89630868a2922b8f87f6b027516fd5ca09c5\": rpc error: code = NotFound desc = could not find container \"c9b554fd3b7cc40398cff34d03ae89630868a2922b8f87f6b027516fd5ca09c5\": container with ID starting with c9b554fd3b7cc40398cff34d03ae89630868a2922b8f87f6b027516fd5ca09c5 not found: ID does not exist"
Sep 11 23:31:56.655973 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:56.655954    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-d1b012-openshift-default-7db67f95bb-tslk8"]
Sep 11 23:31:58.495348 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:31:58.495315    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="181669c0-2cfb-463e-9dd2-3c4c874b8442" path="/var/lib/kubelet/pods/181669c0-2cfb-463e-9dd2-3c4c874b8442/volumes"
Sep 11 23:32:00.095323 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:00.095285    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cd48eb-29819490-m2hcj"]
Sep 11 23:32:00.097970 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:00.097947    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cd48eb-29819490-m2hcj"]
Sep 11 23:32:00.335669 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:00.335624    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cd48eb-lqpls"]
Sep 11 23:32:00.340022 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:00.339991    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cd48eb-lqpls"]
Sep 11 23:32:00.502063 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:00.502014    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38d1bf05-fa75-4d8b-9f46-f843f80c0f86" path="/var/lib/kubelet/pods/38d1bf05-fa75-4d8b-9f46-f843f80c0f86/volumes"
Sep 11 23:32:00.502777 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:00.502751    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a09c628-3668-4da5-a208-5e80fe0d146e" path="/var/lib/kubelet/pods/9a09c628-3668-4da5-a208-5e80fe0d146e/volumes"
Sep 11 23:32:03.790375 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:03.790338    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"]
Sep 11 23:32:07.292784 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:07.292742    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-cd48eb-openshift-default-76cd487f58-fm5sq"]
Sep 11 23:32:07.293218 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:07.292949    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-cd48eb-openshift-default-76cd487f58-fm5sq" podUID="48cd2fcb-f864-4e43-bcfb-cadfd9964332" containerName="istio-proxy" containerID="cri-o://39c40ddc5208abebb3b8b5bf03a456803ce38921ff8f48b6f8684349369088f5" gracePeriod=30
Sep 11 23:32:07.609779 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:07.609683    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-97d9dc-chwvd"]
Sep 11 23:32:07.610139 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:07.610126    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a09c628-3668-4da5-a208-5e80fe0d146e" containerName="revoke-keys"
Sep 11 23:32:07.610228 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:07.610140    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a09c628-3668-4da5-a208-5e80fe0d146e" containerName="revoke-keys"
Sep 11 23:32:07.610228 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:07.610179    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="181669c0-2cfb-463e-9dd2-3c4c874b8442" containerName="istio-proxy"
Sep 11 23:32:07.610228 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:07.610188    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="181669c0-2cfb-463e-9dd2-3c4c874b8442" containerName="istio-proxy"
Sep 11 23:32:07.610228 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:07.610219    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="181669c0-2cfb-463e-9dd2-3c4c874b8442" containerName="istio-proxy"
Sep 11 23:32:07.610228 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:07.610228    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="181669c0-2cfb-463e-9dd2-3c4c874b8442" containerName="istio-proxy"
Sep 11 23:32:07.610389 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:07.610317    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a09c628-3668-4da5-a208-5e80fe0d146e" containerName="revoke-keys"
Sep 11 23:32:07.610389 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:07.610330    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="181669c0-2cfb-463e-9dd2-3c4c874b8442" containerName="istio-proxy"
Sep 11 23:32:07.614927 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:07.614905    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-97d9dc-chwvd"
Sep 11 23:32:07.619540 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:07.619496    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 23:32:07.619816 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:07.619790    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vrx7m\""
Sep 11 23:32:07.624125 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:07.623959    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-97d9dc-chwvd"]
Sep 11 23:32:07.756054 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:07.756009    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2eca8b5d-ddf8-4bd3-a6e1-7cc9787499ba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-97d9dc-chwvd\" (UID: \"2eca8b5d-ddf8-4bd3-a6e1-7cc9787499ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-97d9dc-chwvd"
Sep 11 23:32:07.857244 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:07.857198    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2eca8b5d-ddf8-4bd3-a6e1-7cc9787499ba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-97d9dc-chwvd\" (UID: \"2eca8b5d-ddf8-4bd3-a6e1-7cc9787499ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-97d9dc-chwvd"
Sep 11 23:32:07.857905 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:07.857884    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2eca8b5d-ddf8-4bd3-a6e1-7cc9787499ba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-97d9dc-chwvd\" (UID: \"2eca8b5d-ddf8-4bd3-a6e1-7cc9787499ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-97d9dc-chwvd"
Sep 11 23:32:07.928453 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:07.928415    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-97d9dc-chwvd"
Sep 11 23:32:08.208905 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:08.208878    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-97d9dc-chwvd"]
Sep 11 23:32:08.210325 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:32:08.210296    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eca8b5d_ddf8_4bd3_a6e1_7cc9787499ba.slice/crio-6964b2eea45eccb671f62697e8c5c6f6cad49c1bf762186aadf432d99a9eadd5 WatchSource:0}: Error finding container 6964b2eea45eccb671f62697e8c5c6f6cad49c1bf762186aadf432d99a9eadd5: Status 404 returned error can't find the container with id 6964b2eea45eccb671f62697e8c5c6f6cad49c1bf762186aadf432d99a9eadd5
Sep 11 23:32:08.719129 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:08.719088    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-97d9dc-chwvd" event={"ID":"2eca8b5d-ddf8-4bd3-a6e1-7cc9787499ba","Type":"ContainerStarted","Data":"b7bfeed57d54ae77d1d8db9504ba90dac133ec27b547aba97086113acfea5f51"}
Sep 11 23:32:08.719577 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:08.719180    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-97d9dc-chwvd" event={"ID":"2eca8b5d-ddf8-4bd3-a6e1-7cc9787499ba","Type":"ContainerStarted","Data":"6964b2eea45eccb671f62697e8c5c6f6cad49c1bf762186aadf432d99a9eadd5"}
Sep 11 23:32:08.740416 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:08.740363    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-97d9dc-chwvd" podStartSLOduration=1.740347098 podStartE2EDuration="1.740347098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 23:32:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:32:08.737879476 +0000 UTC m=+1852.771008693" watchObservedRunningTime="2026-09-11 23:32:08.740347098 +0000 UTC m=+1852.773476314"
Sep 11 23:32:09.727734 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:09.727695    2546 generic.go:356] "Generic (PLEG): container finished" podID="2eca8b5d-ddf8-4bd3-a6e1-7cc9787499ba" containerID="b7bfeed57d54ae77d1d8db9504ba90dac133ec27b547aba97086113acfea5f51" exitCode=0
Sep 11 23:32:09.728291 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:09.727790    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-97d9dc-chwvd" event={"ID":"2eca8b5d-ddf8-4bd3-a6e1-7cc9787499ba","Type":"ContainerDied","Data":"b7bfeed57d54ae77d1d8db9504ba90dac133ec27b547aba97086113acfea5f51"}
Sep 11 23:32:10.877861 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:10.877830    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-97d9dc-chwvd"
Sep 11 23:32:10.889197 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:10.889141    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2eca8b5d-ddf8-4bd3-a6e1-7cc9787499ba-maas-api-service-ca\") pod \"2eca8b5d-ddf8-4bd3-a6e1-7cc9787499ba\" (UID: \"2eca8b5d-ddf8-4bd3-a6e1-7cc9787499ba\") "
Sep 11 23:32:10.889522 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:10.889502    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2eca8b5d-ddf8-4bd3-a6e1-7cc9787499ba-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2eca8b5d-ddf8-4bd3-a6e1-7cc9787499ba" (UID: "2eca8b5d-ddf8-4bd3-a6e1-7cc9787499ba"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 23:32:10.990998 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:10.990894    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2eca8b5d-ddf8-4bd3-a6e1-7cc9787499ba-maas-api-service-ca\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:32:11.742355 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:11.742323    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-97d9dc-chwvd"
Sep 11 23:32:11.742536 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:11.742361    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-97d9dc-chwvd" event={"ID":"2eca8b5d-ddf8-4bd3-a6e1-7cc9787499ba","Type":"ContainerDied","Data":"6964b2eea45eccb671f62697e8c5c6f6cad49c1bf762186aadf432d99a9eadd5"}
Sep 11 23:32:11.742536 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:11.742415    2546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6964b2eea45eccb671f62697e8c5c6f6cad49c1bf762186aadf432d99a9eadd5"
Sep 11 23:32:12.364835 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:32:12.364796    2546 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48cd2fcb_f864_4e43_bcfb_cadfd9964332.slice/crio-conmon-39c40ddc5208abebb3b8b5bf03a456803ce38921ff8f48b6f8684349369088f5.scope\": RecentStats: unable to find data in memory cache]"
Sep 11 23:32:12.554617 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:12.554590    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-cd48eb-openshift-default-76cd487f58-fm5sq"
Sep 11 23:32:12.608226 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:12.608113    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48cd2fcb-f864-4e43-bcfb-cadfd9964332-istio-token\") pod \"48cd2fcb-f864-4e43-bcfb-cadfd9964332\" (UID: \"48cd2fcb-f864-4e43-bcfb-cadfd9964332\") "
Sep 11 23:32:12.608226 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:12.608166    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48cd2fcb-f864-4e43-bcfb-cadfd9964332-workload-certs\") pod \"48cd2fcb-f864-4e43-bcfb-cadfd9964332\" (UID: \"48cd2fcb-f864-4e43-bcfb-cadfd9964332\") "
Sep 11 23:32:12.608480 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:12.608263    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48cd2fcb-f864-4e43-bcfb-cadfd9964332-istio-podinfo\") pod \"48cd2fcb-f864-4e43-bcfb-cadfd9964332\" (UID: \"48cd2fcb-f864-4e43-bcfb-cadfd9964332\") "
Sep 11 23:32:12.608480 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:12.608303    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dnv58\" (UniqueName: \"kubernetes.io/projected/48cd2fcb-f864-4e43-bcfb-cadfd9964332-kube-api-access-dnv58\") pod \"48cd2fcb-f864-4e43-bcfb-cadfd9964332\" (UID: \"48cd2fcb-f864-4e43-bcfb-cadfd9964332\") "
Sep 11 23:32:12.608480 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:12.608340    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48cd2fcb-f864-4e43-bcfb-cadfd9964332-istio-envoy\") pod \"48cd2fcb-f864-4e43-bcfb-cadfd9964332\" (UID: \"48cd2fcb-f864-4e43-bcfb-cadfd9964332\") "
Sep 11 23:32:12.608480 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:12.608368    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48cd2fcb-f864-4e43-bcfb-cadfd9964332-istio-data\") pod \"48cd2fcb-f864-4e43-bcfb-cadfd9964332\" (UID: \"48cd2fcb-f864-4e43-bcfb-cadfd9964332\") "
Sep 11 23:32:12.608480 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:12.608427    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48cd2fcb-f864-4e43-bcfb-cadfd9964332-credential-socket\") pod \"48cd2fcb-f864-4e43-bcfb-cadfd9964332\" (UID: \"48cd2fcb-f864-4e43-bcfb-cadfd9964332\") "
Sep 11 23:32:12.608480 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:12.608458    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48cd2fcb-f864-4e43-bcfb-cadfd9964332-istiod-ca-cert\") pod \"48cd2fcb-f864-4e43-bcfb-cadfd9964332\" (UID: \"48cd2fcb-f864-4e43-bcfb-cadfd9964332\") "
Sep 11 23:32:12.608813 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:12.608487    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48cd2fcb-f864-4e43-bcfb-cadfd9964332-workload-socket\") pod \"48cd2fcb-f864-4e43-bcfb-cadfd9964332\" (UID: \"48cd2fcb-f864-4e43-bcfb-cadfd9964332\") "
Sep 11 23:32:12.608813 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:12.608536    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48cd2fcb-f864-4e43-bcfb-cadfd9964332-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "48cd2fcb-f864-4e43-bcfb-cadfd9964332" (UID: "48cd2fcb-f864-4e43-bcfb-cadfd9964332"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:32:12.608929 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:12.608849    2546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48cd2fcb-f864-4e43-bcfb-cadfd9964332-workload-certs\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:32:12.608929 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:12.608885    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48cd2fcb-f864-4e43-bcfb-cadfd9964332-istio-data" (OuterVolumeSpecName: "istio-data") pod "48cd2fcb-f864-4e43-bcfb-cadfd9964332" (UID: "48cd2fcb-f864-4e43-bcfb-cadfd9964332"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:32:12.608929 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:12.608907    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48cd2fcb-f864-4e43-bcfb-cadfd9964332-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "48cd2fcb-f864-4e43-bcfb-cadfd9964332" (UID: "48cd2fcb-f864-4e43-bcfb-cadfd9964332"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 23:32:12.609108 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:12.609004    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48cd2fcb-f864-4e43-bcfb-cadfd9964332-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "48cd2fcb-f864-4e43-bcfb-cadfd9964332" (UID: "48cd2fcb-f864-4e43-bcfb-cadfd9964332"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:32:12.609294 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:12.609274    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48cd2fcb-f864-4e43-bcfb-cadfd9964332-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "48cd2fcb-f864-4e43-bcfb-cadfd9964332" (UID: "48cd2fcb-f864-4e43-bcfb-cadfd9964332"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:32:12.610774 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:12.610749    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48cd2fcb-f864-4e43-bcfb-cadfd9964332-kube-api-access-dnv58" (OuterVolumeSpecName: "kube-api-access-dnv58") pod "48cd2fcb-f864-4e43-bcfb-cadfd9964332" (UID: "48cd2fcb-f864-4e43-bcfb-cadfd9964332"). InnerVolumeSpecName "kube-api-access-dnv58". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 23:32:12.610774 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:12.610769    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/48cd2fcb-f864-4e43-bcfb-cadfd9964332-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "48cd2fcb-f864-4e43-bcfb-cadfd9964332" (UID: "48cd2fcb-f864-4e43-bcfb-cadfd9964332"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 23:32:12.611199 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:12.611178    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48cd2fcb-f864-4e43-bcfb-cadfd9964332-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "48cd2fcb-f864-4e43-bcfb-cadfd9964332" (UID: "48cd2fcb-f864-4e43-bcfb-cadfd9964332"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:32:12.611312 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:12.611290    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48cd2fcb-f864-4e43-bcfb-cadfd9964332-istio-token" (OuterVolumeSpecName: "istio-token") pod "48cd2fcb-f864-4e43-bcfb-cadfd9964332" (UID: "48cd2fcb-f864-4e43-bcfb-cadfd9964332"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 23:32:12.709737 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:12.709702    2546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48cd2fcb-f864-4e43-bcfb-cadfd9964332-istio-podinfo\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:32:12.709737 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:12.709731    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnv58\" (UniqueName: \"kubernetes.io/projected/48cd2fcb-f864-4e43-bcfb-cadfd9964332-kube-api-access-dnv58\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:32:12.709737 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:12.709743    2546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48cd2fcb-f864-4e43-bcfb-cadfd9964332-istio-envoy\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:32:12.709986 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:12.709751    2546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48cd2fcb-f864-4e43-bcfb-cadfd9964332-istio-data\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:32:12.709986 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:12.709759    2546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48cd2fcb-f864-4e43-bcfb-cadfd9964332-credential-socket\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:32:12.709986 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:12.709768    2546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48cd2fcb-f864-4e43-bcfb-cadfd9964332-istiod-ca-cert\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:32:12.709986 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:12.709778    2546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48cd2fcb-f864-4e43-bcfb-cadfd9964332-workload-socket\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:32:12.709986 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:12.709786    2546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48cd2fcb-f864-4e43-bcfb-cadfd9964332-istio-token\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:32:12.757759 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:12.757721    2546 generic.go:356] "Generic (PLEG): container finished" podID="48cd2fcb-f864-4e43-bcfb-cadfd9964332" containerID="39c40ddc5208abebb3b8b5bf03a456803ce38921ff8f48b6f8684349369088f5" exitCode=0
Sep 11 23:32:12.757930 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:12.757781    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-cd48eb-openshift-default-76cd487f58-fm5sq"
Sep 11 23:32:12.757930 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:12.757782    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-cd48eb-openshift-default-76cd487f58-fm5sq" event={"ID":"48cd2fcb-f864-4e43-bcfb-cadfd9964332","Type":"ContainerDied","Data":"39c40ddc5208abebb3b8b5bf03a456803ce38921ff8f48b6f8684349369088f5"}
Sep 11 23:32:12.758077 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:12.757952    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-cd48eb-openshift-default-76cd487f58-fm5sq" event={"ID":"48cd2fcb-f864-4e43-bcfb-cadfd9964332","Type":"ContainerDied","Data":"6e1000816275819923875ffb8ff369f45fff36d95e094f2f2b14874ded68629f"}
Sep 11 23:32:12.758077 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:12.757993    2546 scope.go:117] "RemoveContainer" containerID="39c40ddc5208abebb3b8b5bf03a456803ce38921ff8f48b6f8684349369088f5"
Sep 11 23:32:12.769325 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:12.769306    2546 scope.go:117] "RemoveContainer" containerID="e290b7f75aac5587df44419127c3cb87ba173a649c58400ae948239528fc8896"
Sep 11 23:32:12.779479 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:12.779455    2546 scope.go:117] "RemoveContainer" containerID="39c40ddc5208abebb3b8b5bf03a456803ce38921ff8f48b6f8684349369088f5"
Sep 11 23:32:12.779787 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:32:12.779767    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39c40ddc5208abebb3b8b5bf03a456803ce38921ff8f48b6f8684349369088f5\": container with ID starting with 39c40ddc5208abebb3b8b5bf03a456803ce38921ff8f48b6f8684349369088f5 not found: ID does not exist" containerID="39c40ddc5208abebb3b8b5bf03a456803ce38921ff8f48b6f8684349369088f5"
Sep 11 23:32:12.779839 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:12.779797    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39c40ddc5208abebb3b8b5bf03a456803ce38921ff8f48b6f8684349369088f5"} err="failed to get container status \"39c40ddc5208abebb3b8b5bf03a456803ce38921ff8f48b6f8684349369088f5\": rpc error: code = NotFound desc = could not find container \"39c40ddc5208abebb3b8b5bf03a456803ce38921ff8f48b6f8684349369088f5\": container with ID starting with 39c40ddc5208abebb3b8b5bf03a456803ce38921ff8f48b6f8684349369088f5 not found: ID does not exist"
Sep 11 23:32:12.779839 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:12.779814    2546 scope.go:117] "RemoveContainer" containerID="e290b7f75aac5587df44419127c3cb87ba173a649c58400ae948239528fc8896"
Sep 11 23:32:12.780048 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:32:12.780033    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e290b7f75aac5587df44419127c3cb87ba173a649c58400ae948239528fc8896\": container with ID starting with e290b7f75aac5587df44419127c3cb87ba173a649c58400ae948239528fc8896 not found: ID does not exist" containerID="e290b7f75aac5587df44419127c3cb87ba173a649c58400ae948239528fc8896"
Sep 11 23:32:12.780098 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:12.780052    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e290b7f75aac5587df44419127c3cb87ba173a649c58400ae948239528fc8896"} err="failed to get container status \"e290b7f75aac5587df44419127c3cb87ba173a649c58400ae948239528fc8896\": rpc error: code = NotFound desc = could not find container \"e290b7f75aac5587df44419127c3cb87ba173a649c58400ae948239528fc8896\": container with ID starting with e290b7f75aac5587df44419127c3cb87ba173a649c58400ae948239528fc8896 not found: ID does not exist"
Sep 11 23:32:12.785524 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:12.785502    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-cd48eb-openshift-default-76cd487f58-fm5sq"]
Sep 11 23:32:12.788687 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:12.788665    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-cd48eb-openshift-default-76cd487f58-fm5sq"]
Sep 11 23:32:14.494636 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:14.494605    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48cd2fcb-f864-4e43-bcfb-cadfd9964332" path="/var/lib/kubelet/pods/48cd2fcb-f864-4e43-bcfb-cadfd9964332/volumes"
Sep 11 23:32:16.875354 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:16.875321    2546 scope.go:117] "RemoveContainer" containerID="65c29795831866cf799de0411ba7e8ad6517b2a1e2a46eab790fd720b4639101"
Sep 11 23:32:18.367117 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:18.367084    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-97d9dc-29819490-7b6mr"]
Sep 11 23:32:18.371171 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:18.371126    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-97d9dc-29819490-7b6mr"]
Sep 11 23:32:18.501975 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:18.501931    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1076ba3-1d97-4eda-9e02-52ef3eb30ff1" path="/var/lib/kubelet/pods/b1076ba3-1d97-4eda-9e02-52ef3eb30ff1/volumes"
Sep 11 23:32:18.598299 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:18.598265    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-97d9dc-chwvd"]
Sep 11 23:32:18.602527 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:18.602501    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-97d9dc-chwvd"]
Sep 11 23:32:20.495510 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:20.495462    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2eca8b5d-ddf8-4bd3-a6e1-7cc9787499ba" path="/var/lib/kubelet/pods/2eca8b5d-ddf8-4bd3-a6e1-7cc9787499ba/volumes"
Sep 11 23:32:24.841771 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:24.841728    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-06ef79-77bfb74cfd-5mr6z"]
Sep 11 23:32:24.842283 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:24.842027    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-06ef79-77bfb74cfd-5mr6z" podUID="2f226bf5-fc00-493d-92c2-0c59e9b33dd1" containerName="payload-pre-processing" containerID="cri-o://c9f9ba0be686326d252ae9229c4ee4b263eb57c7a201a53ded15a0fe9cc114d9" gracePeriod=30
Sep 11 23:32:25.115687 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:25.115655    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-06ef79-77bfb74cfd-5mr6z"
Sep 11 23:32:25.245639 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:25.245606    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f226bf5-fc00-493d-92c2-0c59e9b33dd1-plugins-config-volume\") pod \"2f226bf5-fc00-493d-92c2-0c59e9b33dd1\" (UID: \"2f226bf5-fc00-493d-92c2-0c59e9b33dd1\") "
Sep 11 23:32:25.245818 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:25.245722    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jc79d\" (UniqueName: \"kubernetes.io/projected/2f226bf5-fc00-493d-92c2-0c59e9b33dd1-kube-api-access-jc79d\") pod \"2f226bf5-fc00-493d-92c2-0c59e9b33dd1\" (UID: \"2f226bf5-fc00-493d-92c2-0c59e9b33dd1\") "
Sep 11 23:32:25.246080 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:25.246052    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f226bf5-fc00-493d-92c2-0c59e9b33dd1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2f226bf5-fc00-493d-92c2-0c59e9b33dd1" (UID: "2f226bf5-fc00-493d-92c2-0c59e9b33dd1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 23:32:25.247981 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:25.247955    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f226bf5-fc00-493d-92c2-0c59e9b33dd1-kube-api-access-jc79d" (OuterVolumeSpecName: "kube-api-access-jc79d") pod "2f226bf5-fc00-493d-92c2-0c59e9b33dd1" (UID: "2f226bf5-fc00-493d-92c2-0c59e9b33dd1"). InnerVolumeSpecName "kube-api-access-jc79d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 23:32:25.346830 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:25.346730    2546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f226bf5-fc00-493d-92c2-0c59e9b33dd1-plugins-config-volume\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:32:25.346830 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:25.346768    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jc79d\" (UniqueName: \"kubernetes.io/projected/2f226bf5-fc00-493d-92c2-0c59e9b33dd1-kube-api-access-jc79d\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:32:25.855616 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:25.855565    2546 generic.go:356] "Generic (PLEG): container finished" podID="2f226bf5-fc00-493d-92c2-0c59e9b33dd1" containerID="c9f9ba0be686326d252ae9229c4ee4b263eb57c7a201a53ded15a0fe9cc114d9" exitCode=0
Sep 11 23:32:25.856105 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:25.855764    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-06ef79-77bfb74cfd-5mr6z" event={"ID":"2f226bf5-fc00-493d-92c2-0c59e9b33dd1","Type":"ContainerDied","Data":"c9f9ba0be686326d252ae9229c4ee4b263eb57c7a201a53ded15a0fe9cc114d9"}
Sep 11 23:32:25.856105 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:25.855827    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-06ef79-77bfb74cfd-5mr6z" event={"ID":"2f226bf5-fc00-493d-92c2-0c59e9b33dd1","Type":"ContainerDied","Data":"32ace1bbefbe65c2ed545b9ef4a38f7c45b8c5c0c1ec4779f387d9d4810ee63e"}
Sep 11 23:32:25.856105 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:25.855858    2546 scope.go:117] "RemoveContainer" containerID="c9f9ba0be686326d252ae9229c4ee4b263eb57c7a201a53ded15a0fe9cc114d9"
Sep 11 23:32:25.856105 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:25.856049    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-06ef79-77bfb74cfd-5mr6z"
Sep 11 23:32:25.869418 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:25.869393    2546 scope.go:117] "RemoveContainer" containerID="c9f9ba0be686326d252ae9229c4ee4b263eb57c7a201a53ded15a0fe9cc114d9"
Sep 11 23:32:25.869816 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:32:25.869791    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9f9ba0be686326d252ae9229c4ee4b263eb57c7a201a53ded15a0fe9cc114d9\": container with ID starting with c9f9ba0be686326d252ae9229c4ee4b263eb57c7a201a53ded15a0fe9cc114d9 not found: ID does not exist" containerID="c9f9ba0be686326d252ae9229c4ee4b263eb57c7a201a53ded15a0fe9cc114d9"
Sep 11 23:32:25.869909 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:25.869826    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9f9ba0be686326d252ae9229c4ee4b263eb57c7a201a53ded15a0fe9cc114d9"} err="failed to get container status \"c9f9ba0be686326d252ae9229c4ee4b263eb57c7a201a53ded15a0fe9cc114d9\": rpc error: code = NotFound desc = could not find container \"c9f9ba0be686326d252ae9229c4ee4b263eb57c7a201a53ded15a0fe9cc114d9\": container with ID starting with c9f9ba0be686326d252ae9229c4ee4b263eb57c7a201a53ded15a0fe9cc114d9 not found: ID does not exist"
Sep 11 23:32:25.886268 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:25.886235    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-06ef79-77bfb74cfd-5mr6z"]
Sep 11 23:32:25.888747 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:25.888724    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-06ef79-77bfb74cfd-5mr6z"]
Sep 11 23:32:26.043870 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:26.043830    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-97d9dc-openshift-default-7bfd65cf68-sltcx"]
Sep 11 23:32:26.044116 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:26.044090    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-97d9dc-openshift-default-7bfd65cf68-sltcx" podUID="44b0b3b3-a51c-42bf-8539-f11924e397db" containerName="istio-proxy" containerID="cri-o://d6b7df72abe48166ace8a9ce28eb0b3c44e83156bbf1f3d5ead3ae1052840e7d" gracePeriod=30
Sep 11 23:32:26.494635 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:26.494600    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f226bf5-fc00-493d-92c2-0c59e9b33dd1" path="/var/lib/kubelet/pods/2f226bf5-fc00-493d-92c2-0c59e9b33dd1/volumes"
Sep 11 23:32:27.305186 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:27.305138    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"]
Sep 11 23:32:31.309222 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:31.309144    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-97d9dc-openshift-default-7bfd65cf68-sltcx"
Sep 11 23:32:31.413430 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:31.413399    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44b0b3b3-a51c-42bf-8539-f11924e397db-istio-data\") pod \"44b0b3b3-a51c-42bf-8539-f11924e397db\" (UID: \"44b0b3b3-a51c-42bf-8539-f11924e397db\") "
Sep 11 23:32:31.413617 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:31.413478    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44b0b3b3-a51c-42bf-8539-f11924e397db-credential-socket\") pod \"44b0b3b3-a51c-42bf-8539-f11924e397db\" (UID: \"44b0b3b3-a51c-42bf-8539-f11924e397db\") "
Sep 11 23:32:31.413617 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:31.413515    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44b0b3b3-a51c-42bf-8539-f11924e397db-istio-podinfo\") pod \"44b0b3b3-a51c-42bf-8539-f11924e397db\" (UID: \"44b0b3b3-a51c-42bf-8539-f11924e397db\") "
Sep 11 23:32:31.413617 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:31.413534    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44b0b3b3-a51c-42bf-8539-f11924e397db-workload-certs\") pod \"44b0b3b3-a51c-42bf-8539-f11924e397db\" (UID: \"44b0b3b3-a51c-42bf-8539-f11924e397db\") "
Sep 11 23:32:31.413617 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:31.413572    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44b0b3b3-a51c-42bf-8539-f11924e397db-istiod-ca-cert\") pod \"44b0b3b3-a51c-42bf-8539-f11924e397db\" (UID: \"44b0b3b3-a51c-42bf-8539-f11924e397db\") "
Sep 11 23:32:31.413617 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:31.413612    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44b0b3b3-a51c-42bf-8539-f11924e397db-workload-socket\") pod \"44b0b3b3-a51c-42bf-8539-f11924e397db\" (UID: \"44b0b3b3-a51c-42bf-8539-f11924e397db\") "
Sep 11 23:32:31.413947 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:31.413633    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44b0b3b3-a51c-42bf-8539-f11924e397db-istio-envoy\") pod \"44b0b3b3-a51c-42bf-8539-f11924e397db\" (UID: \"44b0b3b3-a51c-42bf-8539-f11924e397db\") "
Sep 11 23:32:31.413947 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:31.413686    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h4tgf\" (UniqueName: \"kubernetes.io/projected/44b0b3b3-a51c-42bf-8539-f11924e397db-kube-api-access-h4tgf\") pod \"44b0b3b3-a51c-42bf-8539-f11924e397db\" (UID: \"44b0b3b3-a51c-42bf-8539-f11924e397db\") "
Sep 11 23:32:31.413947 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:31.413740    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44b0b3b3-a51c-42bf-8539-f11924e397db-istio-data" (OuterVolumeSpecName: "istio-data") pod "44b0b3b3-a51c-42bf-8539-f11924e397db" (UID: "44b0b3b3-a51c-42bf-8539-f11924e397db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:32:31.413947 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:31.413751    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44b0b3b3-a51c-42bf-8539-f11924e397db-istio-token\") pod \"44b0b3b3-a51c-42bf-8539-f11924e397db\" (UID: \"44b0b3b3-a51c-42bf-8539-f11924e397db\") "
Sep 11 23:32:31.414195 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:31.413978    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44b0b3b3-a51c-42bf-8539-f11924e397db-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "44b0b3b3-a51c-42bf-8539-f11924e397db" (UID: "44b0b3b3-a51c-42bf-8539-f11924e397db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:32:31.414195 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:31.414007    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44b0b3b3-a51c-42bf-8539-f11924e397db-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "44b0b3b3-a51c-42bf-8539-f11924e397db" (UID: "44b0b3b3-a51c-42bf-8539-f11924e397db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:32:31.414195 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:31.414097    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44b0b3b3-a51c-42bf-8539-f11924e397db-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "44b0b3b3-a51c-42bf-8539-f11924e397db" (UID: "44b0b3b3-a51c-42bf-8539-f11924e397db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:32:31.414368 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:31.414273    2546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44b0b3b3-a51c-42bf-8539-f11924e397db-istio-data\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:32:31.414368 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:31.414292    2546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44b0b3b3-a51c-42bf-8539-f11924e397db-credential-socket\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:32:31.414368 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:31.414306    2546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44b0b3b3-a51c-42bf-8539-f11924e397db-workload-certs\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:32:31.414368 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:31.414320    2546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44b0b3b3-a51c-42bf-8539-f11924e397db-workload-socket\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:32:31.414526 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:31.414390    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44b0b3b3-a51c-42bf-8539-f11924e397db-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "44b0b3b3-a51c-42bf-8539-f11924e397db" (UID: "44b0b3b3-a51c-42bf-8539-f11924e397db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 23:32:31.416106 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:31.416075    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44b0b3b3-a51c-42bf-8539-f11924e397db-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "44b0b3b3-a51c-42bf-8539-f11924e397db" (UID: "44b0b3b3-a51c-42bf-8539-f11924e397db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:32:31.416233 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:31.416113    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44b0b3b3-a51c-42bf-8539-f11924e397db-istio-token" (OuterVolumeSpecName: "istio-token") pod "44b0b3b3-a51c-42bf-8539-f11924e397db" (UID: "44b0b3b3-a51c-42bf-8539-f11924e397db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 23:32:31.416526 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:31.416506    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/44b0b3b3-a51c-42bf-8539-f11924e397db-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "44b0b3b3-a51c-42bf-8539-f11924e397db" (UID: "44b0b3b3-a51c-42bf-8539-f11924e397db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 23:32:31.416599 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:31.416581    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44b0b3b3-a51c-42bf-8539-f11924e397db-kube-api-access-h4tgf" (OuterVolumeSpecName: "kube-api-access-h4tgf") pod "44b0b3b3-a51c-42bf-8539-f11924e397db" (UID: "44b0b3b3-a51c-42bf-8539-f11924e397db"). InnerVolumeSpecName "kube-api-access-h4tgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 23:32:31.515471 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:31.515433    2546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44b0b3b3-a51c-42bf-8539-f11924e397db-istio-podinfo\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:32:31.515471 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:31.515469    2546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44b0b3b3-a51c-42bf-8539-f11924e397db-istiod-ca-cert\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:32:31.515471 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:31.515481    2546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44b0b3b3-a51c-42bf-8539-f11924e397db-istio-envoy\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:32:31.515724 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:31.515495    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4tgf\" (UniqueName: \"kubernetes.io/projected/44b0b3b3-a51c-42bf-8539-f11924e397db-kube-api-access-h4tgf\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:32:31.515724 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:31.515508    2546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44b0b3b3-a51c-42bf-8539-f11924e397db-istio-token\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:32:31.901821 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:31.901778    2546 generic.go:356] "Generic (PLEG): container finished" podID="44b0b3b3-a51c-42bf-8539-f11924e397db" containerID="d6b7df72abe48166ace8a9ce28eb0b3c44e83156bbf1f3d5ead3ae1052840e7d" exitCode=0
Sep 11 23:32:31.901988 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:31.901817    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-97d9dc-openshift-default-7bfd65cf68-sltcx" event={"ID":"44b0b3b3-a51c-42bf-8539-f11924e397db","Type":"ContainerDied","Data":"d6b7df72abe48166ace8a9ce28eb0b3c44e83156bbf1f3d5ead3ae1052840e7d"}
Sep 11 23:32:31.901988 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:31.901874    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-97d9dc-openshift-default-7bfd65cf68-sltcx"
Sep 11 23:32:31.901988 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:31.901912    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-97d9dc-openshift-default-7bfd65cf68-sltcx" event={"ID":"44b0b3b3-a51c-42bf-8539-f11924e397db","Type":"ContainerDied","Data":"f26e24eea4d0b8274f5dee70967b0edde9420ad812e09d83e7e0aaeae1aa12fd"}
Sep 11 23:32:31.901988 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:31.901943    2546 scope.go:117] "RemoveContainer" containerID="d6b7df72abe48166ace8a9ce28eb0b3c44e83156bbf1f3d5ead3ae1052840e7d"
Sep 11 23:32:31.915408 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:31.914444    2546 scope.go:117] "RemoveContainer" containerID="c9220478099430b9b95d3cc848e6d6b52138fa1f52c837570c992c51b53ee435"
Sep 11 23:32:31.928353 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:31.927539    2546 scope.go:117] "RemoveContainer" containerID="d6b7df72abe48166ace8a9ce28eb0b3c44e83156bbf1f3d5ead3ae1052840e7d"
Sep 11 23:32:31.929449 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:32:31.929414    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6b7df72abe48166ace8a9ce28eb0b3c44e83156bbf1f3d5ead3ae1052840e7d\": container with ID starting with d6b7df72abe48166ace8a9ce28eb0b3c44e83156bbf1f3d5ead3ae1052840e7d not found: ID does not exist" containerID="d6b7df72abe48166ace8a9ce28eb0b3c44e83156bbf1f3d5ead3ae1052840e7d"
Sep 11 23:32:31.929576 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:31.929460    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6b7df72abe48166ace8a9ce28eb0b3c44e83156bbf1f3d5ead3ae1052840e7d"} err="failed to get container status \"d6b7df72abe48166ace8a9ce28eb0b3c44e83156bbf1f3d5ead3ae1052840e7d\": rpc error: code = NotFound desc = could not find container \"d6b7df72abe48166ace8a9ce28eb0b3c44e83156bbf1f3d5ead3ae1052840e7d\": container with ID starting with d6b7df72abe48166ace8a9ce28eb0b3c44e83156bbf1f3d5ead3ae1052840e7d not found: ID does not exist"
Sep 11 23:32:31.929576 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:31.929486    2546 scope.go:117] "RemoveContainer" containerID="c9220478099430b9b95d3cc848e6d6b52138fa1f52c837570c992c51b53ee435"
Sep 11 23:32:31.931479 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:32:31.931450    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9220478099430b9b95d3cc848e6d6b52138fa1f52c837570c992c51b53ee435\": container with ID starting with c9220478099430b9b95d3cc848e6d6b52138fa1f52c837570c992c51b53ee435 not found: ID does not exist" containerID="c9220478099430b9b95d3cc848e6d6b52138fa1f52c837570c992c51b53ee435"
Sep 11 23:32:31.931593 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:31.931489    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9220478099430b9b95d3cc848e6d6b52138fa1f52c837570c992c51b53ee435"} err="failed to get container status \"c9220478099430b9b95d3cc848e6d6b52138fa1f52c837570c992c51b53ee435\": rpc error: code = NotFound desc = could not find container \"c9220478099430b9b95d3cc848e6d6b52138fa1f52c837570c992c51b53ee435\": container with ID starting with c9220478099430b9b95d3cc848e6d6b52138fa1f52c837570c992c51b53ee435 not found: ID does not exist"
Sep 11 23:32:31.940828 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:31.940799    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-97d9dc-openshift-default-7bfd65cf68-sltcx"]
Sep 11 23:32:31.944333 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:31.944310    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-97d9dc-openshift-default-7bfd65cf68-sltcx"]
Sep 11 23:32:32.495929 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:32.495894    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44b0b3b3-a51c-42bf-8539-f11924e397db" path="/var/lib/kubelet/pods/44b0b3b3-a51c-42bf-8539-f11924e397db/volumes"
Sep 11 23:32:46.998847 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:32:46.998809    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"]
Sep 11 23:33:15.003234 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:33:15.003190    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"]
Sep 11 23:33:45.237732 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:33:45.237690    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-da9451/sub-test-model-da9451-kserve-5cb9fd6dc9-spj88"]
Sep 11 23:33:45.238356 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:33:45.238337    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f226bf5-fc00-493d-92c2-0c59e9b33dd1" containerName="payload-pre-processing"
Sep 11 23:33:45.238409 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:33:45.238359    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f226bf5-fc00-493d-92c2-0c59e9b33dd1" containerName="payload-pre-processing"
Sep 11 23:33:45.238409 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:33:45.238375    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48cd2fcb-f864-4e43-bcfb-cadfd9964332" containerName="istio-proxy"
Sep 11 23:33:45.238409 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:33:45.238381    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="48cd2fcb-f864-4e43-bcfb-cadfd9964332" containerName="istio-proxy"
Sep 11 23:33:45.238409 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:33:45.238395    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44b0b3b3-a51c-42bf-8539-f11924e397db" containerName="istio-proxy"
Sep 11 23:33:45.238409 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:33:45.238402    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="44b0b3b3-a51c-42bf-8539-f11924e397db" containerName="istio-proxy"
Sep 11 23:33:45.238569 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:33:45.238415    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2eca8b5d-ddf8-4bd3-a6e1-7cc9787499ba" containerName="revoke-keys"
Sep 11 23:33:45.238569 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:33:45.238420    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="2eca8b5d-ddf8-4bd3-a6e1-7cc9787499ba" containerName="revoke-keys"
Sep 11 23:33:45.238569 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:33:45.238426    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44b0b3b3-a51c-42bf-8539-f11924e397db" containerName="istio-proxy"
Sep 11 23:33:45.238569 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:33:45.238431    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="44b0b3b3-a51c-42bf-8539-f11924e397db" containerName="istio-proxy"
Sep 11 23:33:45.238569 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:33:45.238497    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="44b0b3b3-a51c-42bf-8539-f11924e397db" containerName="istio-proxy"
Sep 11 23:33:45.238569 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:33:45.238504    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f226bf5-fc00-493d-92c2-0c59e9b33dd1" containerName="payload-pre-processing"
Sep 11 23:33:45.238569 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:33:45.238513    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="2eca8b5d-ddf8-4bd3-a6e1-7cc9787499ba" containerName="revoke-keys"
Sep 11 23:33:45.238569 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:33:45.238518    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="48cd2fcb-f864-4e43-bcfb-cadfd9964332" containerName="istio-proxy"
Sep 11 23:33:45.238569 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:33:45.238523    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="48cd2fcb-f864-4e43-bcfb-cadfd9964332" containerName="istio-proxy"
Sep 11 23:33:45.238569 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:33:45.238529    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="181669c0-2cfb-463e-9dd2-3c4c874b8442" containerName="istio-proxy"
Sep 11 23:33:45.241772 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:33:45.241754    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-da9451/sub-test-model-da9451-kserve-5cb9fd6dc9-spj88"
Sep 11 23:33:45.246785 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:33:45.246755    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-da9451\"/\"default-dockercfg-8dm8j\""
Sep 11 23:33:45.246785 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:33:45.246779    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-da9451\"/\"kube-root-ca.crt\""
Sep 11 23:33:45.247010 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:33:45.246818    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-da9451\"/\"sub-test-model-da9451-kserve-self-signed-certs\""
Sep 11 23:33:45.247010 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:33:45.246755    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-da9451\"/\"openshift-service-ca.crt\""
Sep 11 23:33:45.253616 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:33:45.253591    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-da9451/sub-test-model-da9451-kserve-5cb9fd6dc9-spj88"]
Sep 11 23:33:45.336705 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:33:45.336661    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b84bd2b-5f55-4393-8006-dab338dd281f-tls-certs\") pod \"sub-test-model-da9451-kserve-5cb9fd6dc9-spj88\" (UID: \"9b84bd2b-5f55-4393-8006-dab338dd281f\") " pod="ai-tenant-e2e-shared-b-w3-da9451/sub-test-model-da9451-kserve-5cb9fd6dc9-spj88"
Sep 11 23:33:45.336705 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:33:45.336709    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b84bd2b-5f55-4393-8006-dab338dd281f-model-cache\") pod \"sub-test-model-da9451-kserve-5cb9fd6dc9-spj88\" (UID: \"9b84bd2b-5f55-4393-8006-dab338dd281f\") " pod="ai-tenant-e2e-shared-b-w3-da9451/sub-test-model-da9451-kserve-5cb9fd6dc9-spj88"
Sep 11 23:33:45.336926 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:33:45.336745    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b84bd2b-5f55-4393-8006-dab338dd281f-home\") pod \"sub-test-model-da9451-kserve-5cb9fd6dc9-spj88\" (UID: \"9b84bd2b-5f55-4393-8006-dab338dd281f\") " pod="ai-tenant-e2e-shared-b-w3-da9451/sub-test-model-da9451-kserve-5cb9fd6dc9-spj88"
Sep 11 23:33:45.336926 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:33:45.336837    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b84bd2b-5f55-4393-8006-dab338dd281f-tmp-dir\") pod \"sub-test-model-da9451-kserve-5cb9fd6dc9-spj88\" (UID: \"9b84bd2b-5f55-4393-8006-dab338dd281f\") " pod="ai-tenant-e2e-shared-b-w3-da9451/sub-test-model-da9451-kserve-5cb9fd6dc9-spj88"
Sep 11 23:33:45.336926 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:33:45.336871    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b84bd2b-5f55-4393-8006-dab338dd281f-dshm\") pod \"sub-test-model-da9451-kserve-5cb9fd6dc9-spj88\" (UID: \"9b84bd2b-5f55-4393-8006-dab338dd281f\") " pod="ai-tenant-e2e-shared-b-w3-da9451/sub-test-model-da9451-kserve-5cb9fd6dc9-spj88"
Sep 11 23:33:45.336926 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:33:45.336914    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwpmr\" (UniqueName: \"kubernetes.io/projected/9b84bd2b-5f55-4393-8006-dab338dd281f-kube-api-access-vwpmr\") pod \"sub-test-model-da9451-kserve-5cb9fd6dc9-spj88\" (UID: \"9b84bd2b-5f55-4393-8006-dab338dd281f\") " pod="ai-tenant-e2e-shared-b-w3-da9451/sub-test-model-da9451-kserve-5cb9fd6dc9-spj88"
Sep 11 23:33:45.438612 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:33:45.438563    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b84bd2b-5f55-4393-8006-dab338dd281f-tls-certs\") pod \"sub-test-model-da9451-kserve-5cb9fd6dc9-spj88\" (UID: \"9b84bd2b-5f55-4393-8006-dab338dd281f\") " pod="ai-tenant-e2e-shared-b-w3-da9451/sub-test-model-da9451-kserve-5cb9fd6dc9-spj88"
Sep 11 23:33:45.438813 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:33:45.438625    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b84bd2b-5f55-4393-8006-dab338dd281f-model-cache\") pod \"sub-test-model-da9451-kserve-5cb9fd6dc9-spj88\" (UID: \"9b84bd2b-5f55-4393-8006-dab338dd281f\") " pod="ai-tenant-e2e-shared-b-w3-da9451/sub-test-model-da9451-kserve-5cb9fd6dc9-spj88"
Sep 11 23:33:45.438813 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:33:45.438672    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b84bd2b-5f55-4393-8006-dab338dd281f-home\") pod \"sub-test-model-da9451-kserve-5cb9fd6dc9-spj88\" (UID: \"9b84bd2b-5f55-4393-8006-dab338dd281f\") " pod="ai-tenant-e2e-shared-b-w3-da9451/sub-test-model-da9451-kserve-5cb9fd6dc9-spj88"
Sep 11 23:33:45.438813 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:33:45.438704    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b84bd2b-5f55-4393-8006-dab338dd281f-tmp-dir\") pod \"sub-test-model-da9451-kserve-5cb9fd6dc9-spj88\" (UID: \"9b84bd2b-5f55-4393-8006-dab338dd281f\") " pod="ai-tenant-e2e-shared-b-w3-da9451/sub-test-model-da9451-kserve-5cb9fd6dc9-spj88"
Sep 11 23:33:45.438813 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:33:45.438732    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b84bd2b-5f55-4393-8006-dab338dd281f-dshm\") pod \"sub-test-model-da9451-kserve-5cb9fd6dc9-spj88\" (UID: \"9b84bd2b-5f55-4393-8006-dab338dd281f\") " pod="ai-tenant-e2e-shared-b-w3-da9451/sub-test-model-da9451-kserve-5cb9fd6dc9-spj88"
Sep 11 23:33:45.438813 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:33:45.438792    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwpmr\" (UniqueName: \"kubernetes.io/projected/9b84bd2b-5f55-4393-8006-dab338dd281f-kube-api-access-vwpmr\") pod \"sub-test-model-da9451-kserve-5cb9fd6dc9-spj88\" (UID: \"9b84bd2b-5f55-4393-8006-dab338dd281f\") " pod="ai-tenant-e2e-shared-b-w3-da9451/sub-test-model-da9451-kserve-5cb9fd6dc9-spj88"
Sep 11 23:33:45.439515 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:33:45.439483    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b84bd2b-5f55-4393-8006-dab338dd281f-home\") pod \"sub-test-model-da9451-kserve-5cb9fd6dc9-spj88\" (UID: \"9b84bd2b-5f55-4393-8006-dab338dd281f\") " pod="ai-tenant-e2e-shared-b-w3-da9451/sub-test-model-da9451-kserve-5cb9fd6dc9-spj88"
Sep 11 23:33:45.439643 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:33:45.439590    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b84bd2b-5f55-4393-8006-dab338dd281f-tmp-dir\") pod \"sub-test-model-da9451-kserve-5cb9fd6dc9-spj88\" (UID: \"9b84bd2b-5f55-4393-8006-dab338dd281f\") " pod="ai-tenant-e2e-shared-b-w3-da9451/sub-test-model-da9451-kserve-5cb9fd6dc9-spj88"
Sep 11 23:33:45.439728 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:33:45.439711    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b84bd2b-5f55-4393-8006-dab338dd281f-model-cache\") pod \"sub-test-model-da9451-kserve-5cb9fd6dc9-spj88\" (UID: \"9b84bd2b-5f55-4393-8006-dab338dd281f\") " pod="ai-tenant-e2e-shared-b-w3-da9451/sub-test-model-da9451-kserve-5cb9fd6dc9-spj88"
Sep 11 23:33:45.442448 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:33:45.442414    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b84bd2b-5f55-4393-8006-dab338dd281f-tls-certs\") pod \"sub-test-model-da9451-kserve-5cb9fd6dc9-spj88\" (UID: \"9b84bd2b-5f55-4393-8006-dab338dd281f\") " pod="ai-tenant-e2e-shared-b-w3-da9451/sub-test-model-da9451-kserve-5cb9fd6dc9-spj88"
Sep 11 23:33:45.443587 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:33:45.443558    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b84bd2b-5f55-4393-8006-dab338dd281f-dshm\") pod \"sub-test-model-da9451-kserve-5cb9fd6dc9-spj88\" (UID: \"9b84bd2b-5f55-4393-8006-dab338dd281f\") " pod="ai-tenant-e2e-shared-b-w3-da9451/sub-test-model-da9451-kserve-5cb9fd6dc9-spj88"
Sep 11 23:33:45.450444 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:33:45.450382    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwpmr\" (UniqueName: \"kubernetes.io/projected/9b84bd2b-5f55-4393-8006-dab338dd281f-kube-api-access-vwpmr\") pod \"sub-test-model-da9451-kserve-5cb9fd6dc9-spj88\" (UID: \"9b84bd2b-5f55-4393-8006-dab338dd281f\") " pod="ai-tenant-e2e-shared-b-w3-da9451/sub-test-model-da9451-kserve-5cb9fd6dc9-spj88"
Sep 11 23:33:45.553121 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:33:45.553016    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-da9451/sub-test-model-da9451-kserve-5cb9fd6dc9-spj88"
Sep 11 23:33:45.834718 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:33:45.834687    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-da9451/sub-test-model-da9451-kserve-5cb9fd6dc9-spj88"]
Sep 11 23:33:45.836135 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:33:45.836103    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9b84bd2b_5f55_4393_8006_dab338dd281f.slice/crio-879d0998d59376a817f3b3d8098cebe4ae45659d5253cc4a22c87324d9b089f9 WatchSource:0}: Error finding container 879d0998d59376a817f3b3d8098cebe4ae45659d5253cc4a22c87324d9b089f9: Status 404 returned error can't find the container with id 879d0998d59376a817f3b3d8098cebe4ae45659d5253cc4a22c87324d9b089f9
Sep 11 23:33:46.410329 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:33:46.410292    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-da9451/sub-test-model-da9451-kserve-5cb9fd6dc9-spj88" event={"ID":"9b84bd2b-5f55-4393-8006-dab338dd281f","Type":"ContainerStarted","Data":"cc070618494c3b40cdcd43d0dcb33605ff2416f676da9ce4c8abc122658f3a1e"}
Sep 11 23:33:46.410329 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:33:46.410344    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-da9451/sub-test-model-da9451-kserve-5cb9fd6dc9-spj88" event={"ID":"9b84bd2b-5f55-4393-8006-dab338dd281f","Type":"ContainerStarted","Data":"879d0998d59376a817f3b3d8098cebe4ae45659d5253cc4a22c87324d9b089f9"}
Sep 11 23:33:46.432306 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:33:46.432247    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-da9451/sub-test-model-da9451-kserve-5cb9fd6dc9-spj88" podStartSLOduration=1.4322286530000001 podStartE2EDuration="1.432228653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 23:33:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:33:46.428732218 +0000 UTC m=+1950.461861439" watchObservedRunningTime="2026-09-11 23:33:46.432228653 +0000 UTC m=+1950.465357888"
Sep 11 23:33:49.600832 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:33:49.600796    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"]
Sep 11 23:33:55.553269 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:33:55.553225    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-da9451/sub-test-model-da9451-kserve-5cb9fd6dc9-spj88"
Sep 11 23:33:55.553681 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:33:55.553325    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-da9451/sub-test-model-da9451-kserve-5cb9fd6dc9-spj88"
Sep 11 23:33:55.566651 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:33:55.566622    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-da9451/sub-test-model-da9451-kserve-5cb9fd6dc9-spj88"
Sep 11 23:33:56.496104 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:33:56.496072    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-da9451/sub-test-model-da9451-kserve-5cb9fd6dc9-spj88"
Sep 11 23:34:03.502737 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:34:03.502698    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"]
Sep 11 23:34:16.947201 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:34:16.947104    2546 scope.go:117] "RemoveContainer" containerID="78ce1543cc85dcb8ae5cbdd1308a2f714ba8f6755fc4437dac7bb881f16edc6e"
Sep 11 23:34:19.642977 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:34:19.642935    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"]
Sep 11 23:34:22.054678 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:34:22.054642    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-da9451/sub-test-model-da9451-kserve-5cb9fd6dc9-spj88"]
Sep 11 23:34:31.195207 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:34:31.195167    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"]
Sep 11 23:34:37.074625 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:34:37.074556    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-da9451/sub-test-model-da9451-kserve-5cb9fd6dc9-spj88" podUID="9b84bd2b-5f55-4393-8006-dab338dd281f" containerName="main" containerID="cri-o://cc070618494c3b40cdcd43d0dcb33605ff2416f676da9ce4c8abc122658f3a1e" gracePeriod=45
Sep 11 23:34:37.323670 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:34:37.323643    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-da9451/sub-test-model-da9451-kserve-5cb9fd6dc9-spj88"
Sep 11 23:34:37.373779 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:34:37.373690    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b84bd2b-5f55-4393-8006-dab338dd281f-dshm\") pod \"9b84bd2b-5f55-4393-8006-dab338dd281f\" (UID: \"9b84bd2b-5f55-4393-8006-dab338dd281f\") "
Sep 11 23:34:37.373779 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:34:37.373743    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b84bd2b-5f55-4393-8006-dab338dd281f-tls-certs\") pod \"9b84bd2b-5f55-4393-8006-dab338dd281f\" (UID: \"9b84bd2b-5f55-4393-8006-dab338dd281f\") "
Sep 11 23:34:37.373779 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:34:37.373770    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b84bd2b-5f55-4393-8006-dab338dd281f-tmp-dir\") pod \"9b84bd2b-5f55-4393-8006-dab338dd281f\" (UID: \"9b84bd2b-5f55-4393-8006-dab338dd281f\") "
Sep 11 23:34:37.374061 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:34:37.373813    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b84bd2b-5f55-4393-8006-dab338dd281f-model-cache\") pod \"9b84bd2b-5f55-4393-8006-dab338dd281f\" (UID: \"9b84bd2b-5f55-4393-8006-dab338dd281f\") "
Sep 11 23:34:37.374061 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:34:37.373854    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vwpmr\" (UniqueName: \"kubernetes.io/projected/9b84bd2b-5f55-4393-8006-dab338dd281f-kube-api-access-vwpmr\") pod \"9b84bd2b-5f55-4393-8006-dab338dd281f\" (UID: \"9b84bd2b-5f55-4393-8006-dab338dd281f\") "
Sep 11 23:34:37.374061 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:34:37.373887    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b84bd2b-5f55-4393-8006-dab338dd281f-home\") pod \"9b84bd2b-5f55-4393-8006-dab338dd281f\" (UID: \"9b84bd2b-5f55-4393-8006-dab338dd281f\") "
Sep 11 23:34:37.374240 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:34:37.374096    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b84bd2b-5f55-4393-8006-dab338dd281f-model-cache" (OuterVolumeSpecName: "model-cache") pod "9b84bd2b-5f55-4393-8006-dab338dd281f" (UID: "9b84bd2b-5f55-4393-8006-dab338dd281f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:34:37.374240 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:34:37.374107    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b84bd2b-5f55-4393-8006-dab338dd281f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9b84bd2b-5f55-4393-8006-dab338dd281f" (UID: "9b84bd2b-5f55-4393-8006-dab338dd281f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:34:37.374240 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:34:37.374226    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b84bd2b-5f55-4393-8006-dab338dd281f-home" (OuterVolumeSpecName: "home") pod "9b84bd2b-5f55-4393-8006-dab338dd281f" (UID: "9b84bd2b-5f55-4393-8006-dab338dd281f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:34:37.376278 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:34:37.376257    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b84bd2b-5f55-4393-8006-dab338dd281f-kube-api-access-vwpmr" (OuterVolumeSpecName: "kube-api-access-vwpmr") pod "9b84bd2b-5f55-4393-8006-dab338dd281f" (UID: "9b84bd2b-5f55-4393-8006-dab338dd281f"). InnerVolumeSpecName "kube-api-access-vwpmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 23:34:37.376519 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:34:37.376491    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b84bd2b-5f55-4393-8006-dab338dd281f-dshm" (OuterVolumeSpecName: "dshm") pod "9b84bd2b-5f55-4393-8006-dab338dd281f" (UID: "9b84bd2b-5f55-4393-8006-dab338dd281f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:34:37.376519 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:34:37.376512    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b84bd2b-5f55-4393-8006-dab338dd281f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9b84bd2b-5f55-4393-8006-dab338dd281f" (UID: "9b84bd2b-5f55-4393-8006-dab338dd281f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 23:34:37.475452 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:34:37.475410    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwpmr\" (UniqueName: \"kubernetes.io/projected/9b84bd2b-5f55-4393-8006-dab338dd281f-kube-api-access-vwpmr\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:34:37.475452 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:34:37.475449    2546 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b84bd2b-5f55-4393-8006-dab338dd281f-home\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:34:37.475452 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:34:37.475459    2546 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b84bd2b-5f55-4393-8006-dab338dd281f-dshm\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:34:37.475708 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:34:37.475467    2546 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b84bd2b-5f55-4393-8006-dab338dd281f-tls-certs\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:34:37.475708 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:34:37.475476    2546 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b84bd2b-5f55-4393-8006-dab338dd281f-tmp-dir\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:34:37.475708 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:34:37.475485    2546 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b84bd2b-5f55-4393-8006-dab338dd281f-model-cache\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:34:37.756597 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:34:37.756557    2546 generic.go:356] "Generic (PLEG): container finished" podID="9b84bd2b-5f55-4393-8006-dab338dd281f" containerID="cc070618494c3b40cdcd43d0dcb33605ff2416f676da9ce4c8abc122658f3a1e" exitCode=0
Sep 11 23:34:37.756778 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:34:37.756608    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-da9451/sub-test-model-da9451-kserve-5cb9fd6dc9-spj88"
Sep 11 23:34:37.756778 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:34:37.756615    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-da9451/sub-test-model-da9451-kserve-5cb9fd6dc9-spj88" event={"ID":"9b84bd2b-5f55-4393-8006-dab338dd281f","Type":"ContainerDied","Data":"cc070618494c3b40cdcd43d0dcb33605ff2416f676da9ce4c8abc122658f3a1e"}
Sep 11 23:34:37.756778 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:34:37.756654    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-da9451/sub-test-model-da9451-kserve-5cb9fd6dc9-spj88" event={"ID":"9b84bd2b-5f55-4393-8006-dab338dd281f","Type":"ContainerDied","Data":"879d0998d59376a817f3b3d8098cebe4ae45659d5253cc4a22c87324d9b089f9"}
Sep 11 23:34:37.756778 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:34:37.756677    2546 scope.go:117] "RemoveContainer" containerID="cc070618494c3b40cdcd43d0dcb33605ff2416f676da9ce4c8abc122658f3a1e"
Sep 11 23:34:37.767611 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:34:37.767589    2546 scope.go:117] "RemoveContainer" containerID="cc070618494c3b40cdcd43d0dcb33605ff2416f676da9ce4c8abc122658f3a1e"
Sep 11 23:34:37.767917 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:34:37.767898    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc070618494c3b40cdcd43d0dcb33605ff2416f676da9ce4c8abc122658f3a1e\": container with ID starting with cc070618494c3b40cdcd43d0dcb33605ff2416f676da9ce4c8abc122658f3a1e not found: ID does not exist" containerID="cc070618494c3b40cdcd43d0dcb33605ff2416f676da9ce4c8abc122658f3a1e"
Sep 11 23:34:37.767956 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:34:37.767927    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc070618494c3b40cdcd43d0dcb33605ff2416f676da9ce4c8abc122658f3a1e"} err="failed to get container status \"cc070618494c3b40cdcd43d0dcb33605ff2416f676da9ce4c8abc122658f3a1e\": rpc error: code = NotFound desc = could not find container \"cc070618494c3b40cdcd43d0dcb33605ff2416f676da9ce4c8abc122658f3a1e\": container with ID starting with cc070618494c3b40cdcd43d0dcb33605ff2416f676da9ce4c8abc122658f3a1e not found: ID does not exist"
Sep 11 23:34:37.781340 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:34:37.781310    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-da9451/sub-test-model-da9451-kserve-5cb9fd6dc9-spj88"]
Sep 11 23:34:37.785533 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:34:37.785506    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-da9451/sub-test-model-da9451-kserve-5cb9fd6dc9-spj88"]
Sep 11 23:34:38.494847 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:34:38.494811    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b84bd2b-5f55-4393-8006-dab338dd281f" path="/var/lib/kubelet/pods/9b84bd2b-5f55-4393-8006-dab338dd281f/volumes"
Sep 11 23:34:42.400286 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:34:42.400246    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"]
Sep 11 23:35:14.832276 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:35:14.832232    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-da9451/rate-test-model-da9451-kserve-6864b69cc5-th8hg"]
Sep 11 23:35:14.832824 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:35:14.832750    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48cd2fcb-f864-4e43-bcfb-cadfd9964332" containerName="istio-proxy"
Sep 11 23:35:14.832824 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:35:14.832765    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="48cd2fcb-f864-4e43-bcfb-cadfd9964332" containerName="istio-proxy"
Sep 11 23:35:14.832824 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:35:14.832780    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b84bd2b-5f55-4393-8006-dab338dd281f" containerName="main"
Sep 11 23:35:14.832824 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:35:14.832785    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b84bd2b-5f55-4393-8006-dab338dd281f" containerName="main"
Sep 11 23:35:14.833067 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:35:14.832851    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="44b0b3b3-a51c-42bf-8539-f11924e397db" containerName="istio-proxy"
Sep 11 23:35:14.833067 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:35:14.832861    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b84bd2b-5f55-4393-8006-dab338dd281f" containerName="main"
Sep 11 23:35:14.836275 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:35:14.836251    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-da9451/rate-test-model-da9451-kserve-6864b69cc5-th8hg"
Sep 11 23:35:14.840986 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:35:14.840862    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-da9451\"/\"rate-test-model-da9451-kserve-self-signed-certs\""
Sep 11 23:35:14.840986 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:35:14.840908    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-da9451\"/\"openshift-service-ca.crt\""
Sep 11 23:35:14.840986 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:35:14.840916    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0e2a488-50b3-4f87-a09d-a7317e80c5f7-model-cache\") pod \"rate-test-model-da9451-kserve-6864b69cc5-th8hg\" (UID: \"a0e2a488-50b3-4f87-a09d-a7317e80c5f7\") " pod="ai-tenant-e2e-shared-b-w3-da9451/rate-test-model-da9451-kserve-6864b69cc5-th8hg"
Sep 11 23:35:14.840986 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:35:14.840934    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-da9451\"/\"default-dockercfg-8dm8j\""
Sep 11 23:35:14.840986 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:35:14.840941    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-da9451\"/\"kube-root-ca.crt\""
Sep 11 23:35:14.840986 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:35:14.840981    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0e2a488-50b3-4f87-a09d-a7317e80c5f7-dshm\") pod \"rate-test-model-da9451-kserve-6864b69cc5-th8hg\" (UID: \"a0e2a488-50b3-4f87-a09d-a7317e80c5f7\") " pod="ai-tenant-e2e-shared-b-w3-da9451/rate-test-model-da9451-kserve-6864b69cc5-th8hg"
Sep 11 23:35:14.841458 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:35:14.841061    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0e2a488-50b3-4f87-a09d-a7317e80c5f7-tmp-dir\") pod \"rate-test-model-da9451-kserve-6864b69cc5-th8hg\" (UID: \"a0e2a488-50b3-4f87-a09d-a7317e80c5f7\") " pod="ai-tenant-e2e-shared-b-w3-da9451/rate-test-model-da9451-kserve-6864b69cc5-th8hg"
Sep 11 23:35:14.841458 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:35:14.841139    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5b8m\" (UniqueName: \"kubernetes.io/projected/a0e2a488-50b3-4f87-a09d-a7317e80c5f7-kube-api-access-z5b8m\") pod \"rate-test-model-da9451-kserve-6864b69cc5-th8hg\" (UID: \"a0e2a488-50b3-4f87-a09d-a7317e80c5f7\") " pod="ai-tenant-e2e-shared-b-w3-da9451/rate-test-model-da9451-kserve-6864b69cc5-th8hg"
Sep 11 23:35:14.841458 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:35:14.841205    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0e2a488-50b3-4f87-a09d-a7317e80c5f7-home\") pod \"rate-test-model-da9451-kserve-6864b69cc5-th8hg\" (UID: \"a0e2a488-50b3-4f87-a09d-a7317e80c5f7\") " pod="ai-tenant-e2e-shared-b-w3-da9451/rate-test-model-da9451-kserve-6864b69cc5-th8hg"
Sep 11 23:35:14.841458 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:35:14.841261    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0e2a488-50b3-4f87-a09d-a7317e80c5f7-tls-certs\") pod \"rate-test-model-da9451-kserve-6864b69cc5-th8hg\" (UID: \"a0e2a488-50b3-4f87-a09d-a7317e80c5f7\") " pod="ai-tenant-e2e-shared-b-w3-da9451/rate-test-model-da9451-kserve-6864b69cc5-th8hg"
Sep 11 23:35:14.847235 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:35:14.847207    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-da9451/rate-test-model-da9451-kserve-6864b69cc5-th8hg"]
Sep 11 23:35:14.941975 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:35:14.941936    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0e2a488-50b3-4f87-a09d-a7317e80c5f7-dshm\") pod \"rate-test-model-da9451-kserve-6864b69cc5-th8hg\" (UID: \"a0e2a488-50b3-4f87-a09d-a7317e80c5f7\") " pod="ai-tenant-e2e-shared-b-w3-da9451/rate-test-model-da9451-kserve-6864b69cc5-th8hg"
Sep 11 23:35:14.941975 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:35:14.941981    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0e2a488-50b3-4f87-a09d-a7317e80c5f7-tmp-dir\") pod \"rate-test-model-da9451-kserve-6864b69cc5-th8hg\" (UID: \"a0e2a488-50b3-4f87-a09d-a7317e80c5f7\") " pod="ai-tenant-e2e-shared-b-w3-da9451/rate-test-model-da9451-kserve-6864b69cc5-th8hg"
Sep 11 23:35:14.942334 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:35:14.942055    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5b8m\" (UniqueName: \"kubernetes.io/projected/a0e2a488-50b3-4f87-a09d-a7317e80c5f7-kube-api-access-z5b8m\") pod \"rate-test-model-da9451-kserve-6864b69cc5-th8hg\" (UID: \"a0e2a488-50b3-4f87-a09d-a7317e80c5f7\") " pod="ai-tenant-e2e-shared-b-w3-da9451/rate-test-model-da9451-kserve-6864b69cc5-th8hg"
Sep 11 23:35:14.942334 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:35:14.942090    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0e2a488-50b3-4f87-a09d-a7317e80c5f7-home\") pod \"rate-test-model-da9451-kserve-6864b69cc5-th8hg\" (UID: \"a0e2a488-50b3-4f87-a09d-a7317e80c5f7\") " pod="ai-tenant-e2e-shared-b-w3-da9451/rate-test-model-da9451-kserve-6864b69cc5-th8hg"
Sep 11 23:35:14.942334 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:35:14.942126    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0e2a488-50b3-4f87-a09d-a7317e80c5f7-tls-certs\") pod \"rate-test-model-da9451-kserve-6864b69cc5-th8hg\" (UID: \"a0e2a488-50b3-4f87-a09d-a7317e80c5f7\") " pod="ai-tenant-e2e-shared-b-w3-da9451/rate-test-model-da9451-kserve-6864b69cc5-th8hg"
Sep 11 23:35:14.942334 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:35:14.942244    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0e2a488-50b3-4f87-a09d-a7317e80c5f7-model-cache\") pod \"rate-test-model-da9451-kserve-6864b69cc5-th8hg\" (UID: \"a0e2a488-50b3-4f87-a09d-a7317e80c5f7\") " pod="ai-tenant-e2e-shared-b-w3-da9451/rate-test-model-da9451-kserve-6864b69cc5-th8hg"
Sep 11 23:35:14.942595 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:35:14.942574    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0e2a488-50b3-4f87-a09d-a7317e80c5f7-home\") pod \"rate-test-model-da9451-kserve-6864b69cc5-th8hg\" (UID: \"a0e2a488-50b3-4f87-a09d-a7317e80c5f7\") " pod="ai-tenant-e2e-shared-b-w3-da9451/rate-test-model-da9451-kserve-6864b69cc5-th8hg"
Sep 11 23:35:14.942668 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:35:14.942634    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0e2a488-50b3-4f87-a09d-a7317e80c5f7-tmp-dir\") pod \"rate-test-model-da9451-kserve-6864b69cc5-th8hg\" (UID: \"a0e2a488-50b3-4f87-a09d-a7317e80c5f7\") " pod="ai-tenant-e2e-shared-b-w3-da9451/rate-test-model-da9451-kserve-6864b69cc5-th8hg"
Sep 11 23:35:14.942722 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:35:14.942663    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0e2a488-50b3-4f87-a09d-a7317e80c5f7-model-cache\") pod \"rate-test-model-da9451-kserve-6864b69cc5-th8hg\" (UID: \"a0e2a488-50b3-4f87-a09d-a7317e80c5f7\") " pod="ai-tenant-e2e-shared-b-w3-da9451/rate-test-model-da9451-kserve-6864b69cc5-th8hg"
Sep 11 23:35:14.944603 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:35:14.944580    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0e2a488-50b3-4f87-a09d-a7317e80c5f7-dshm\") pod \"rate-test-model-da9451-kserve-6864b69cc5-th8hg\" (UID: \"a0e2a488-50b3-4f87-a09d-a7317e80c5f7\") " pod="ai-tenant-e2e-shared-b-w3-da9451/rate-test-model-da9451-kserve-6864b69cc5-th8hg"
Sep 11 23:35:14.945119 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:35:14.945098    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0e2a488-50b3-4f87-a09d-a7317e80c5f7-tls-certs\") pod \"rate-test-model-da9451-kserve-6864b69cc5-th8hg\" (UID: \"a0e2a488-50b3-4f87-a09d-a7317e80c5f7\") " pod="ai-tenant-e2e-shared-b-w3-da9451/rate-test-model-da9451-kserve-6864b69cc5-th8hg"
Sep 11 23:35:14.951773 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:35:14.951734    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5b8m\" (UniqueName: \"kubernetes.io/projected/a0e2a488-50b3-4f87-a09d-a7317e80c5f7-kube-api-access-z5b8m\") pod \"rate-test-model-da9451-kserve-6864b69cc5-th8hg\" (UID: \"a0e2a488-50b3-4f87-a09d-a7317e80c5f7\") " pod="ai-tenant-e2e-shared-b-w3-da9451/rate-test-model-da9451-kserve-6864b69cc5-th8hg"
Sep 11 23:35:15.147494 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:35:15.147398    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-da9451/rate-test-model-da9451-kserve-6864b69cc5-th8hg"
Sep 11 23:35:15.430215 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:35:15.430180    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda0e2a488_50b3_4f87_a09d_a7317e80c5f7.slice/crio-e684d993c991b8278fface8bcb6465fff67c8935b3f088cabb459268cca41639 WatchSource:0}: Error finding container e684d993c991b8278fface8bcb6465fff67c8935b3f088cabb459268cca41639: Status 404 returned error can't find the container with id e684d993c991b8278fface8bcb6465fff67c8935b3f088cabb459268cca41639
Sep 11 23:35:15.431085 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:35:15.431055    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-da9451/rate-test-model-da9451-kserve-6864b69cc5-th8hg"]
Sep 11 23:35:16.000839 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:35:16.000800    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-da9451/rate-test-model-da9451-kserve-6864b69cc5-th8hg" event={"ID":"a0e2a488-50b3-4f87-a09d-a7317e80c5f7","Type":"ContainerStarted","Data":"06cdc3ed89107d60415436c7290a22ca52bb7f3788b5a478ac617a32d045c677"}
Sep 11 23:35:16.000839 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:35:16.000845    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-da9451/rate-test-model-da9451-kserve-6864b69cc5-th8hg" event={"ID":"a0e2a488-50b3-4f87-a09d-a7317e80c5f7","Type":"ContainerStarted","Data":"e684d993c991b8278fface8bcb6465fff67c8935b3f088cabb459268cca41639"}
Sep 11 23:35:16.021048 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:35:16.020977    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-da9451/rate-test-model-da9451-kserve-6864b69cc5-th8hg" podStartSLOduration=2.020960498 podStartE2EDuration="2.020960498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 23:35:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:35:16.019807125 +0000 UTC m=+2040.052936356" watchObservedRunningTime="2026-09-11 23:35:16.020960498 +0000 UTC m=+2040.054089714"
Sep 11 23:35:18.497225 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:35:18.497185    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"]
Sep 11 23:35:25.148054 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:35:25.148013    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-da9451/rate-test-model-da9451-kserve-6864b69cc5-th8hg"
Sep 11 23:35:25.148662 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:35:25.148292    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-da9451/rate-test-model-da9451-kserve-6864b69cc5-th8hg"
Sep 11 23:35:25.161050 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:35:25.161019    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-da9451/rate-test-model-da9451-kserve-6864b69cc5-th8hg"
Sep 11 23:35:26.082068 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:35:26.082035    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-da9451/rate-test-model-da9451-kserve-6864b69cc5-th8hg"
Sep 11 23:35:45.302432 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:35:45.302345    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"]
Sep 11 23:35:57.690495 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:35:57.690452    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"]
Sep 11 23:36:04.340197 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:04.337630    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-da9451/rate-test-model-da9451-kserve-6864b69cc5-th8hg"]
Sep 11 23:36:05.180120 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:05.180080    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl"]
Sep 11 23:36:05.183912 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:05.183892    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl"
Sep 11 23:36:05.187937 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:05.187912    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-1fb90b-openshift-default-dockercfg-jtwvh\""
Sep 11 23:36:05.199624 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:05.199601    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl"]
Sep 11 23:36:05.255481 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:05.255436    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5-istiod-ca-cert\") pod \"e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl\" (UID: \"ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5\") " pod="openshift-ingress/e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl"
Sep 11 23:36:05.255687 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:05.255547    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5-istio-podinfo\") pod \"e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl\" (UID: \"ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5\") " pod="openshift-ingress/e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl"
Sep 11 23:36:05.255687 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:05.255646    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5-istio-envoy\") pod \"e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl\" (UID: \"ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5\") " pod="openshift-ingress/e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl"
Sep 11 23:36:05.255687 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:05.255672    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5-workload-certs\") pod \"e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl\" (UID: \"ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5\") " pod="openshift-ingress/e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl"
Sep 11 23:36:05.255868 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:05.255722    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5-istio-data\") pod \"e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl\" (UID: \"ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5\") " pod="openshift-ingress/e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl"
Sep 11 23:36:05.255868 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:05.255830    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5-credential-socket\") pod \"e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl\" (UID: \"ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5\") " pod="openshift-ingress/e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl"
Sep 11 23:36:05.255984 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:05.255890    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5-workload-socket\") pod \"e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl\" (UID: \"ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5\") " pod="openshift-ingress/e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl"
Sep 11 23:36:05.255984 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:05.255951    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5-istio-token\") pod \"e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl\" (UID: \"ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5\") " pod="openshift-ingress/e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl"
Sep 11 23:36:05.256092 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:05.256006    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjtpk\" (UniqueName: \"kubernetes.io/projected/ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5-kube-api-access-fjtpk\") pod \"e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl\" (UID: \"ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5\") " pod="openshift-ingress/e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl"
Sep 11 23:36:05.357046 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:05.356988    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5-istio-data\") pod \"e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl\" (UID: \"ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5\") " pod="openshift-ingress/e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl"
Sep 11 23:36:05.357046 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:05.357048    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5-credential-socket\") pod \"e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl\" (UID: \"ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5\") " pod="openshift-ingress/e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl"
Sep 11 23:36:05.357638 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:05.357082    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5-workload-socket\") pod \"e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl\" (UID: \"ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5\") " pod="openshift-ingress/e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl"
Sep 11 23:36:05.357638 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:05.357114    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5-istio-token\") pod \"e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl\" (UID: \"ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5\") " pod="openshift-ingress/e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl"
Sep 11 23:36:05.357638 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:05.357179    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjtpk\" (UniqueName: \"kubernetes.io/projected/ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5-kube-api-access-fjtpk\") pod \"e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl\" (UID: \"ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5\") " pod="openshift-ingress/e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl"
Sep 11 23:36:05.357638 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:05.357338    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5-istiod-ca-cert\") pod \"e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl\" (UID: \"ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5\") " pod="openshift-ingress/e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl"
Sep 11 23:36:05.357638 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:05.357422    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5-istio-podinfo\") pod \"e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl\" (UID: \"ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5\") " pod="openshift-ingress/e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl"
Sep 11 23:36:05.357638 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:05.357486    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5-istio-data\") pod \"e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl\" (UID: \"ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5\") " pod="openshift-ingress/e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl"
Sep 11 23:36:05.357638 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:05.357521    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5-workload-socket\") pod \"e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl\" (UID: \"ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5\") " pod="openshift-ingress/e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl"
Sep 11 23:36:05.357638 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:05.357572    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5-credential-socket\") pod \"e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl\" (UID: \"ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5\") " pod="openshift-ingress/e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl"
Sep 11 23:36:05.357638 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:05.357580    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5-istio-envoy\") pod \"e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl\" (UID: \"ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5\") " pod="openshift-ingress/e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl"
Sep 11 23:36:05.357638 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:05.357606    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5-workload-certs\") pod \"e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl\" (UID: \"ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5\") " pod="openshift-ingress/e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl"
Sep 11 23:36:05.358063 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:05.357841    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5-workload-certs\") pod \"e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl\" (UID: \"ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5\") " pod="openshift-ingress/e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl"
Sep 11 23:36:05.358063 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:05.358053    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5-istiod-ca-cert\") pod \"e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl\" (UID: \"ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5\") " pod="openshift-ingress/e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl"
Sep 11 23:36:05.360775 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:05.360747    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5-istio-envoy\") pod \"e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl\" (UID: \"ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5\") " pod="openshift-ingress/e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl"
Sep 11 23:36:05.361017 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:05.360984    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5-istio-podinfo\") pod \"e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl\" (UID: \"ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5\") " pod="openshift-ingress/e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl"
Sep 11 23:36:05.368805 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:05.368777    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjtpk\" (UniqueName: \"kubernetes.io/projected/ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5-kube-api-access-fjtpk\") pod \"e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl\" (UID: \"ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5\") " pod="openshift-ingress/e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl"
Sep 11 23:36:05.368905 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:05.368871    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5-istio-token\") pod \"e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl\" (UID: \"ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5\") " pod="openshift-ingress/e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl"
Sep 11 23:36:05.495392 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:05.495292    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl"
Sep 11 23:36:05.785921 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:05.785887    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl"]
Sep 11 23:36:05.787107 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:36:05.787076    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddd94cab_ad4d_4204_90b3_ba5d48d3e3c5.slice/crio-5af5076642cb5ec8232dc3514c304f70ea7d1d71f134028d9bdf892872217fec WatchSource:0}: Error finding container 5af5076642cb5ec8232dc3514c304f70ea7d1d71f134028d9bdf892872217fec: Status 404 returned error can't find the container with id 5af5076642cb5ec8232dc3514c304f70ea7d1d71f134028d9bdf892872217fec
Sep 11 23:36:05.789293 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:05.789255    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:36:05.789384 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:05.789327    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:36:05.789384 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:05.789357    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:36:06.351590 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:06.351550    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl" event={"ID":"ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5","Type":"ContainerStarted","Data":"55dfa5b33089714962a7c194b138be1a069aac7a783097bb3687e3f13e4af9b5"}
Sep 11 23:36:06.351787 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:06.351608    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl" event={"ID":"ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5","Type":"ContainerStarted","Data":"5af5076642cb5ec8232dc3514c304f70ea7d1d71f134028d9bdf892872217fec"}
Sep 11 23:36:06.376973 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:06.376907    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl" podStartSLOduration=1.376885426 podStartE2EDuration="1.376885426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 23:36:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:36:06.375704267 +0000 UTC m=+2090.408833499" watchObservedRunningTime="2026-09-11 23:36:06.376885426 +0000 UTC m=+2090.410014646"
Sep 11 23:36:06.505214 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:06.505174    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl"
Sep 11 23:36:06.506783 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:06.506761    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl"
Sep 11 23:36:07.360065 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:07.360029    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl"
Sep 11 23:36:07.361196 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:07.361146    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl"
Sep 11 23:36:08.615069 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:08.615021    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"]
Sep 11 23:36:16.697645 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:16.697613    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7244_6b0a8af3-932c-4091-b7a0-180fe1751a3a/ovn-acl-logging/0.log"
Sep 11 23:36:16.700388 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:16.700369    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7244_6b0a8af3-932c-4091-b7a0-180fe1751a3a/ovn-acl-logging/0.log"
Sep 11 23:36:16.989961 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:16.989870    2546 scope.go:117] "RemoveContainer" containerID="5eb31700a5b6a4306eefb1b5d145ec9fd4d3823315e3ebe80f44a3ba66be7060"
Sep 11 23:36:16.999485 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:16.999452    2546 scope.go:117] "RemoveContainer" containerID="3067ffd280000032781c905c3b4337da13c075689a673e68ad39d0da26fac3bf"
Sep 11 23:36:17.008934 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:17.008914    2546 scope.go:117] "RemoveContainer" containerID="7aa01b141c38d4cb2f50a806742c15a8b50fe63de303b8cea0bf09893b578803"
Sep 11 23:36:19.365104 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:19.365034    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-da9451/rate-test-model-da9451-kserve-6864b69cc5-th8hg" podUID="a0e2a488-50b3-4f87-a09d-a7317e80c5f7" containerName="main" containerID="cri-o://06cdc3ed89107d60415436c7290a22ca52bb7f3788b5a478ac617a32d045c677" gracePeriod=45
Sep 11 23:36:19.674619 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:19.674592    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-da9451/rate-test-model-da9451-kserve-6864b69cc5-th8hg"
Sep 11 23:36:19.807397 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:19.807353    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z5b8m\" (UniqueName: \"kubernetes.io/projected/a0e2a488-50b3-4f87-a09d-a7317e80c5f7-kube-api-access-z5b8m\") pod \"a0e2a488-50b3-4f87-a09d-a7317e80c5f7\" (UID: \"a0e2a488-50b3-4f87-a09d-a7317e80c5f7\") "
Sep 11 23:36:19.807612 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:19.807450    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0e2a488-50b3-4f87-a09d-a7317e80c5f7-tls-certs\") pod \"a0e2a488-50b3-4f87-a09d-a7317e80c5f7\" (UID: \"a0e2a488-50b3-4f87-a09d-a7317e80c5f7\") "
Sep 11 23:36:19.807849 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:19.807819    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0e2a488-50b3-4f87-a09d-a7317e80c5f7-dshm\") pod \"a0e2a488-50b3-4f87-a09d-a7317e80c5f7\" (UID: \"a0e2a488-50b3-4f87-a09d-a7317e80c5f7\") "
Sep 11 23:36:19.807925 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:19.807909    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0e2a488-50b3-4f87-a09d-a7317e80c5f7-home\") pod \"a0e2a488-50b3-4f87-a09d-a7317e80c5f7\" (UID: \"a0e2a488-50b3-4f87-a09d-a7317e80c5f7\") "
Sep 11 23:36:19.808193 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:19.807942    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0e2a488-50b3-4f87-a09d-a7317e80c5f7-model-cache\") pod \"a0e2a488-50b3-4f87-a09d-a7317e80c5f7\" (UID: \"a0e2a488-50b3-4f87-a09d-a7317e80c5f7\") "
Sep 11 23:36:19.808291 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:19.808216    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0e2a488-50b3-4f87-a09d-a7317e80c5f7-tmp-dir\") pod \"a0e2a488-50b3-4f87-a09d-a7317e80c5f7\" (UID: \"a0e2a488-50b3-4f87-a09d-a7317e80c5f7\") "
Sep 11 23:36:19.808291 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:19.808143    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0e2a488-50b3-4f87-a09d-a7317e80c5f7-home" (OuterVolumeSpecName: "home") pod "a0e2a488-50b3-4f87-a09d-a7317e80c5f7" (UID: "a0e2a488-50b3-4f87-a09d-a7317e80c5f7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:36:19.808291 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:19.808187    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0e2a488-50b3-4f87-a09d-a7317e80c5f7-model-cache" (OuterVolumeSpecName: "model-cache") pod "a0e2a488-50b3-4f87-a09d-a7317e80c5f7" (UID: "a0e2a488-50b3-4f87-a09d-a7317e80c5f7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:36:19.808476 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:19.808459    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0e2a488-50b3-4f87-a09d-a7317e80c5f7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a0e2a488-50b3-4f87-a09d-a7317e80c5f7" (UID: "a0e2a488-50b3-4f87-a09d-a7317e80c5f7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:36:19.808725 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:19.808711    2546 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0e2a488-50b3-4f87-a09d-a7317e80c5f7-home\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:36:19.808767 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:19.808728    2546 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0e2a488-50b3-4f87-a09d-a7317e80c5f7-model-cache\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:36:19.808767 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:19.808738    2546 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0e2a488-50b3-4f87-a09d-a7317e80c5f7-tmp-dir\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:36:19.809994 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:19.809975    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0e2a488-50b3-4f87-a09d-a7317e80c5f7-dshm" (OuterVolumeSpecName: "dshm") pod "a0e2a488-50b3-4f87-a09d-a7317e80c5f7" (UID: "a0e2a488-50b3-4f87-a09d-a7317e80c5f7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:36:19.810306 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:19.810283    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0e2a488-50b3-4f87-a09d-a7317e80c5f7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a0e2a488-50b3-4f87-a09d-a7317e80c5f7" (UID: "a0e2a488-50b3-4f87-a09d-a7317e80c5f7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 23:36:19.810426 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:19.810289    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0e2a488-50b3-4f87-a09d-a7317e80c5f7-kube-api-access-z5b8m" (OuterVolumeSpecName: "kube-api-access-z5b8m") pod "a0e2a488-50b3-4f87-a09d-a7317e80c5f7" (UID: "a0e2a488-50b3-4f87-a09d-a7317e80c5f7"). InnerVolumeSpecName "kube-api-access-z5b8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 23:36:19.910243 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:19.910213    2546 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0e2a488-50b3-4f87-a09d-a7317e80c5f7-tls-certs\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:36:19.910243 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:19.910244    2546 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0e2a488-50b3-4f87-a09d-a7317e80c5f7-dshm\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:36:19.910448 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:19.910253    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5b8m\" (UniqueName: \"kubernetes.io/projected/a0e2a488-50b3-4f87-a09d-a7317e80c5f7-kube-api-access-z5b8m\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:36:20.458435 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:20.458401    2546 generic.go:356] "Generic (PLEG): container finished" podID="a0e2a488-50b3-4f87-a09d-a7317e80c5f7" containerID="06cdc3ed89107d60415436c7290a22ca52bb7f3788b5a478ac617a32d045c677" exitCode=0
Sep 11 23:36:20.458891 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:20.458451    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-da9451/rate-test-model-da9451-kserve-6864b69cc5-th8hg"
Sep 11 23:36:20.459029 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:20.458470    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-da9451/rate-test-model-da9451-kserve-6864b69cc5-th8hg" event={"ID":"a0e2a488-50b3-4f87-a09d-a7317e80c5f7","Type":"ContainerDied","Data":"06cdc3ed89107d60415436c7290a22ca52bb7f3788b5a478ac617a32d045c677"}
Sep 11 23:36:20.459127 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:20.459116    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-da9451/rate-test-model-da9451-kserve-6864b69cc5-th8hg" event={"ID":"a0e2a488-50b3-4f87-a09d-a7317e80c5f7","Type":"ContainerDied","Data":"e684d993c991b8278fface8bcb6465fff67c8935b3f088cabb459268cca41639"}
Sep 11 23:36:20.459225 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:20.459185    2546 scope.go:117] "RemoveContainer" containerID="06cdc3ed89107d60415436c7290a22ca52bb7f3788b5a478ac617a32d045c677"
Sep 11 23:36:20.469844 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:20.469816    2546 scope.go:117] "RemoveContainer" containerID="06cdc3ed89107d60415436c7290a22ca52bb7f3788b5a478ac617a32d045c677"
Sep 11 23:36:20.470214 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:36:20.470193    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06cdc3ed89107d60415436c7290a22ca52bb7f3788b5a478ac617a32d045c677\": container with ID starting with 06cdc3ed89107d60415436c7290a22ca52bb7f3788b5a478ac617a32d045c677 not found: ID does not exist" containerID="06cdc3ed89107d60415436c7290a22ca52bb7f3788b5a478ac617a32d045c677"
Sep 11 23:36:20.470273 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:20.470225    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06cdc3ed89107d60415436c7290a22ca52bb7f3788b5a478ac617a32d045c677"} err="failed to get container status \"06cdc3ed89107d60415436c7290a22ca52bb7f3788b5a478ac617a32d045c677\": rpc error: code = NotFound desc = could not find container \"06cdc3ed89107d60415436c7290a22ca52bb7f3788b5a478ac617a32d045c677\": container with ID starting with 06cdc3ed89107d60415436c7290a22ca52bb7f3788b5a478ac617a32d045c677 not found: ID does not exist"
Sep 11 23:36:20.483500 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:20.483465    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-da9451/rate-test-model-da9451-kserve-6864b69cc5-th8hg"]
Sep 11 23:36:20.487679 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:20.487648    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-da9451/rate-test-model-da9451-kserve-6864b69cc5-th8hg"]
Sep 11 23:36:20.495515 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:20.495486    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0e2a488-50b3-4f87-a09d-a7317e80c5f7" path="/var/lib/kubelet/pods/a0e2a488-50b3-4f87-a09d-a7317e80c5f7/volumes"
Sep 11 23:36:21.893268 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:21.893224    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"]
Sep 11 23:36:23.401312 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:23.401274    2546 patch_prober.go:28] interesting pod/e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 11 23:36:23.401841 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:23.401357    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl" podUID="ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 11 23:36:25.504374 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:25.504338    2546 generic.go:356] "Generic (PLEG): container finished" podID="ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5" containerID="55dfa5b33089714962a7c194b138be1a069aac7a783097bb3687e3f13e4af9b5" exitCode=0
Sep 11 23:36:25.504968 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:25.504415    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl" event={"ID":"ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5","Type":"ContainerDied","Data":"55dfa5b33089714962a7c194b138be1a069aac7a783097bb3687e3f13e4af9b5"}
Sep 11 23:36:25.504968 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:25.504842    2546 scope.go:117] "RemoveContainer" containerID="55dfa5b33089714962a7c194b138be1a069aac7a783097bb3687e3f13e4af9b5"
Sep 11 23:36:25.505918 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:25.505892    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:36:25.505984 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:25.505968    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:36:25.506023 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:25.505998    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:36:26.513325 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:26.513282    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl" event={"ID":"ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5","Type":"ContainerStarted","Data":"872973c8afd0a664a6f2a314cbd6b1b93a803a8f0536f6aa4b416e02086dbcf7"}
Sep 11 23:36:27.495510 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:27.495459    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl"
Sep 11 23:36:27.495738 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:27.495716    2546 patch_prober.go:28] interesting pod/e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Sep 11 23:36:27.495827 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:27.495793    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl" podUID="ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Sep 11 23:36:28.495798 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:28.495760    2546 patch_prober.go:28] interesting pod/e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Sep 11 23:36:28.496245 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:28.495827    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl" podUID="ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Sep 11 23:36:29.496360 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:29.496315    2546 patch_prober.go:28] interesting pod/e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Sep 11 23:36:29.496918 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:29.496380    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl" podUID="ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Sep 11 23:36:30.496293 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:30.496261    2546 patch_prober.go:28] interesting pod/e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Sep 11 23:36:30.496474 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:30.496314    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl" podUID="ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Sep 11 23:36:31.495980 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:31.495941    2546 patch_prober.go:28] interesting pod/e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Sep 11 23:36:31.496200 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:31.496017    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl" podUID="ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Sep 11 23:36:32.499301 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:32.499270    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl"
Sep 11 23:36:32.499775 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:32.499736    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl"
Sep 11 23:36:32.500567 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:32.500547    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl"
Sep 11 23:36:34.177455 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:34.177421    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-778566-openshift-default-5f49f5c898-cjpxf"]
Sep 11 23:36:34.177964 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:34.177894    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0e2a488-50b3-4f87-a09d-a7317e80c5f7" containerName="main"
Sep 11 23:36:34.177964 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:34.177908    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0e2a488-50b3-4f87-a09d-a7317e80c5f7" containerName="main"
Sep 11 23:36:34.178102 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:34.178002    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0e2a488-50b3-4f87-a09d-a7317e80c5f7" containerName="main"
Sep 11 23:36:34.181436 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:34.181407    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-778566-openshift-default-5f49f5c898-cjpxf"
Sep 11 23:36:34.187866 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:34.187837    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-778566-openshift-default-dockercfg-88vks\""
Sep 11 23:36:34.200048 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:34.200012    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-778566-openshift-default-5f49f5c898-cjpxf"]
Sep 11 23:36:34.266259 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:34.266221    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e61c589e-76b3-4eef-b717-1262329daba2-istio-envoy\") pod \"e2e-ipp-b-778566-openshift-default-5f49f5c898-cjpxf\" (UID: \"e61c589e-76b3-4eef-b717-1262329daba2\") " pod="openshift-ingress/e2e-ipp-b-778566-openshift-default-5f49f5c898-cjpxf"
Sep 11 23:36:34.266463 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:34.266268    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6dbf\" (UniqueName: \"kubernetes.io/projected/e61c589e-76b3-4eef-b717-1262329daba2-kube-api-access-b6dbf\") pod \"e2e-ipp-b-778566-openshift-default-5f49f5c898-cjpxf\" (UID: \"e61c589e-76b3-4eef-b717-1262329daba2\") " pod="openshift-ingress/e2e-ipp-b-778566-openshift-default-5f49f5c898-cjpxf"
Sep 11 23:36:34.266463 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:34.266306    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e61c589e-76b3-4eef-b717-1262329daba2-credential-socket\") pod \"e2e-ipp-b-778566-openshift-default-5f49f5c898-cjpxf\" (UID: \"e61c589e-76b3-4eef-b717-1262329daba2\") " pod="openshift-ingress/e2e-ipp-b-778566-openshift-default-5f49f5c898-cjpxf"
Sep 11 23:36:34.266463 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:34.266326    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e61c589e-76b3-4eef-b717-1262329daba2-istio-data\") pod \"e2e-ipp-b-778566-openshift-default-5f49f5c898-cjpxf\" (UID: \"e61c589e-76b3-4eef-b717-1262329daba2\") " pod="openshift-ingress/e2e-ipp-b-778566-openshift-default-5f49f5c898-cjpxf"
Sep 11 23:36:34.266463 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:34.266373    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e61c589e-76b3-4eef-b717-1262329daba2-istio-podinfo\") pod \"e2e-ipp-b-778566-openshift-default-5f49f5c898-cjpxf\" (UID: \"e61c589e-76b3-4eef-b717-1262329daba2\") " pod="openshift-ingress/e2e-ipp-b-778566-openshift-default-5f49f5c898-cjpxf"
Sep 11 23:36:34.266463 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:34.266393    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e61c589e-76b3-4eef-b717-1262329daba2-workload-certs\") pod \"e2e-ipp-b-778566-openshift-default-5f49f5c898-cjpxf\" (UID: \"e61c589e-76b3-4eef-b717-1262329daba2\") " pod="openshift-ingress/e2e-ipp-b-778566-openshift-default-5f49f5c898-cjpxf"
Sep 11 23:36:34.266463 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:34.266410    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e61c589e-76b3-4eef-b717-1262329daba2-workload-socket\") pod \"e2e-ipp-b-778566-openshift-default-5f49f5c898-cjpxf\" (UID: \"e61c589e-76b3-4eef-b717-1262329daba2\") " pod="openshift-ingress/e2e-ipp-b-778566-openshift-default-5f49f5c898-cjpxf"
Sep 11 23:36:34.266823 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:34.266519    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e61c589e-76b3-4eef-b717-1262329daba2-istiod-ca-cert\") pod \"e2e-ipp-b-778566-openshift-default-5f49f5c898-cjpxf\" (UID: \"e61c589e-76b3-4eef-b717-1262329daba2\") " pod="openshift-ingress/e2e-ipp-b-778566-openshift-default-5f49f5c898-cjpxf"
Sep 11 23:36:34.266823 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:34.266609    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e61c589e-76b3-4eef-b717-1262329daba2-istio-token\") pod \"e2e-ipp-b-778566-openshift-default-5f49f5c898-cjpxf\" (UID: \"e61c589e-76b3-4eef-b717-1262329daba2\") " pod="openshift-ingress/e2e-ipp-b-778566-openshift-default-5f49f5c898-cjpxf"
Sep 11 23:36:34.367785 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:34.367741    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e61c589e-76b3-4eef-b717-1262329daba2-istio-token\") pod \"e2e-ipp-b-778566-openshift-default-5f49f5c898-cjpxf\" (UID: \"e61c589e-76b3-4eef-b717-1262329daba2\") " pod="openshift-ingress/e2e-ipp-b-778566-openshift-default-5f49f5c898-cjpxf"
Sep 11 23:36:34.367962 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:34.367910    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e61c589e-76b3-4eef-b717-1262329daba2-istio-envoy\") pod \"e2e-ipp-b-778566-openshift-default-5f49f5c898-cjpxf\" (UID: \"e61c589e-76b3-4eef-b717-1262329daba2\") " pod="openshift-ingress/e2e-ipp-b-778566-openshift-default-5f49f5c898-cjpxf"
Sep 11 23:36:34.367962 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:34.367938    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6dbf\" (UniqueName: \"kubernetes.io/projected/e61c589e-76b3-4eef-b717-1262329daba2-kube-api-access-b6dbf\") pod \"e2e-ipp-b-778566-openshift-default-5f49f5c898-cjpxf\" (UID: \"e61c589e-76b3-4eef-b717-1262329daba2\") " pod="openshift-ingress/e2e-ipp-b-778566-openshift-default-5f49f5c898-cjpxf"
Sep 11 23:36:34.368055 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:34.367963    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e61c589e-76b3-4eef-b717-1262329daba2-credential-socket\") pod \"e2e-ipp-b-778566-openshift-default-5f49f5c898-cjpxf\" (UID: \"e61c589e-76b3-4eef-b717-1262329daba2\") " pod="openshift-ingress/e2e-ipp-b-778566-openshift-default-5f49f5c898-cjpxf"
Sep 11 23:36:34.368055 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:34.367993    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e61c589e-76b3-4eef-b717-1262329daba2-istio-data\") pod \"e2e-ipp-b-778566-openshift-default-5f49f5c898-cjpxf\" (UID: \"e61c589e-76b3-4eef-b717-1262329daba2\") " pod="openshift-ingress/e2e-ipp-b-778566-openshift-default-5f49f5c898-cjpxf"
Sep 11 23:36:34.368055 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:34.368029    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e61c589e-76b3-4eef-b717-1262329daba2-istio-podinfo\") pod \"e2e-ipp-b-778566-openshift-default-5f49f5c898-cjpxf\" (UID: \"e61c589e-76b3-4eef-b717-1262329daba2\") " pod="openshift-ingress/e2e-ipp-b-778566-openshift-default-5f49f5c898-cjpxf"
Sep 11 23:36:34.368265 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:34.368056    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e61c589e-76b3-4eef-b717-1262329daba2-workload-certs\") pod \"e2e-ipp-b-778566-openshift-default-5f49f5c898-cjpxf\" (UID: \"e61c589e-76b3-4eef-b717-1262329daba2\") " pod="openshift-ingress/e2e-ipp-b-778566-openshift-default-5f49f5c898-cjpxf"
Sep 11 23:36:34.368265 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:34.368083    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e61c589e-76b3-4eef-b717-1262329daba2-workload-socket\") pod \"e2e-ipp-b-778566-openshift-default-5f49f5c898-cjpxf\" (UID: \"e61c589e-76b3-4eef-b717-1262329daba2\") " pod="openshift-ingress/e2e-ipp-b-778566-openshift-default-5f49f5c898-cjpxf"
Sep 11 23:36:34.368265 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:34.368187    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e61c589e-76b3-4eef-b717-1262329daba2-istiod-ca-cert\") pod \"e2e-ipp-b-778566-openshift-default-5f49f5c898-cjpxf\" (UID: \"e61c589e-76b3-4eef-b717-1262329daba2\") " pod="openshift-ingress/e2e-ipp-b-778566-openshift-default-5f49f5c898-cjpxf"
Sep 11 23:36:34.368515 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:34.368486    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e61c589e-76b3-4eef-b717-1262329daba2-workload-certs\") pod \"e2e-ipp-b-778566-openshift-default-5f49f5c898-cjpxf\" (UID: \"e61c589e-76b3-4eef-b717-1262329daba2\") " pod="openshift-ingress/e2e-ipp-b-778566-openshift-default-5f49f5c898-cjpxf"
Sep 11 23:36:34.368515 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:34.368505    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e61c589e-76b3-4eef-b717-1262329daba2-credential-socket\") pod \"e2e-ipp-b-778566-openshift-default-5f49f5c898-cjpxf\" (UID: \"e61c589e-76b3-4eef-b717-1262329daba2\") " pod="openshift-ingress/e2e-ipp-b-778566-openshift-default-5f49f5c898-cjpxf"
Sep 11 23:36:34.368681 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:34.368586    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e61c589e-76b3-4eef-b717-1262329daba2-istio-data\") pod \"e2e-ipp-b-778566-openshift-default-5f49f5c898-cjpxf\" (UID: \"e61c589e-76b3-4eef-b717-1262329daba2\") " pod="openshift-ingress/e2e-ipp-b-778566-openshift-default-5f49f5c898-cjpxf"
Sep 11 23:36:34.368681 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:34.368634    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e61c589e-76b3-4eef-b717-1262329daba2-workload-socket\") pod \"e2e-ipp-b-778566-openshift-default-5f49f5c898-cjpxf\" (UID: \"e61c589e-76b3-4eef-b717-1262329daba2\") " pod="openshift-ingress/e2e-ipp-b-778566-openshift-default-5f49f5c898-cjpxf"
Sep 11 23:36:34.369014 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:34.368971    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e61c589e-76b3-4eef-b717-1262329daba2-istiod-ca-cert\") pod \"e2e-ipp-b-778566-openshift-default-5f49f5c898-cjpxf\" (UID: \"e61c589e-76b3-4eef-b717-1262329daba2\") " pod="openshift-ingress/e2e-ipp-b-778566-openshift-default-5f49f5c898-cjpxf"
Sep 11 23:36:34.370496 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:34.370476    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e61c589e-76b3-4eef-b717-1262329daba2-istio-envoy\") pod \"e2e-ipp-b-778566-openshift-default-5f49f5c898-cjpxf\" (UID: \"e61c589e-76b3-4eef-b717-1262329daba2\") " pod="openshift-ingress/e2e-ipp-b-778566-openshift-default-5f49f5c898-cjpxf"
Sep 11 23:36:34.370797 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:34.370778    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e61c589e-76b3-4eef-b717-1262329daba2-istio-podinfo\") pod \"e2e-ipp-b-778566-openshift-default-5f49f5c898-cjpxf\" (UID: \"e61c589e-76b3-4eef-b717-1262329daba2\") " pod="openshift-ingress/e2e-ipp-b-778566-openshift-default-5f49f5c898-cjpxf"
Sep 11 23:36:34.376343 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:34.376318    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e61c589e-76b3-4eef-b717-1262329daba2-istio-token\") pod \"e2e-ipp-b-778566-openshift-default-5f49f5c898-cjpxf\" (UID: \"e61c589e-76b3-4eef-b717-1262329daba2\") " pod="openshift-ingress/e2e-ipp-b-778566-openshift-default-5f49f5c898-cjpxf"
Sep 11 23:36:34.376705 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:34.376684    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6dbf\" (UniqueName: \"kubernetes.io/projected/e61c589e-76b3-4eef-b717-1262329daba2-kube-api-access-b6dbf\") pod \"e2e-ipp-b-778566-openshift-default-5f49f5c898-cjpxf\" (UID: \"e61c589e-76b3-4eef-b717-1262329daba2\") " pod="openshift-ingress/e2e-ipp-b-778566-openshift-default-5f49f5c898-cjpxf"
Sep 11 23:36:34.492537 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:34.492447    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-778566-openshift-default-5f49f5c898-cjpxf"
Sep 11 23:36:34.779902 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:34.779875    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-778566-openshift-default-5f49f5c898-cjpxf"]
Sep 11 23:36:34.782612 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:36:34.782579    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode61c589e_76b3_4eef_b717_1262329daba2.slice/crio-9976495b6dc5dedc4031dd2bcbc99d6985415b2b93d6dcce6bd82cf009f0c9de WatchSource:0}: Error finding container 9976495b6dc5dedc4031dd2bcbc99d6985415b2b93d6dcce6bd82cf009f0c9de: Status 404 returned error can't find the container with id 9976495b6dc5dedc4031dd2bcbc99d6985415b2b93d6dcce6bd82cf009f0c9de
Sep 11 23:36:34.792335 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:34.792275    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:36:34.792480 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:34.792432    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:36:34.792541 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:34.792511    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:36:35.593525 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:35.593478    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-778566-openshift-default-5f49f5c898-cjpxf" event={"ID":"e61c589e-76b3-4eef-b717-1262329daba2","Type":"ContainerStarted","Data":"309b53403bef3b77ec46382e97efc39d4e94934996ab31898300094af5da6fb5"}
Sep 11 23:36:35.594032 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:35.593586    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-778566-openshift-default-5f49f5c898-cjpxf" event={"ID":"e61c589e-76b3-4eef-b717-1262329daba2","Type":"ContainerStarted","Data":"9976495b6dc5dedc4031dd2bcbc99d6985415b2b93d6dcce6bd82cf009f0c9de"}
Sep 11 23:36:35.631608 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:35.631544    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-778566-openshift-default-5f49f5c898-cjpxf" podStartSLOduration=1.631525006 podStartE2EDuration="1.631525006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 23:36:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:36:35.629321343 +0000 UTC m=+2119.662450561" watchObservedRunningTime="2026-09-11 23:36:35.631525006 +0000 UTC m=+2119.664654222"
Sep 11 23:36:36.495209 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:36.495147    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-778566-openshift-default-5f49f5c898-cjpxf"
Sep 11 23:36:36.498227 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:36.498204    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-778566-openshift-default-5f49f5c898-cjpxf"
Sep 11 23:36:36.600200 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:36.600140    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-778566-openshift-default-5f49f5c898-cjpxf"
Sep 11 23:36:36.601067 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:36.601048    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-778566-openshift-default-5f49f5c898-cjpxf"
Sep 11 23:36:45.891068 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:45.891025    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"]
Sep 11 23:36:56.747269 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:56.747230    2546 generic.go:356] "Generic (PLEG): container finished" podID="e61c589e-76b3-4eef-b717-1262329daba2" containerID="309b53403bef3b77ec46382e97efc39d4e94934996ab31898300094af5da6fb5" exitCode=0
Sep 11 23:36:56.747697 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:56.747310    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-778566-openshift-default-5f49f5c898-cjpxf" event={"ID":"e61c589e-76b3-4eef-b717-1262329daba2","Type":"ContainerDied","Data":"309b53403bef3b77ec46382e97efc39d4e94934996ab31898300094af5da6fb5"}
Sep 11 23:36:56.747773 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:56.747758    2546 scope.go:117] "RemoveContainer" containerID="309b53403bef3b77ec46382e97efc39d4e94934996ab31898300094af5da6fb5"
Sep 11 23:36:56.748970 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:56.748944    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:36:56.749039 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:56.749025    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:36:56.749077 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:56.749064    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:36:57.758140 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:57.758104    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-778566-openshift-default-5f49f5c898-cjpxf" event={"ID":"e61c589e-76b3-4eef-b717-1262329daba2","Type":"ContainerStarted","Data":"614ca7d8998a7b0077d25ec359ad8602b43ef3baa69024b88eb2accbfd2c1843"}
Sep 11 23:36:58.493583 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:58.493542    2546 patch_prober.go:28] interesting pod/e2e-ipp-b-778566-openshift-default-5f49f5c898-cjpxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Sep 11 23:36:58.493762 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:58.493614    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-778566-openshift-default-5f49f5c898-cjpxf" podUID="e61c589e-76b3-4eef-b717-1262329daba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Sep 11 23:36:58.494638 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:58.494620    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-778566-openshift-default-5f49f5c898-cjpxf"
Sep 11 23:36:59.493687 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:59.493649    2546 patch_prober.go:28] interesting pod/e2e-ipp-b-778566-openshift-default-5f49f5c898-cjpxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Sep 11 23:36:59.494132 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:36:59.493714    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-778566-openshift-default-5f49f5c898-cjpxf" podUID="e61c589e-76b3-4eef-b717-1262329daba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Sep 11 23:37:00.492757 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:37:00.492727    2546 patch_prober.go:28] interesting pod/e2e-ipp-b-778566-openshift-default-5f49f5c898-cjpxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Sep 11 23:37:00.492934 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:37:00.492781    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-778566-openshift-default-5f49f5c898-cjpxf" podUID="e61c589e-76b3-4eef-b717-1262329daba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Sep 11 23:37:01.493449 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:37:01.493413    2546 patch_prober.go:28] interesting pod/e2e-ipp-b-778566-openshift-default-5f49f5c898-cjpxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Sep 11 23:37:01.493850 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:37:01.493471    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-778566-openshift-default-5f49f5c898-cjpxf" podUID="e61c589e-76b3-4eef-b717-1262329daba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Sep 11 23:37:02.493004 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:37:02.492954    2546 patch_prober.go:28] interesting pod/e2e-ipp-b-778566-openshift-default-5f49f5c898-cjpxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Sep 11 23:37:02.493237 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:37:02.493047    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-778566-openshift-default-5f49f5c898-cjpxf" podUID="e61c589e-76b3-4eef-b717-1262329daba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Sep 11 23:37:03.497596 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:37:03.497556    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-778566-openshift-default-5f49f5c898-cjpxf"
Sep 11 23:37:03.498039 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:37:03.497846    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-778566-openshift-default-5f49f5c898-cjpxf"
Sep 11 23:37:03.498610 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:37:03.498593    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-778566-openshift-default-5f49f5c898-cjpxf"
Sep 11 23:37:17.066729 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:37:17.066646    2546 scope.go:117] "RemoveContainer" containerID="f55206e553cbef925be9237271e7c65ff353cb8cbaa70ebcf65db14b26db9f57"
Sep 11 23:37:17.076357 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:37:17.076337    2546 scope.go:117] "RemoveContainer" containerID="afa50b0afaf4647971b34dc1b281837dbe3f24c5b03ec0cf932508393b684264"
Sep 11 23:37:30.727559 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:37:30.727516    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-1fb90b/ipp-route-1fb90b-kserve-687df66fbc-vms2m"]
Sep 11 23:37:30.731076 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:37:30.731056    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-1fb90b/ipp-route-1fb90b-kserve-687df66fbc-vms2m"
Sep 11 23:37:30.735391 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:37:30.735357    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-1fb90b\"/\"openshift-service-ca.crt\""
Sep 11 23:37:30.735558 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:37:30.735403    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-1fb90b\"/\"ipp-route-1fb90b-kserve-self-signed-certs\""
Sep 11 23:37:30.735558 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:37:30.735461    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-1fb90b\"/\"kube-root-ca.crt\""
Sep 11 23:37:30.735558 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:37:30.735476    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-1fb90b\"/\"default-dockercfg-f9d8h\""
Sep 11 23:37:30.741252 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:37:30.741230    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-1fb90b/ipp-route-1fb90b-kserve-687df66fbc-vms2m"]
Sep 11 23:37:30.864660 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:37:30.864608    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9j44\" (UniqueName: \"kubernetes.io/projected/b2c79637-9420-4da1-8ffb-9c916f2b62f3-kube-api-access-g9j44\") pod \"ipp-route-1fb90b-kserve-687df66fbc-vms2m\" (UID: \"b2c79637-9420-4da1-8ffb-9c916f2b62f3\") " pod="ai-tenant-e2e-ipp-a-1fb90b/ipp-route-1fb90b-kserve-687df66fbc-vms2m"
Sep 11 23:37:30.864880 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:37:30.864733    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2c79637-9420-4da1-8ffb-9c916f2b62f3-tls-certs\") pod \"ipp-route-1fb90b-kserve-687df66fbc-vms2m\" (UID: \"b2c79637-9420-4da1-8ffb-9c916f2b62f3\") " pod="ai-tenant-e2e-ipp-a-1fb90b/ipp-route-1fb90b-kserve-687df66fbc-vms2m"
Sep 11 23:37:30.864880 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:37:30.864794    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2c79637-9420-4da1-8ffb-9c916f2b62f3-model-cache\") pod \"ipp-route-1fb90b-kserve-687df66fbc-vms2m\" (UID: \"b2c79637-9420-4da1-8ffb-9c916f2b62f3\") " pod="ai-tenant-e2e-ipp-a-1fb90b/ipp-route-1fb90b-kserve-687df66fbc-vms2m"
Sep 11 23:37:30.864880 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:37:30.864842    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2c79637-9420-4da1-8ffb-9c916f2b62f3-tmp-dir\") pod \"ipp-route-1fb90b-kserve-687df66fbc-vms2m\" (UID: \"b2c79637-9420-4da1-8ffb-9c916f2b62f3\") " pod="ai-tenant-e2e-ipp-a-1fb90b/ipp-route-1fb90b-kserve-687df66fbc-vms2m"
Sep 11 23:37:30.865034 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:37:30.864884    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2c79637-9420-4da1-8ffb-9c916f2b62f3-dshm\") pod \"ipp-route-1fb90b-kserve-687df66fbc-vms2m\" (UID: \"b2c79637-9420-4da1-8ffb-9c916f2b62f3\") " pod="ai-tenant-e2e-ipp-a-1fb90b/ipp-route-1fb90b-kserve-687df66fbc-vms2m"
Sep 11 23:37:30.865034 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:37:30.865013    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2c79637-9420-4da1-8ffb-9c916f2b62f3-home\") pod \"ipp-route-1fb90b-kserve-687df66fbc-vms2m\" (UID: \"b2c79637-9420-4da1-8ffb-9c916f2b62f3\") " pod="ai-tenant-e2e-ipp-a-1fb90b/ipp-route-1fb90b-kserve-687df66fbc-vms2m"
Sep 11 23:37:30.965682 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:37:30.965643    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2c79637-9420-4da1-8ffb-9c916f2b62f3-tls-certs\") pod \"ipp-route-1fb90b-kserve-687df66fbc-vms2m\" (UID: \"b2c79637-9420-4da1-8ffb-9c916f2b62f3\") " pod="ai-tenant-e2e-ipp-a-1fb90b/ipp-route-1fb90b-kserve-687df66fbc-vms2m"
Sep 11 23:37:30.965682 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:37:30.965680    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2c79637-9420-4da1-8ffb-9c916f2b62f3-model-cache\") pod \"ipp-route-1fb90b-kserve-687df66fbc-vms2m\" (UID: \"b2c79637-9420-4da1-8ffb-9c916f2b62f3\") " pod="ai-tenant-e2e-ipp-a-1fb90b/ipp-route-1fb90b-kserve-687df66fbc-vms2m"
Sep 11 23:37:30.965949 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:37:30.965709    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2c79637-9420-4da1-8ffb-9c916f2b62f3-tmp-dir\") pod \"ipp-route-1fb90b-kserve-687df66fbc-vms2m\" (UID: \"b2c79637-9420-4da1-8ffb-9c916f2b62f3\") " pod="ai-tenant-e2e-ipp-a-1fb90b/ipp-route-1fb90b-kserve-687df66fbc-vms2m"
Sep 11 23:37:30.965949 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:37:30.965750    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2c79637-9420-4da1-8ffb-9c916f2b62f3-dshm\") pod \"ipp-route-1fb90b-kserve-687df66fbc-vms2m\" (UID: \"b2c79637-9420-4da1-8ffb-9c916f2b62f3\") " pod="ai-tenant-e2e-ipp-a-1fb90b/ipp-route-1fb90b-kserve-687df66fbc-vms2m"
Sep 11 23:37:30.965949 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:37:30.965816    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2c79637-9420-4da1-8ffb-9c916f2b62f3-home\") pod \"ipp-route-1fb90b-kserve-687df66fbc-vms2m\" (UID: \"b2c79637-9420-4da1-8ffb-9c916f2b62f3\") " pod="ai-tenant-e2e-ipp-a-1fb90b/ipp-route-1fb90b-kserve-687df66fbc-vms2m"
Sep 11 23:37:30.965949 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:37:30.965892    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9j44\" (UniqueName: \"kubernetes.io/projected/b2c79637-9420-4da1-8ffb-9c916f2b62f3-kube-api-access-g9j44\") pod \"ipp-route-1fb90b-kserve-687df66fbc-vms2m\" (UID: \"b2c79637-9420-4da1-8ffb-9c916f2b62f3\") " pod="ai-tenant-e2e-ipp-a-1fb90b/ipp-route-1fb90b-kserve-687df66fbc-vms2m"
Sep 11 23:37:30.966198 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:37:30.966130    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2c79637-9420-4da1-8ffb-9c916f2b62f3-model-cache\") pod \"ipp-route-1fb90b-kserve-687df66fbc-vms2m\" (UID: \"b2c79637-9420-4da1-8ffb-9c916f2b62f3\") " pod="ai-tenant-e2e-ipp-a-1fb90b/ipp-route-1fb90b-kserve-687df66fbc-vms2m"
Sep 11 23:37:30.966272 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:37:30.966205    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2c79637-9420-4da1-8ffb-9c916f2b62f3-tmp-dir\") pod \"ipp-route-1fb90b-kserve-687df66fbc-vms2m\" (UID: \"b2c79637-9420-4da1-8ffb-9c916f2b62f3\") " pod="ai-tenant-e2e-ipp-a-1fb90b/ipp-route-1fb90b-kserve-687df66fbc-vms2m"
Sep 11 23:37:30.966331 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:37:30.966287    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2c79637-9420-4da1-8ffb-9c916f2b62f3-home\") pod \"ipp-route-1fb90b-kserve-687df66fbc-vms2m\" (UID: \"b2c79637-9420-4da1-8ffb-9c916f2b62f3\") " pod="ai-tenant-e2e-ipp-a-1fb90b/ipp-route-1fb90b-kserve-687df66fbc-vms2m"
Sep 11 23:37:30.968356 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:37:30.968325    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2c79637-9420-4da1-8ffb-9c916f2b62f3-dshm\") pod \"ipp-route-1fb90b-kserve-687df66fbc-vms2m\" (UID: \"b2c79637-9420-4da1-8ffb-9c916f2b62f3\") " pod="ai-tenant-e2e-ipp-a-1fb90b/ipp-route-1fb90b-kserve-687df66fbc-vms2m"
Sep 11 23:37:30.968770 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:37:30.968748    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2c79637-9420-4da1-8ffb-9c916f2b62f3-tls-certs\") pod \"ipp-route-1fb90b-kserve-687df66fbc-vms2m\" (UID: \"b2c79637-9420-4da1-8ffb-9c916f2b62f3\") " pod="ai-tenant-e2e-ipp-a-1fb90b/ipp-route-1fb90b-kserve-687df66fbc-vms2m"
Sep 11 23:37:30.976970 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:37:30.976942    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9j44\" (UniqueName: \"kubernetes.io/projected/b2c79637-9420-4da1-8ffb-9c916f2b62f3-kube-api-access-g9j44\") pod \"ipp-route-1fb90b-kserve-687df66fbc-vms2m\" (UID: \"b2c79637-9420-4da1-8ffb-9c916f2b62f3\") " pod="ai-tenant-e2e-ipp-a-1fb90b/ipp-route-1fb90b-kserve-687df66fbc-vms2m"
Sep 11 23:37:31.041037 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:37:31.040935    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-1fb90b/ipp-route-1fb90b-kserve-687df66fbc-vms2m"
Sep 11 23:37:31.324317 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:37:31.324276    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-1fb90b/ipp-route-1fb90b-kserve-687df66fbc-vms2m"]
Sep 11 23:37:31.325684 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:37:31.325650    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb2c79637_9420_4da1_8ffb_9c916f2b62f3.slice/crio-22965f356ddbb2d6a72ff6c70f8da250b533e9122f032f73c8e6461cc0871324 WatchSource:0}: Error finding container 22965f356ddbb2d6a72ff6c70f8da250b533e9122f032f73c8e6461cc0871324: Status 404 returned error can't find the container with id 22965f356ddbb2d6a72ff6c70f8da250b533e9122f032f73c8e6461cc0871324
Sep 11 23:37:31.978373 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:37:31.978331    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-1fb90b/ipp-route-1fb90b-kserve-687df66fbc-vms2m" event={"ID":"b2c79637-9420-4da1-8ffb-9c916f2b62f3","Type":"ContainerStarted","Data":"6e43e23894225a64572a22c70af670464f3290877e3f8d37c5eae7f5f2d44923"}
Sep 11 23:37:31.978826 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:37:31.978384    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-1fb90b/ipp-route-1fb90b-kserve-687df66fbc-vms2m" event={"ID":"b2c79637-9420-4da1-8ffb-9c916f2b62f3","Type":"ContainerStarted","Data":"22965f356ddbb2d6a72ff6c70f8da250b533e9122f032f73c8e6461cc0871324"}
Sep 11 23:37:32.005228 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:37:32.005146    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-1fb90b/ipp-route-1fb90b-kserve-687df66fbc-vms2m" podStartSLOduration=2.005123213 podStartE2EDuration="2.005123213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 23:37:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:37:32.004959414 +0000 UTC m=+2176.038088631" watchObservedRunningTime="2026-09-11 23:37:32.005123213 +0000 UTC m=+2176.038252431"
Sep 11 23:37:35.186069 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:37:35.186018    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"]
Sep 11 23:37:41.041089 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:37:41.041050    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-1fb90b/ipp-route-1fb90b-kserve-687df66fbc-vms2m"
Sep 11 23:37:41.041530 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:37:41.041106    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-1fb90b/ipp-route-1fb90b-kserve-687df66fbc-vms2m"
Sep 11 23:37:41.054318 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:37:41.054288    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-1fb90b/ipp-route-1fb90b-kserve-687df66fbc-vms2m"
Sep 11 23:37:42.058594 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:37:42.058550    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-1fb90b/ipp-route-1fb90b-kserve-687df66fbc-vms2m"
Sep 11 23:37:48.791997 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:37:48.791961    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"]
Sep 11 23:37:56.449909 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:37:56.449861    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-e231ef-openshift-default-77d77f5c44-bfvjr"]
Sep 11 23:37:56.453678 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:37:56.453653    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-e231ef-openshift-default-77d77f5c44-bfvjr"
Sep 11 23:37:56.456785 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:37:56.456764    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-e231ef-openshift-default-dockercfg-8gmwb\""
Sep 11 23:37:56.468287 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:37:56.468257    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-e231ef-openshift-default-77d77f5c44-bfvjr"]
Sep 11 23:37:56.521661 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:37:56.521611    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d12db36-8bf9-4091-aefa-1b69eb86da59-istio-token\") pod \"e2e-ipp-cleanup-e231ef-openshift-default-77d77f5c44-bfvjr\" (UID: \"0d12db36-8bf9-4091-aefa-1b69eb86da59\") " pod="openshift-ingress/e2e-ipp-cleanup-e231ef-openshift-default-77d77f5c44-bfvjr"
Sep 11 23:37:56.521661 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:37:56.521662    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d12db36-8bf9-4091-aefa-1b69eb86da59-workload-socket\") pod \"e2e-ipp-cleanup-e231ef-openshift-default-77d77f5c44-bfvjr\" (UID: \"0d12db36-8bf9-4091-aefa-1b69eb86da59\") " pod="openshift-ingress/e2e-ipp-cleanup-e231ef-openshift-default-77d77f5c44-bfvjr"
Sep 11 23:37:56.521936 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:37:56.521693    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d12db36-8bf9-4091-aefa-1b69eb86da59-istiod-ca-cert\") pod \"e2e-ipp-cleanup-e231ef-openshift-default-77d77f5c44-bfvjr\" (UID: \"0d12db36-8bf9-4091-aefa-1b69eb86da59\") " pod="openshift-ingress/e2e-ipp-cleanup-e231ef-openshift-default-77d77f5c44-bfvjr"
Sep 11 23:37:56.521936 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:37:56.521748    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d12db36-8bf9-4091-aefa-1b69eb86da59-istio-podinfo\") pod \"e2e-ipp-cleanup-e231ef-openshift-default-77d77f5c44-bfvjr\" (UID: \"0d12db36-8bf9-4091-aefa-1b69eb86da59\") " pod="openshift-ingress/e2e-ipp-cleanup-e231ef-openshift-default-77d77f5c44-bfvjr"
Sep 11 23:37:56.521936 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:37:56.521852    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d12db36-8bf9-4091-aefa-1b69eb86da59-workload-certs\") pod \"e2e-ipp-cleanup-e231ef-openshift-default-77d77f5c44-bfvjr\" (UID: \"0d12db36-8bf9-4091-aefa-1b69eb86da59\") " pod="openshift-ingress/e2e-ipp-cleanup-e231ef-openshift-default-77d77f5c44-bfvjr"
Sep 11 23:37:56.521936 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:37:56.521895    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wn5xl\" (UniqueName: \"kubernetes.io/projected/0d12db36-8bf9-4091-aefa-1b69eb86da59-kube-api-access-wn5xl\") pod \"e2e-ipp-cleanup-e231ef-openshift-default-77d77f5c44-bfvjr\" (UID: \"0d12db36-8bf9-4091-aefa-1b69eb86da59\") " pod="openshift-ingress/e2e-ipp-cleanup-e231ef-openshift-default-77d77f5c44-bfvjr"
Sep 11 23:37:56.522096 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:37:56.521985    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d12db36-8bf9-4091-aefa-1b69eb86da59-credential-socket\") pod \"e2e-ipp-cleanup-e231ef-openshift-default-77d77f5c44-bfvjr\" (UID: \"0d12db36-8bf9-4091-aefa-1b69eb86da59\") " pod="openshift-ingress/e2e-ipp-cleanup-e231ef-openshift-default-77d77f5c44-bfvjr"
Sep 11 23:37:56.522096 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:37:56.522030    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d12db36-8bf9-4091-aefa-1b69eb86da59-istio-envoy\") pod \"e2e-ipp-cleanup-e231ef-openshift-default-77d77f5c44-bfvjr\" (UID: \"0d12db36-8bf9-4091-aefa-1b69eb86da59\") " pod="openshift-ingress/e2e-ipp-cleanup-e231ef-openshift-default-77d77f5c44-bfvjr"
Sep 11 23:37:56.522096 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:37:56.522088    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d12db36-8bf9-4091-aefa-1b69eb86da59-istio-data\") pod \"e2e-ipp-cleanup-e231ef-openshift-default-77d77f5c44-bfvjr\" (UID: \"0d12db36-8bf9-4091-aefa-1b69eb86da59\") " pod="openshift-ingress/e2e-ipp-cleanup-e231ef-openshift-default-77d77f5c44-bfvjr"
Sep 11 23:37:56.622929 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:37:56.622884    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d12db36-8bf9-4091-aefa-1b69eb86da59-istio-envoy\") pod \"e2e-ipp-cleanup-e231ef-openshift-default-77d77f5c44-bfvjr\" (UID: \"0d12db36-8bf9-4091-aefa-1b69eb86da59\") " pod="openshift-ingress/e2e-ipp-cleanup-e231ef-openshift-default-77d77f5c44-bfvjr"
Sep 11 23:37:56.622929 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:37:56.622936    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d12db36-8bf9-4091-aefa-1b69eb86da59-istio-data\") pod \"e2e-ipp-cleanup-e231ef-openshift-default-77d77f5c44-bfvjr\" (UID: \"0d12db36-8bf9-4091-aefa-1b69eb86da59\") " pod="openshift-ingress/e2e-ipp-cleanup-e231ef-openshift-default-77d77f5c44-bfvjr"
Sep 11 23:37:56.623239 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:37:56.622971    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d12db36-8bf9-4091-aefa-1b69eb86da59-istio-token\") pod \"e2e-ipp-cleanup-e231ef-openshift-default-77d77f5c44-bfvjr\" (UID: \"0d12db36-8bf9-4091-aefa-1b69eb86da59\") " pod="openshift-ingress/e2e-ipp-cleanup-e231ef-openshift-default-77d77f5c44-bfvjr"
Sep 11 23:37:56.623239 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:37:56.622993    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d12db36-8bf9-4091-aefa-1b69eb86da59-workload-socket\") pod \"e2e-ipp-cleanup-e231ef-openshift-default-77d77f5c44-bfvjr\" (UID: \"0d12db36-8bf9-4091-aefa-1b69eb86da59\") " pod="openshift-ingress/e2e-ipp-cleanup-e231ef-openshift-default-77d77f5c44-bfvjr"
Sep 11 23:37:56.623239 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:37:56.623010    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d12db36-8bf9-4091-aefa-1b69eb86da59-istiod-ca-cert\") pod \"e2e-ipp-cleanup-e231ef-openshift-default-77d77f5c44-bfvjr\" (UID: \"0d12db36-8bf9-4091-aefa-1b69eb86da59\") " pod="openshift-ingress/e2e-ipp-cleanup-e231ef-openshift-default-77d77f5c44-bfvjr"
Sep 11 23:37:56.623239 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:37:56.623036    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d12db36-8bf9-4091-aefa-1b69eb86da59-istio-podinfo\") pod \"e2e-ipp-cleanup-e231ef-openshift-default-77d77f5c44-bfvjr\" (UID: \"0d12db36-8bf9-4091-aefa-1b69eb86da59\") " pod="openshift-ingress/e2e-ipp-cleanup-e231ef-openshift-default-77d77f5c44-bfvjr"
Sep 11 23:37:56.623239 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:37:56.623100    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d12db36-8bf9-4091-aefa-1b69eb86da59-workload-certs\") pod \"e2e-ipp-cleanup-e231ef-openshift-default-77d77f5c44-bfvjr\" (UID: \"0d12db36-8bf9-4091-aefa-1b69eb86da59\") " pod="openshift-ingress/e2e-ipp-cleanup-e231ef-openshift-default-77d77f5c44-bfvjr"
Sep 11 23:37:56.623239 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:37:56.623136    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn5xl\" (UniqueName: \"kubernetes.io/projected/0d12db36-8bf9-4091-aefa-1b69eb86da59-kube-api-access-wn5xl\") pod \"e2e-ipp-cleanup-e231ef-openshift-default-77d77f5c44-bfvjr\" (UID: \"0d12db36-8bf9-4091-aefa-1b69eb86da59\") " pod="openshift-ingress/e2e-ipp-cleanup-e231ef-openshift-default-77d77f5c44-bfvjr"
Sep 11 23:37:56.623239 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:37:56.623235    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d12db36-8bf9-4091-aefa-1b69eb86da59-credential-socket\") pod \"e2e-ipp-cleanup-e231ef-openshift-default-77d77f5c44-bfvjr\" (UID: \"0d12db36-8bf9-4091-aefa-1b69eb86da59\") " pod="openshift-ingress/e2e-ipp-cleanup-e231ef-openshift-default-77d77f5c44-bfvjr"
Sep 11 23:37:56.623605 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:37:56.623485    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d12db36-8bf9-4091-aefa-1b69eb86da59-istio-data\") pod \"e2e-ipp-cleanup-e231ef-openshift-default-77d77f5c44-bfvjr\" (UID: \"0d12db36-8bf9-4091-aefa-1b69eb86da59\") " pod="openshift-ingress/e2e-ipp-cleanup-e231ef-openshift-default-77d77f5c44-bfvjr"
Sep 11 23:37:56.623605 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:37:56.623593    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d12db36-8bf9-4091-aefa-1b69eb86da59-workload-certs\") pod \"e2e-ipp-cleanup-e231ef-openshift-default-77d77f5c44-bfvjr\" (UID: \"0d12db36-8bf9-4091-aefa-1b69eb86da59\") " pod="openshift-ingress/e2e-ipp-cleanup-e231ef-openshift-default-77d77f5c44-bfvjr"
Sep 11 23:37:56.623707 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:37:56.623610    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d12db36-8bf9-4091-aefa-1b69eb86da59-credential-socket\") pod \"e2e-ipp-cleanup-e231ef-openshift-default-77d77f5c44-bfvjr\" (UID: \"0d12db36-8bf9-4091-aefa-1b69eb86da59\") " pod="openshift-ingress/e2e-ipp-cleanup-e231ef-openshift-default-77d77f5c44-bfvjr"
Sep 11 23:37:56.623707 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:37:56.623690    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d12db36-8bf9-4091-aefa-1b69eb86da59-workload-socket\") pod \"e2e-ipp-cleanup-e231ef-openshift-default-77d77f5c44-bfvjr\" (UID: \"0d12db36-8bf9-4091-aefa-1b69eb86da59\") " pod="openshift-ingress/e2e-ipp-cleanup-e231ef-openshift-default-77d77f5c44-bfvjr"
Sep 11 23:37:56.624143 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:37:56.624117    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d12db36-8bf9-4091-aefa-1b69eb86da59-istiod-ca-cert\") pod \"e2e-ipp-cleanup-e231ef-openshift-default-77d77f5c44-bfvjr\" (UID: \"0d12db36-8bf9-4091-aefa-1b69eb86da59\") " pod="openshift-ingress/e2e-ipp-cleanup-e231ef-openshift-default-77d77f5c44-bfvjr"
Sep 11 23:37:56.625600 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:37:56.625577    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d12db36-8bf9-4091-aefa-1b69eb86da59-istio-envoy\") pod \"e2e-ipp-cleanup-e231ef-openshift-default-77d77f5c44-bfvjr\" (UID: \"0d12db36-8bf9-4091-aefa-1b69eb86da59\") " pod="openshift-ingress/e2e-ipp-cleanup-e231ef-openshift-default-77d77f5c44-bfvjr"
Sep 11 23:37:56.625843 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:37:56.625823    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d12db36-8bf9-4091-aefa-1b69eb86da59-istio-podinfo\") pod \"e2e-ipp-cleanup-e231ef-openshift-default-77d77f5c44-bfvjr\" (UID: \"0d12db36-8bf9-4091-aefa-1b69eb86da59\") " pod="openshift-ingress/e2e-ipp-cleanup-e231ef-openshift-default-77d77f5c44-bfvjr"
Sep 11 23:37:56.631692 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:37:56.631667    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d12db36-8bf9-4091-aefa-1b69eb86da59-istio-token\") pod \"e2e-ipp-cleanup-e231ef-openshift-default-77d77f5c44-bfvjr\" (UID: \"0d12db36-8bf9-4091-aefa-1b69eb86da59\") " pod="openshift-ingress/e2e-ipp-cleanup-e231ef-openshift-default-77d77f5c44-bfvjr"
Sep 11 23:37:56.631967 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:37:56.631946    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn5xl\" (UniqueName: \"kubernetes.io/projected/0d12db36-8bf9-4091-aefa-1b69eb86da59-kube-api-access-wn5xl\") pod \"e2e-ipp-cleanup-e231ef-openshift-default-77d77f5c44-bfvjr\" (UID: \"0d12db36-8bf9-4091-aefa-1b69eb86da59\") " pod="openshift-ingress/e2e-ipp-cleanup-e231ef-openshift-default-77d77f5c44-bfvjr"
Sep 11 23:37:56.765991 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:37:56.765892    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-e231ef-openshift-default-77d77f5c44-bfvjr"
Sep 11 23:37:57.050977 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:37:57.050941    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-e231ef-openshift-default-77d77f5c44-bfvjr"]
Sep 11 23:37:57.051678 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:37:57.051648    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d12db36_8bf9_4091_aefa_1b69eb86da59.slice/crio-600d22cef2caa14decbcacc2436c7452827347d0b9aad238893468fa8c6e09f7 WatchSource:0}: Error finding container 600d22cef2caa14decbcacc2436c7452827347d0b9aad238893468fa8c6e09f7: Status 404 returned error can't find the container with id 600d22cef2caa14decbcacc2436c7452827347d0b9aad238893468fa8c6e09f7
Sep 11 23:37:57.054182 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:37:57.054129    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:37:57.054267 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:37:57.054223    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:37:57.054267 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:37:57.054255    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:37:57.152916 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:37:57.152858    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-e231ef-openshift-default-77d77f5c44-bfvjr" event={"ID":"0d12db36-8bf9-4091-aefa-1b69eb86da59","Type":"ContainerStarted","Data":"53378db500ee74ce0b2ef882893f18f402ee68f6fca682ae957ae2a34278275c"}
Sep 11 23:37:57.153200 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:37:57.152934    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-e231ef-openshift-default-77d77f5c44-bfvjr" event={"ID":"0d12db36-8bf9-4091-aefa-1b69eb86da59","Type":"ContainerStarted","Data":"600d22cef2caa14decbcacc2436c7452827347d0b9aad238893468fa8c6e09f7"}
Sep 11 23:37:57.176621 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:37:57.176564    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-e231ef-openshift-default-77d77f5c44-bfvjr" podStartSLOduration=1.176541791 podStartE2EDuration="1.176541791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 23:37:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:37:57.175216137 +0000 UTC m=+2201.208345354" watchObservedRunningTime="2026-09-11 23:37:57.176541791 +0000 UTC m=+2201.209671009"
Sep 11 23:37:57.766350 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:37:57.766290    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-e231ef-openshift-default-77d77f5c44-bfvjr"
Sep 11 23:37:58.771717 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:37:58.771681    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-e231ef-openshift-default-77d77f5c44-bfvjr"
Sep 11 23:37:59.166055 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:37:59.166027    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-e231ef-openshift-default-77d77f5c44-bfvjr"
Sep 11 23:37:59.167072 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:37:59.167053    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-e231ef-openshift-default-77d77f5c44-bfvjr"
Sep 11 23:38:07.786248 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:07.786211    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"]
Sep 11 23:38:17.113811 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:17.113774    2546 scope.go:117] "RemoveContainer" containerID="b7bfeed57d54ae77d1d8db9504ba90dac133ec27b547aba97086113acfea5f51"
Sep 11 23:38:17.126025 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:17.125992    2546 scope.go:117] "RemoveContainer" containerID="d0fc31256b89259adfbcc8f693c1974724634d5fe2ebaf0d6d337e42adc2bfa9"
Sep 11 23:38:17.136055 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:17.136025    2546 scope.go:117] "RemoveContainer" containerID="2bb70bee19c1f18b364a2d0064b66381a7c017d1932c852bb41c11e000bd6b09"
Sep 11 23:38:18.305820 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:18.305733    2546 generic.go:356] "Generic (PLEG): container finished" podID="0d12db36-8bf9-4091-aefa-1b69eb86da59" containerID="53378db500ee74ce0b2ef882893f18f402ee68f6fca682ae957ae2a34278275c" exitCode=0
Sep 11 23:38:18.305820 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:18.305804    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-e231ef-openshift-default-77d77f5c44-bfvjr" event={"ID":"0d12db36-8bf9-4091-aefa-1b69eb86da59","Type":"ContainerDied","Data":"53378db500ee74ce0b2ef882893f18f402ee68f6fca682ae957ae2a34278275c"}
Sep 11 23:38:18.306301 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:18.306258    2546 scope.go:117] "RemoveContainer" containerID="53378db500ee74ce0b2ef882893f18f402ee68f6fca682ae957ae2a34278275c"
Sep 11 23:38:18.307436 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:18.307409    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:38:18.307513 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:18.307479    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:38:18.307513 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:18.307509    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 11 23:38:19.314115 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:19.314078    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-e231ef-openshift-default-77d77f5c44-bfvjr" event={"ID":"0d12db36-8bf9-4091-aefa-1b69eb86da59","Type":"ContainerStarted","Data":"6db46269965ea0b52982874f2aa1695d27119a038c6d82c5a557a9528a5e26df"}
Sep 11 23:38:19.766453 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:19.766401    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-e231ef-openshift-default-77d77f5c44-bfvjr"
Sep 11 23:38:19.766736 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:19.766711    2546 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-e231ef-openshift-default-77d77f5c44-bfvjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Sep 11 23:38:19.766843 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:19.766784    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-e231ef-openshift-default-77d77f5c44-bfvjr" podUID="0d12db36-8bf9-4091-aefa-1b69eb86da59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Sep 11 23:38:20.767175 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:20.767112    2546 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-e231ef-openshift-default-77d77f5c44-bfvjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Sep 11 23:38:20.767569 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:20.767226    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-e231ef-openshift-default-77d77f5c44-bfvjr" podUID="0d12db36-8bf9-4091-aefa-1b69eb86da59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Sep 11 23:38:21.767286 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:21.767241    2546 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-e231ef-openshift-default-77d77f5c44-bfvjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Sep 11 23:38:21.767691 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:21.767317    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-e231ef-openshift-default-77d77f5c44-bfvjr" podUID="0d12db36-8bf9-4091-aefa-1b69eb86da59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Sep 11 23:38:22.766821 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:22.766782    2546 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-e231ef-openshift-default-77d77f5c44-bfvjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Sep 11 23:38:22.767016 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:22.766844    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-e231ef-openshift-default-77d77f5c44-bfvjr" podUID="0d12db36-8bf9-4091-aefa-1b69eb86da59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Sep 11 23:38:23.766531 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:23.766488    2546 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-e231ef-openshift-default-77d77f5c44-bfvjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Sep 11 23:38:23.767013 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:23.766554    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-e231ef-openshift-default-77d77f5c44-bfvjr" podUID="0d12db36-8bf9-4091-aefa-1b69eb86da59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Sep 11 23:38:24.771258 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:24.771226    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-e231ef-openshift-default-77d77f5c44-bfvjr"
Sep 11 23:38:24.771691 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:24.771613    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-e231ef-openshift-default-77d77f5c44-bfvjr"
Sep 11 23:38:24.772371 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:24.772354    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-e231ef-openshift-default-77d77f5c44-bfvjr"
Sep 11 23:38:29.796945 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:29.796907    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e231ef-h749t"]
Sep 11 23:38:29.800869 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:29.800851    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e231ef-h749t"
Sep 11 23:38:29.804781 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:29.804759    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 23:38:29.804926 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:29.804847    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vrx7m\""
Sep 11 23:38:29.807488 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:29.807465    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e231ef-h749t"]
Sep 11 23:38:29.859146 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:29.859117    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9df8ed2a-1fca-469a-9643-e37f9ba15625-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-e231ef-h749t\" (UID: \"9df8ed2a-1fca-469a-9643-e37f9ba15625\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e231ef-h749t"
Sep 11 23:38:29.960548 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:29.960505    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9df8ed2a-1fca-469a-9643-e37f9ba15625-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-e231ef-h749t\" (UID: \"9df8ed2a-1fca-469a-9643-e37f9ba15625\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e231ef-h749t"
Sep 11 23:38:29.961115 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:29.961095    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9df8ed2a-1fca-469a-9643-e37f9ba15625-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-e231ef-h749t\" (UID: \"9df8ed2a-1fca-469a-9643-e37f9ba15625\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e231ef-h749t"
Sep 11 23:38:30.113873 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:30.113810    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e231ef-h749t"
Sep 11 23:38:30.390409 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:30.390382    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e231ef-h749t"]
Sep 11 23:38:30.390806 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:38:30.390770    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9df8ed2a_1fca_469a_9643_e37f9ba15625.slice/crio-52159782f853d8df98bef4d5a2bb46e9183b02384ffae6b2bb1e45cdefe70ea6 WatchSource:0}: Error finding container 52159782f853d8df98bef4d5a2bb46e9183b02384ffae6b2bb1e45cdefe70ea6: Status 404 returned error can't find the container with id 52159782f853d8df98bef4d5a2bb46e9183b02384ffae6b2bb1e45cdefe70ea6
Sep 11 23:38:30.400236 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:30.400208    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e231ef-h749t" event={"ID":"9df8ed2a-1fca-469a-9643-e37f9ba15625","Type":"ContainerStarted","Data":"52159782f853d8df98bef4d5a2bb46e9183b02384ffae6b2bb1e45cdefe70ea6"}
Sep 11 23:38:31.407990 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:31.407960    2546 generic.go:356] "Generic (PLEG): container finished" podID="9df8ed2a-1fca-469a-9643-e37f9ba15625" containerID="9b1b70be734d74c421ce077ecae27fca28e4309417aa554ee7db1e585846311c" exitCode=0
Sep 11 23:38:31.408439 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:31.408056    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e231ef-h749t" event={"ID":"9df8ed2a-1fca-469a-9643-e37f9ba15625","Type":"ContainerDied","Data":"9b1b70be734d74c421ce077ecae27fca28e4309417aa554ee7db1e585846311c"}
Sep 11 23:38:32.566316 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:32.566285    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e231ef-h749t"
Sep 11 23:38:32.693317 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:32.693217    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9df8ed2a-1fca-469a-9643-e37f9ba15625-maas-api-service-ca\") pod \"9df8ed2a-1fca-469a-9643-e37f9ba15625\" (UID: \"9df8ed2a-1fca-469a-9643-e37f9ba15625\") "
Sep 11 23:38:32.693665 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:32.693639    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9df8ed2a-1fca-469a-9643-e37f9ba15625-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9df8ed2a-1fca-469a-9643-e37f9ba15625" (UID: "9df8ed2a-1fca-469a-9643-e37f9ba15625"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 23:38:32.794277 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:32.794227    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9df8ed2a-1fca-469a-9643-e37f9ba15625-maas-api-service-ca\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:38:33.423519 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:33.423481    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e231ef-h749t"
Sep 11 23:38:33.423519 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:33.423510    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e231ef-h749t" event={"ID":"9df8ed2a-1fca-469a-9643-e37f9ba15625","Type":"ContainerDied","Data":"52159782f853d8df98bef4d5a2bb46e9183b02384ffae6b2bb1e45cdefe70ea6"}
Sep 11 23:38:33.423747 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:33.423560    2546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52159782f853d8df98bef4d5a2bb46e9183b02384ffae6b2bb1e45cdefe70ea6"
Sep 11 23:38:41.166631 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:41.166542    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e231ef-h749t"]
Sep 11 23:38:41.169561 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:41.169535    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e231ef-h749t"]
Sep 11 23:38:42.494630 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:42.494594    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9df8ed2a-1fca-469a-9643-e37f9ba15625" path="/var/lib/kubelet/pods/9df8ed2a-1fca-469a-9643-e37f9ba15625/volumes"
Sep 11 23:38:45.889600 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:45.889562    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"]
Sep 11 23:38:47.652601 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:47.652563    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-e231ef-openshift-default-77d77f5c44-bfvjr"]
Sep 11 23:38:47.653036 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:47.652816    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-e231ef-openshift-default-77d77f5c44-bfvjr" podUID="0d12db36-8bf9-4091-aefa-1b69eb86da59" containerName="istio-proxy" containerID="cri-o://6db46269965ea0b52982874f2aa1695d27119a038c6d82c5a557a9528a5e26df" gracePeriod=30
Sep 11 23:38:52.907480 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:52.907448    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-e231ef-openshift-default-77d77f5c44-bfvjr"
Sep 11 23:38:53.008445 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:53.008355    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d12db36-8bf9-4091-aefa-1b69eb86da59-istio-token\") pod \"0d12db36-8bf9-4091-aefa-1b69eb86da59\" (UID: \"0d12db36-8bf9-4091-aefa-1b69eb86da59\") "
Sep 11 23:38:53.008445 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:53.008418    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d12db36-8bf9-4091-aefa-1b69eb86da59-credential-socket\") pod \"0d12db36-8bf9-4091-aefa-1b69eb86da59\" (UID: \"0d12db36-8bf9-4091-aefa-1b69eb86da59\") "
Sep 11 23:38:53.008681 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:53.008467    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d12db36-8bf9-4091-aefa-1b69eb86da59-istio-envoy\") pod \"0d12db36-8bf9-4091-aefa-1b69eb86da59\" (UID: \"0d12db36-8bf9-4091-aefa-1b69eb86da59\") "
Sep 11 23:38:53.008681 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:53.008491    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d12db36-8bf9-4091-aefa-1b69eb86da59-workload-socket\") pod \"0d12db36-8bf9-4091-aefa-1b69eb86da59\" (UID: \"0d12db36-8bf9-4091-aefa-1b69eb86da59\") "
Sep 11 23:38:53.008681 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:53.008515    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d12db36-8bf9-4091-aefa-1b69eb86da59-istio-data\") pod \"0d12db36-8bf9-4091-aefa-1b69eb86da59\" (UID: \"0d12db36-8bf9-4091-aefa-1b69eb86da59\") "
Sep 11 23:38:53.008681 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:53.008541    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d12db36-8bf9-4091-aefa-1b69eb86da59-workload-certs\") pod \"0d12db36-8bf9-4091-aefa-1b69eb86da59\" (UID: \"0d12db36-8bf9-4091-aefa-1b69eb86da59\") "
Sep 11 23:38:53.008681 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:53.008577    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d12db36-8bf9-4091-aefa-1b69eb86da59-istio-podinfo\") pod \"0d12db36-8bf9-4091-aefa-1b69eb86da59\" (UID: \"0d12db36-8bf9-4091-aefa-1b69eb86da59\") "
Sep 11 23:38:53.008681 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:53.008640    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d12db36-8bf9-4091-aefa-1b69eb86da59-istiod-ca-cert\") pod \"0d12db36-8bf9-4091-aefa-1b69eb86da59\" (UID: \"0d12db36-8bf9-4091-aefa-1b69eb86da59\") "
Sep 11 23:38:53.008681 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:53.008665    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wn5xl\" (UniqueName: \"kubernetes.io/projected/0d12db36-8bf9-4091-aefa-1b69eb86da59-kube-api-access-wn5xl\") pod \"0d12db36-8bf9-4091-aefa-1b69eb86da59\" (UID: \"0d12db36-8bf9-4091-aefa-1b69eb86da59\") "
Sep 11 23:38:53.009048 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:53.008817    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d12db36-8bf9-4091-aefa-1b69eb86da59-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0d12db36-8bf9-4091-aefa-1b69eb86da59" (UID: "0d12db36-8bf9-4091-aefa-1b69eb86da59"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:38:53.009048 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:53.008840    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d12db36-8bf9-4091-aefa-1b69eb86da59-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0d12db36-8bf9-4091-aefa-1b69eb86da59" (UID: "0d12db36-8bf9-4091-aefa-1b69eb86da59"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:38:53.009048 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:53.008832    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d12db36-8bf9-4091-aefa-1b69eb86da59-istio-data" (OuterVolumeSpecName: "istio-data") pod "0d12db36-8bf9-4091-aefa-1b69eb86da59" (UID: "0d12db36-8bf9-4091-aefa-1b69eb86da59"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:38:53.009048 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:53.008997    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d12db36-8bf9-4091-aefa-1b69eb86da59-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0d12db36-8bf9-4091-aefa-1b69eb86da59" (UID: "0d12db36-8bf9-4091-aefa-1b69eb86da59"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:38:53.009455 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:53.009140    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d12db36-8bf9-4091-aefa-1b69eb86da59-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0d12db36-8bf9-4091-aefa-1b69eb86da59" (UID: "0d12db36-8bf9-4091-aefa-1b69eb86da59"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 23:38:53.009455 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:53.009210    2546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d12db36-8bf9-4091-aefa-1b69eb86da59-workload-socket\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:38:53.009455 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:53.009232    2546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d12db36-8bf9-4091-aefa-1b69eb86da59-istio-data\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:38:53.009455 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:53.009246    2546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d12db36-8bf9-4091-aefa-1b69eb86da59-workload-certs\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:38:53.009455 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:53.009259    2546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d12db36-8bf9-4091-aefa-1b69eb86da59-credential-socket\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:38:53.010937 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:53.010911    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d12db36-8bf9-4091-aefa-1b69eb86da59-istio-token" (OuterVolumeSpecName: "istio-token") pod "0d12db36-8bf9-4091-aefa-1b69eb86da59" (UID: "0d12db36-8bf9-4091-aefa-1b69eb86da59"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 23:38:53.011072 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:53.010955    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0d12db36-8bf9-4091-aefa-1b69eb86da59-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0d12db36-8bf9-4091-aefa-1b69eb86da59" (UID: "0d12db36-8bf9-4091-aefa-1b69eb86da59"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 23:38:53.011400 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:53.011377    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d12db36-8bf9-4091-aefa-1b69eb86da59-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0d12db36-8bf9-4091-aefa-1b69eb86da59" (UID: "0d12db36-8bf9-4091-aefa-1b69eb86da59"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:38:53.011462 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:53.011386    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d12db36-8bf9-4091-aefa-1b69eb86da59-kube-api-access-wn5xl" (OuterVolumeSpecName: "kube-api-access-wn5xl") pod "0d12db36-8bf9-4091-aefa-1b69eb86da59" (UID: "0d12db36-8bf9-4091-aefa-1b69eb86da59"). InnerVolumeSpecName "kube-api-access-wn5xl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 23:38:53.110873 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:53.110828    2546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d12db36-8bf9-4091-aefa-1b69eb86da59-istio-podinfo\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:38:53.110873 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:53.110865    2546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d12db36-8bf9-4091-aefa-1b69eb86da59-istiod-ca-cert\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:38:53.110873 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:53.110881    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wn5xl\" (UniqueName: \"kubernetes.io/projected/0d12db36-8bf9-4091-aefa-1b69eb86da59-kube-api-access-wn5xl\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:38:53.111124 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:53.110896    2546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d12db36-8bf9-4091-aefa-1b69eb86da59-istio-token\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:38:53.111124 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:53.110908    2546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d12db36-8bf9-4091-aefa-1b69eb86da59-istio-envoy\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:38:53.554428 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:53.554393    2546 generic.go:356] "Generic (PLEG): container finished" podID="0d12db36-8bf9-4091-aefa-1b69eb86da59" containerID="6db46269965ea0b52982874f2aa1695d27119a038c6d82c5a557a9528a5e26df" exitCode=0
Sep 11 23:38:53.554650 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:53.554448    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-e231ef-openshift-default-77d77f5c44-bfvjr"
Sep 11 23:38:53.554650 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:53.554484    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-e231ef-openshift-default-77d77f5c44-bfvjr" event={"ID":"0d12db36-8bf9-4091-aefa-1b69eb86da59","Type":"ContainerDied","Data":"6db46269965ea0b52982874f2aa1695d27119a038c6d82c5a557a9528a5e26df"}
Sep 11 23:38:53.554650 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:53.554540    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-e231ef-openshift-default-77d77f5c44-bfvjr" event={"ID":"0d12db36-8bf9-4091-aefa-1b69eb86da59","Type":"ContainerDied","Data":"600d22cef2caa14decbcacc2436c7452827347d0b9aad238893468fa8c6e09f7"}
Sep 11 23:38:53.554650 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:53.554564    2546 scope.go:117] "RemoveContainer" containerID="6db46269965ea0b52982874f2aa1695d27119a038c6d82c5a557a9528a5e26df"
Sep 11 23:38:53.565381 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:53.565363    2546 scope.go:117] "RemoveContainer" containerID="53378db500ee74ce0b2ef882893f18f402ee68f6fca682ae957ae2a34278275c"
Sep 11 23:38:53.574570 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:53.574547    2546 scope.go:117] "RemoveContainer" containerID="6db46269965ea0b52982874f2aa1695d27119a038c6d82c5a557a9528a5e26df"
Sep 11 23:38:53.575133 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:38:53.574963    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6db46269965ea0b52982874f2aa1695d27119a038c6d82c5a557a9528a5e26df\": container with ID starting with 6db46269965ea0b52982874f2aa1695d27119a038c6d82c5a557a9528a5e26df not found: ID does not exist" containerID="6db46269965ea0b52982874f2aa1695d27119a038c6d82c5a557a9528a5e26df"
Sep 11 23:38:53.575133 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:53.575019    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6db46269965ea0b52982874f2aa1695d27119a038c6d82c5a557a9528a5e26df"} err="failed to get container status \"6db46269965ea0b52982874f2aa1695d27119a038c6d82c5a557a9528a5e26df\": rpc error: code = NotFound desc = could not find container \"6db46269965ea0b52982874f2aa1695d27119a038c6d82c5a557a9528a5e26df\": container with ID starting with 6db46269965ea0b52982874f2aa1695d27119a038c6d82c5a557a9528a5e26df not found: ID does not exist"
Sep 11 23:38:53.575133 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:53.575045    2546 scope.go:117] "RemoveContainer" containerID="53378db500ee74ce0b2ef882893f18f402ee68f6fca682ae957ae2a34278275c"
Sep 11 23:38:53.575425 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:38:53.575395    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53378db500ee74ce0b2ef882893f18f402ee68f6fca682ae957ae2a34278275c\": container with ID starting with 53378db500ee74ce0b2ef882893f18f402ee68f6fca682ae957ae2a34278275c not found: ID does not exist" containerID="53378db500ee74ce0b2ef882893f18f402ee68f6fca682ae957ae2a34278275c"
Sep 11 23:38:53.575501 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:53.575447    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53378db500ee74ce0b2ef882893f18f402ee68f6fca682ae957ae2a34278275c"} err="failed to get container status \"53378db500ee74ce0b2ef882893f18f402ee68f6fca682ae957ae2a34278275c\": rpc error: code = NotFound desc = could not find container \"53378db500ee74ce0b2ef882893f18f402ee68f6fca682ae957ae2a34278275c\": container with ID starting with 53378db500ee74ce0b2ef882893f18f402ee68f6fca682ae957ae2a34278275c not found: ID does not exist"
Sep 11 23:38:53.578367 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:53.578347    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-e231ef-openshift-default-77d77f5c44-bfvjr"]
Sep 11 23:38:53.582301 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:53.582276    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-e231ef-openshift-default-77d77f5c44-bfvjr"]
Sep 11 23:38:54.494540 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:38:54.494510    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d12db36-8bf9-4091-aefa-1b69eb86da59" path="/var/lib/kubelet/pods/0d12db36-8bf9-4091-aefa-1b69eb86da59/volumes"
Sep 11 23:39:04.691328 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:39:04.691294    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"]
Sep 11 23:39:12.265773 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:39:12.265732    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-1fb90b/ipp-route-1fb90b-kserve-687df66fbc-vms2m"]
Sep 11 23:39:17.732096 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:39:17.732053    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"]
Sep 11 23:39:27.286442 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:39:27.286370    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-1fb90b/ipp-route-1fb90b-kserve-687df66fbc-vms2m" podUID="b2c79637-9420-4da1-8ffb-9c916f2b62f3" containerName="main" containerID="cri-o://6e43e23894225a64572a22c70af670464f3290877e3f8d37c5eae7f5f2d44923" gracePeriod=45
Sep 11 23:39:27.547934 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:39:27.547865    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-1fb90b/ipp-route-1fb90b-kserve-687df66fbc-vms2m"
Sep 11 23:39:27.656121 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:39:27.656080    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2c79637-9420-4da1-8ffb-9c916f2b62f3-tmp-dir\") pod \"b2c79637-9420-4da1-8ffb-9c916f2b62f3\" (UID: \"b2c79637-9420-4da1-8ffb-9c916f2b62f3\") "
Sep 11 23:39:27.656121 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:39:27.656131    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2c79637-9420-4da1-8ffb-9c916f2b62f3-home\") pod \"b2c79637-9420-4da1-8ffb-9c916f2b62f3\" (UID: \"b2c79637-9420-4da1-8ffb-9c916f2b62f3\") "
Sep 11 23:39:27.656385 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:39:27.656200    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2c79637-9420-4da1-8ffb-9c916f2b62f3-tls-certs\") pod \"b2c79637-9420-4da1-8ffb-9c916f2b62f3\" (UID: \"b2c79637-9420-4da1-8ffb-9c916f2b62f3\") "
Sep 11 23:39:27.656385 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:39:27.656256    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2c79637-9420-4da1-8ffb-9c916f2b62f3-dshm\") pod \"b2c79637-9420-4da1-8ffb-9c916f2b62f3\" (UID: \"b2c79637-9420-4da1-8ffb-9c916f2b62f3\") "
Sep 11 23:39:27.656385 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:39:27.656308    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2c79637-9420-4da1-8ffb-9c916f2b62f3-model-cache\") pod \"b2c79637-9420-4da1-8ffb-9c916f2b62f3\" (UID: \"b2c79637-9420-4da1-8ffb-9c916f2b62f3\") "
Sep 11 23:39:27.656385 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:39:27.656344    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g9j44\" (UniqueName: \"kubernetes.io/projected/b2c79637-9420-4da1-8ffb-9c916f2b62f3-kube-api-access-g9j44\") pod \"b2c79637-9420-4da1-8ffb-9c916f2b62f3\" (UID: \"b2c79637-9420-4da1-8ffb-9c916f2b62f3\") "
Sep 11 23:39:27.656578 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:39:27.656444    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2c79637-9420-4da1-8ffb-9c916f2b62f3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b2c79637-9420-4da1-8ffb-9c916f2b62f3" (UID: "b2c79637-9420-4da1-8ffb-9c916f2b62f3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:39:27.656578 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:39:27.656486    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2c79637-9420-4da1-8ffb-9c916f2b62f3-home" (OuterVolumeSpecName: "home") pod "b2c79637-9420-4da1-8ffb-9c916f2b62f3" (UID: "b2c79637-9420-4da1-8ffb-9c916f2b62f3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:39:27.656693 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:39:27.656636    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2c79637-9420-4da1-8ffb-9c916f2b62f3-model-cache" (OuterVolumeSpecName: "model-cache") pod "b2c79637-9420-4da1-8ffb-9c916f2b62f3" (UID: "b2c79637-9420-4da1-8ffb-9c916f2b62f3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:39:27.657251 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:39:27.657224    2546 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2c79637-9420-4da1-8ffb-9c916f2b62f3-model-cache\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:39:27.657251 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:39:27.657245    2546 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2c79637-9420-4da1-8ffb-9c916f2b62f3-tmp-dir\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:39:27.657441 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:39:27.657261    2546 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2c79637-9420-4da1-8ffb-9c916f2b62f3-home\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:39:27.658787 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:39:27.658761    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2c79637-9420-4da1-8ffb-9c916f2b62f3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b2c79637-9420-4da1-8ffb-9c916f2b62f3" (UID: "b2c79637-9420-4da1-8ffb-9c916f2b62f3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 23:39:27.658892 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:39:27.658793    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2c79637-9420-4da1-8ffb-9c916f2b62f3-kube-api-access-g9j44" (OuterVolumeSpecName: "kube-api-access-g9j44") pod "b2c79637-9420-4da1-8ffb-9c916f2b62f3" (UID: "b2c79637-9420-4da1-8ffb-9c916f2b62f3"). InnerVolumeSpecName "kube-api-access-g9j44". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 23:39:27.658892 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:39:27.658834    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2c79637-9420-4da1-8ffb-9c916f2b62f3-dshm" (OuterVolumeSpecName: "dshm") pod "b2c79637-9420-4da1-8ffb-9c916f2b62f3" (UID: "b2c79637-9420-4da1-8ffb-9c916f2b62f3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:39:27.758632 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:39:27.758594    2546 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2c79637-9420-4da1-8ffb-9c916f2b62f3-tls-certs\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:39:27.758632 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:39:27.758626    2546 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2c79637-9420-4da1-8ffb-9c916f2b62f3-dshm\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:39:27.758632 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:39:27.758635    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9j44\" (UniqueName: \"kubernetes.io/projected/b2c79637-9420-4da1-8ffb-9c916f2b62f3-kube-api-access-g9j44\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:39:27.784904 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:39:27.784869    2546 generic.go:356] "Generic (PLEG): container finished" podID="b2c79637-9420-4da1-8ffb-9c916f2b62f3" containerID="6e43e23894225a64572a22c70af670464f3290877e3f8d37c5eae7f5f2d44923" exitCode=0
Sep 11 23:39:27.785106 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:39:27.784918    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-1fb90b/ipp-route-1fb90b-kserve-687df66fbc-vms2m"
Sep 11 23:39:27.785106 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:39:27.784953    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-1fb90b/ipp-route-1fb90b-kserve-687df66fbc-vms2m" event={"ID":"b2c79637-9420-4da1-8ffb-9c916f2b62f3","Type":"ContainerDied","Data":"6e43e23894225a64572a22c70af670464f3290877e3f8d37c5eae7f5f2d44923"}
Sep 11 23:39:27.785106 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:39:27.785007    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-1fb90b/ipp-route-1fb90b-kserve-687df66fbc-vms2m" event={"ID":"b2c79637-9420-4da1-8ffb-9c916f2b62f3","Type":"ContainerDied","Data":"22965f356ddbb2d6a72ff6c70f8da250b533e9122f032f73c8e6461cc0871324"}
Sep 11 23:39:27.785106 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:39:27.785025    2546 scope.go:117] "RemoveContainer" containerID="6e43e23894225a64572a22c70af670464f3290877e3f8d37c5eae7f5f2d44923"
Sep 11 23:39:27.796252 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:39:27.796233    2546 scope.go:117] "RemoveContainer" containerID="6e43e23894225a64572a22c70af670464f3290877e3f8d37c5eae7f5f2d44923"
Sep 11 23:39:27.796603 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:39:27.796577    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e43e23894225a64572a22c70af670464f3290877e3f8d37c5eae7f5f2d44923\": container with ID starting with 6e43e23894225a64572a22c70af670464f3290877e3f8d37c5eae7f5f2d44923 not found: ID does not exist" containerID="6e43e23894225a64572a22c70af670464f3290877e3f8d37c5eae7f5f2d44923"
Sep 11 23:39:27.796686 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:39:27.796612    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e43e23894225a64572a22c70af670464f3290877e3f8d37c5eae7f5f2d44923"} err="failed to get container status \"6e43e23894225a64572a22c70af670464f3290877e3f8d37c5eae7f5f2d44923\": rpc error: code = NotFound desc = could not find container \"6e43e23894225a64572a22c70af670464f3290877e3f8d37c5eae7f5f2d44923\": container with ID starting with 6e43e23894225a64572a22c70af670464f3290877e3f8d37c5eae7f5f2d44923 not found: ID does not exist"
Sep 11 23:39:27.808631 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:39:27.808559    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-1fb90b/ipp-route-1fb90b-kserve-687df66fbc-vms2m"]
Sep 11 23:39:27.814220 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:39:27.814192    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-1fb90b/ipp-route-1fb90b-kserve-687df66fbc-vms2m"]
Sep 11 23:39:28.495581 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:39:28.495547    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2c79637-9420-4da1-8ffb-9c916f2b62f3" path="/var/lib/kubelet/pods/b2c79637-9420-4da1-8ffb-9c916f2b62f3/volumes"
Sep 11 23:40:09.126174 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:09.126061    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl"]
Sep 11 23:40:09.126693 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:09.126338    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl" podUID="ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5" containerName="istio-proxy" containerID="cri-o://872973c8afd0a664a6f2a314cbd6b1b93a803a8f0536f6aa4b416e02086dbcf7" gracePeriod=30
Sep 11 23:40:09.477582 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:09.477536    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-778566-fdz7b"]
Sep 11 23:40:09.478020 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:09.478005    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9df8ed2a-1fca-469a-9643-e37f9ba15625" containerName="revoke-keys"
Sep 11 23:40:09.478066 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:09.478022    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="9df8ed2a-1fca-469a-9643-e37f9ba15625" containerName="revoke-keys"
Sep 11 23:40:09.478066 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:09.478042    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2c79637-9420-4da1-8ffb-9c916f2b62f3" containerName="main"
Sep 11 23:40:09.478066 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:09.478048    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2c79637-9420-4da1-8ffb-9c916f2b62f3" containerName="main"
Sep 11 23:40:09.478066 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:09.478055    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d12db36-8bf9-4091-aefa-1b69eb86da59" containerName="istio-proxy"
Sep 11 23:40:09.478066 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:09.478060    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d12db36-8bf9-4091-aefa-1b69eb86da59" containerName="istio-proxy"
Sep 11 23:40:09.478261 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:09.478127    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d12db36-8bf9-4091-aefa-1b69eb86da59" containerName="istio-proxy"
Sep 11 23:40:09.478261 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:09.478135    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d12db36-8bf9-4091-aefa-1b69eb86da59" containerName="istio-proxy"
Sep 11 23:40:09.478261 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:09.478144    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2c79637-9420-4da1-8ffb-9c916f2b62f3" containerName="main"
Sep 11 23:40:09.478261 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:09.478168    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="9df8ed2a-1fca-469a-9643-e37f9ba15625" containerName="revoke-keys"
Sep 11 23:40:09.481531 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:09.481513    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-778566-fdz7b"
Sep 11 23:40:09.484418 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:09.484388    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vrx7m\""
Sep 11 23:40:09.484565 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:09.484417    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 23:40:09.488302 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:09.488274    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-778566-fdz7b"]
Sep 11 23:40:09.578793 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:09.578758    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/391f5a53-5270-42c5-95b9-668053c07ae0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-778566-fdz7b\" (UID: \"391f5a53-5270-42c5-95b9-668053c07ae0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-778566-fdz7b"
Sep 11 23:40:09.680034 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:09.679991    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/391f5a53-5270-42c5-95b9-668053c07ae0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-778566-fdz7b\" (UID: \"391f5a53-5270-42c5-95b9-668053c07ae0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-778566-fdz7b"
Sep 11 23:40:09.680723 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:09.680702    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/391f5a53-5270-42c5-95b9-668053c07ae0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-778566-fdz7b\" (UID: \"391f5a53-5270-42c5-95b9-668053c07ae0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-778566-fdz7b"
Sep 11 23:40:09.793301 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:09.793220    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-778566-fdz7b"
Sep 11 23:40:10.073008 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:10.072953    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-778566-fdz7b"]
Sep 11 23:40:10.075517 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:40:10.075475    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod391f5a53_5270_42c5_95b9_668053c07ae0.slice/crio-449927a70c8a4e4eb2ed10e669b5ffc69e5da72874fb47ee221425b1b2069aa4 WatchSource:0}: Error finding container 449927a70c8a4e4eb2ed10e669b5ffc69e5da72874fb47ee221425b1b2069aa4: Status 404 returned error can't find the container with id 449927a70c8a4e4eb2ed10e669b5ffc69e5da72874fb47ee221425b1b2069aa4
Sep 11 23:40:11.062385 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:11.062300    2546 generic.go:356] "Generic (PLEG): container finished" podID="391f5a53-5270-42c5-95b9-668053c07ae0" containerID="6768ae84b1352bd083d48da30a81cf471fe1ae38093047eb04c2c237ffd11eb8" exitCode=0
Sep 11 23:40:11.062754 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:11.062388    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-778566-fdz7b" event={"ID":"391f5a53-5270-42c5-95b9-668053c07ae0","Type":"ContainerDied","Data":"6768ae84b1352bd083d48da30a81cf471fe1ae38093047eb04c2c237ffd11eb8"}
Sep 11 23:40:11.062754 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:11.062436    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-778566-fdz7b" event={"ID":"391f5a53-5270-42c5-95b9-668053c07ae0","Type":"ContainerStarted","Data":"449927a70c8a4e4eb2ed10e669b5ffc69e5da72874fb47ee221425b1b2069aa4"}
Sep 11 23:40:12.217466 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:12.217442    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-778566-fdz7b"
Sep 11 23:40:12.304607 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:12.304572    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/391f5a53-5270-42c5-95b9-668053c07ae0-maas-api-service-ca\") pod \"391f5a53-5270-42c5-95b9-668053c07ae0\" (UID: \"391f5a53-5270-42c5-95b9-668053c07ae0\") "
Sep 11 23:40:12.304993 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:12.304958    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/391f5a53-5270-42c5-95b9-668053c07ae0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "391f5a53-5270-42c5-95b9-668053c07ae0" (UID: "391f5a53-5270-42c5-95b9-668053c07ae0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 23:40:12.405708 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:12.405676    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/391f5a53-5270-42c5-95b9-668053c07ae0-maas-api-service-ca\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:40:13.077956 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:13.077925    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-778566-fdz7b"
Sep 11 23:40:13.078180 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:13.077955    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-778566-fdz7b" event={"ID":"391f5a53-5270-42c5-95b9-668053c07ae0","Type":"ContainerDied","Data":"449927a70c8a4e4eb2ed10e669b5ffc69e5da72874fb47ee221425b1b2069aa4"}
Sep 11 23:40:13.078180 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:13.078015    2546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="449927a70c8a4e4eb2ed10e669b5ffc69e5da72874fb47ee221425b1b2069aa4"
Sep 11 23:40:14.392491 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:14.392417    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl"
Sep 11 23:40:14.430835 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:14.430799    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5-istio-data\") pod \"ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5\" (UID: \"ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5\") "
Sep 11 23:40:14.430981 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:14.430895    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5-istiod-ca-cert\") pod \"ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5\" (UID: \"ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5\") "
Sep 11 23:40:14.430981 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:14.430930    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5-istio-envoy\") pod \"ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5\" (UID: \"ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5\") "
Sep 11 23:40:14.430981 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:14.430976    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5-workload-socket\") pod \"ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5\" (UID: \"ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5\") "
Sep 11 23:40:14.431174 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:14.431012    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5-credential-socket\") pod \"ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5\" (UID: \"ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5\") "
Sep 11 23:40:14.431174 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:14.431046    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fjtpk\" (UniqueName: \"kubernetes.io/projected/ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5-kube-api-access-fjtpk\") pod \"ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5\" (UID: \"ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5\") "
Sep 11 23:40:14.431174 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:14.431093    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5-istio-podinfo\") pod \"ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5\" (UID: \"ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5\") "
Sep 11 23:40:14.431174 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:14.431089    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5-istio-data" (OuterVolumeSpecName: "istio-data") pod "ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5" (UID: "ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:40:14.431382 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:14.431206    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5-istio-token\") pod \"ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5\" (UID: \"ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5\") "
Sep 11 23:40:14.431382 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:14.431244    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5-workload-certs\") pod \"ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5\" (UID: \"ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5\") "
Sep 11 23:40:14.431382 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:14.431361    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5" (UID: "ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 23:40:14.431532 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:14.431473    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5" (UID: "ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:40:14.431612 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:14.431587    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5" (UID: "ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:40:14.431838 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:14.431705    2546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5-istio-data\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:40:14.431838 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:14.431796    2546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5-istiod-ca-cert\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:40:14.431838 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:14.431816    2546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5-workload-socket\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:40:14.432105 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:14.431897    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5" (UID: "ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:40:14.433407 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:14.433380    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5" (UID: "ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:40:14.433660 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:14.433643    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5-istio-token" (OuterVolumeSpecName: "istio-token") pod "ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5" (UID: "ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 23:40:14.434066 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:14.434042    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5-kube-api-access-fjtpk" (OuterVolumeSpecName: "kube-api-access-fjtpk") pod "ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5" (UID: "ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5"). InnerVolumeSpecName "kube-api-access-fjtpk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 23:40:14.434206 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:14.434110    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5" (UID: "ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 23:40:14.533593 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:14.533558    2546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5-credential-socket\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:40:14.533593 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:14.533591    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjtpk\" (UniqueName: \"kubernetes.io/projected/ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5-kube-api-access-fjtpk\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:40:14.533593 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:14.533601    2546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5-istio-podinfo\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:40:14.533852 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:14.533611    2546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5-istio-token\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:40:14.533852 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:14.533619    2546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5-workload-certs\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:40:14.533852 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:14.533627    2546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5-istio-envoy\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:40:15.092147 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:15.092114    2546 generic.go:356] "Generic (PLEG): container finished" podID="ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5" containerID="872973c8afd0a664a6f2a314cbd6b1b93a803a8f0536f6aa4b416e02086dbcf7" exitCode=0
Sep 11 23:40:15.092367 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:15.092199    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl"
Sep 11 23:40:15.092367 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:15.092210    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl" event={"ID":"ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5","Type":"ContainerDied","Data":"872973c8afd0a664a6f2a314cbd6b1b93a803a8f0536f6aa4b416e02086dbcf7"}
Sep 11 23:40:15.092367 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:15.092277    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl" event={"ID":"ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5","Type":"ContainerDied","Data":"5af5076642cb5ec8232dc3514c304f70ea7d1d71f134028d9bdf892872217fec"}
Sep 11 23:40:15.092367 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:15.092325    2546 scope.go:117] "RemoveContainer" containerID="872973c8afd0a664a6f2a314cbd6b1b93a803a8f0536f6aa4b416e02086dbcf7"
Sep 11 23:40:15.102249 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:15.102229    2546 scope.go:117] "RemoveContainer" containerID="55dfa5b33089714962a7c194b138be1a069aac7a783097bb3687e3f13e4af9b5"
Sep 11 23:40:15.112057 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:15.112037    2546 scope.go:117] "RemoveContainer" containerID="872973c8afd0a664a6f2a314cbd6b1b93a803a8f0536f6aa4b416e02086dbcf7"
Sep 11 23:40:15.112405 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:40:15.112386    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"872973c8afd0a664a6f2a314cbd6b1b93a803a8f0536f6aa4b416e02086dbcf7\": container with ID starting with 872973c8afd0a664a6f2a314cbd6b1b93a803a8f0536f6aa4b416e02086dbcf7 not found: ID does not exist" containerID="872973c8afd0a664a6f2a314cbd6b1b93a803a8f0536f6aa4b416e02086dbcf7"
Sep 11 23:40:15.112473 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:15.112412    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"872973c8afd0a664a6f2a314cbd6b1b93a803a8f0536f6aa4b416e02086dbcf7"} err="failed to get container status \"872973c8afd0a664a6f2a314cbd6b1b93a803a8f0536f6aa4b416e02086dbcf7\": rpc error: code = NotFound desc = could not find container \"872973c8afd0a664a6f2a314cbd6b1b93a803a8f0536f6aa4b416e02086dbcf7\": container with ID starting with 872973c8afd0a664a6f2a314cbd6b1b93a803a8f0536f6aa4b416e02086dbcf7 not found: ID does not exist"
Sep 11 23:40:15.112473 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:15.112430    2546 scope.go:117] "RemoveContainer" containerID="55dfa5b33089714962a7c194b138be1a069aac7a783097bb3687e3f13e4af9b5"
Sep 11 23:40:15.112694 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:40:15.112680    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55dfa5b33089714962a7c194b138be1a069aac7a783097bb3687e3f13e4af9b5\": container with ID starting with 55dfa5b33089714962a7c194b138be1a069aac7a783097bb3687e3f13e4af9b5 not found: ID does not exist" containerID="55dfa5b33089714962a7c194b138be1a069aac7a783097bb3687e3f13e4af9b5"
Sep 11 23:40:15.112758 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:15.112697    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55dfa5b33089714962a7c194b138be1a069aac7a783097bb3687e3f13e4af9b5"} err="failed to get container status \"55dfa5b33089714962a7c194b138be1a069aac7a783097bb3687e3f13e4af9b5\": rpc error: code = NotFound desc = could not find container \"55dfa5b33089714962a7c194b138be1a069aac7a783097bb3687e3f13e4af9b5\": container with ID starting with 55dfa5b33089714962a7c194b138be1a069aac7a783097bb3687e3f13e4af9b5 not found: ID does not exist"
Sep 11 23:40:15.113979 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:15.113958    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl"]
Sep 11 23:40:15.119518 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:15.119496    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-1fb90b-openshift-default-5749d8ff7d-kt4cl"]
Sep 11 23:40:16.495531 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:16.495496    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5" path="/var/lib/kubelet/pods/ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5/volumes"
Sep 11 23:40:20.175608 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:20.175569    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-778566-fdz7b"]
Sep 11 23:40:20.180357 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:20.180326    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-778566-fdz7b"]
Sep 11 23:40:20.500350 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:20.500266    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="391f5a53-5270-42c5-95b9-668053c07ae0" path="/var/lib/kubelet/pods/391f5a53-5270-42c5-95b9-668053c07ae0/volumes"
Sep 11 23:40:23.896239 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:23.896199    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"]
Sep 11 23:40:26.615902 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:26.615862    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-778566-openshift-default-5f49f5c898-cjpxf"]
Sep 11 23:40:26.616397 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:26.616204    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-778566-openshift-default-5f49f5c898-cjpxf" podUID="e61c589e-76b3-4eef-b717-1262329daba2" containerName="istio-proxy" containerID="cri-o://614ca7d8998a7b0077d25ec359ad8602b43ef3baa69024b88eb2accbfd2c1843" gracePeriod=30
Sep 11 23:40:26.968400 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:26.968365    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1c4628-s7gkp"]
Sep 11 23:40:26.968829 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:26.968816    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5" containerName="istio-proxy"
Sep 11 23:40:26.968879 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:26.968831    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5" containerName="istio-proxy"
Sep 11 23:40:26.968879 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:26.968844    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="391f5a53-5270-42c5-95b9-668053c07ae0" containerName="revoke-keys"
Sep 11 23:40:26.968879 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:26.968850    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="391f5a53-5270-42c5-95b9-668053c07ae0" containerName="revoke-keys"
Sep 11 23:40:26.968879 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:26.968861    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d12db36-8bf9-4091-aefa-1b69eb86da59" containerName="istio-proxy"
Sep 11 23:40:26.968879 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:26.968867    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d12db36-8bf9-4091-aefa-1b69eb86da59" containerName="istio-proxy"
Sep 11 23:40:26.969054 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:26.968956    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="391f5a53-5270-42c5-95b9-668053c07ae0" containerName="revoke-keys"
Sep 11 23:40:26.969054 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:26.968964    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5" containerName="istio-proxy"
Sep 11 23:40:26.969054 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:26.968972    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5" containerName="istio-proxy"
Sep 11 23:40:26.972165 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:26.972141    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1c4628-s7gkp"
Sep 11 23:40:26.974841 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:26.974820    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 23:40:26.974967 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:26.974899    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vrx7m\""
Sep 11 23:40:26.979745 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:26.979722    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1c4628-s7gkp"]
Sep 11 23:40:27.060256 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:27.060217    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76c14d3e-8de1-4cdd-b302-4a01c7d22714-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-1c4628-s7gkp\" (UID: \"76c14d3e-8de1-4cdd-b302-4a01c7d22714\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1c4628-s7gkp"
Sep 11 23:40:27.161526 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:27.161492    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76c14d3e-8de1-4cdd-b302-4a01c7d22714-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-1c4628-s7gkp\" (UID: \"76c14d3e-8de1-4cdd-b302-4a01c7d22714\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1c4628-s7gkp"
Sep 11 23:40:27.162434 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:27.162410    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76c14d3e-8de1-4cdd-b302-4a01c7d22714-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-1c4628-s7gkp\" (UID: \"76c14d3e-8de1-4cdd-b302-4a01c7d22714\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1c4628-s7gkp"
Sep 11 23:40:27.284371 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:27.284280    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1c4628-s7gkp"
Sep 11 23:40:27.567167 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:27.567126    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1c4628-s7gkp"]
Sep 11 23:40:27.569093 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:40:27.569068    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76c14d3e_8de1_4cdd_b302_4a01c7d22714.slice/crio-750bb89f21f49c1560cb65c06007fded2af038eb212acbd1b5a084ac8b3a213c WatchSource:0}: Error finding container 750bb89f21f49c1560cb65c06007fded2af038eb212acbd1b5a084ac8b3a213c: Status 404 returned error can't find the container with id 750bb89f21f49c1560cb65c06007fded2af038eb212acbd1b5a084ac8b3a213c
Sep 11 23:40:28.175467 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:28.175432    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1c4628-s7gkp" event={"ID":"76c14d3e-8de1-4cdd-b302-4a01c7d22714","Type":"ContainerStarted","Data":"1cb931065690e4abcdb8e0e9d638fdbfefc01b6e1f83c48febc72cd36bd56bb8"}
Sep 11 23:40:28.175859 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:28.175487    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1c4628-s7gkp" event={"ID":"76c14d3e-8de1-4cdd-b302-4a01c7d22714","Type":"ContainerStarted","Data":"750bb89f21f49c1560cb65c06007fded2af038eb212acbd1b5a084ac8b3a213c"}
Sep 11 23:40:28.192296 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:28.192245    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1c4628-s7gkp" podStartSLOduration=2.192227549 podStartE2EDuration="2.192227549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 23:40:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:40:28.18984763 +0000 UTC m=+2352.222976850" watchObservedRunningTime="2026-09-11 23:40:28.192227549 +0000 UTC m=+2352.225356765"
Sep 11 23:40:29.183399 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:29.183357    2546 generic.go:356] "Generic (PLEG): container finished" podID="76c14d3e-8de1-4cdd-b302-4a01c7d22714" containerID="1cb931065690e4abcdb8e0e9d638fdbfefc01b6e1f83c48febc72cd36bd56bb8" exitCode=0
Sep 11 23:40:29.183842 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:29.183424    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1c4628-s7gkp" event={"ID":"76c14d3e-8de1-4cdd-b302-4a01c7d22714","Type":"ContainerDied","Data":"1cb931065690e4abcdb8e0e9d638fdbfefc01b6e1f83c48febc72cd36bd56bb8"}
Sep 11 23:40:30.331211 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:30.331186    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1c4628-s7gkp"
Sep 11 23:40:30.499340 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:30.499246    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76c14d3e-8de1-4cdd-b302-4a01c7d22714-maas-api-service-ca\") pod \"76c14d3e-8de1-4cdd-b302-4a01c7d22714\" (UID: \"76c14d3e-8de1-4cdd-b302-4a01c7d22714\") "
Sep 11 23:40:30.499608 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:30.499584    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76c14d3e-8de1-4cdd-b302-4a01c7d22714-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "76c14d3e-8de1-4cdd-b302-4a01c7d22714" (UID: "76c14d3e-8de1-4cdd-b302-4a01c7d22714"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 23:40:30.600741 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:30.600699    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76c14d3e-8de1-4cdd-b302-4a01c7d22714-maas-api-service-ca\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:40:31.197259 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:31.197223    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1c4628-s7gkp"
Sep 11 23:40:31.197436 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:31.197256    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1c4628-s7gkp" event={"ID":"76c14d3e-8de1-4cdd-b302-4a01c7d22714","Type":"ContainerDied","Data":"750bb89f21f49c1560cb65c06007fded2af038eb212acbd1b5a084ac8b3a213c"}
Sep 11 23:40:31.197436 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:31.197313    2546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="750bb89f21f49c1560cb65c06007fded2af038eb212acbd1b5a084ac8b3a213c"
Sep 11 23:40:31.878651 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:31.878587    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-778566-openshift-default-5f49f5c898-cjpxf"
Sep 11 23:40:32.013755 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:32.013695    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e61c589e-76b3-4eef-b717-1262329daba2-istio-podinfo\") pod \"e61c589e-76b3-4eef-b717-1262329daba2\" (UID: \"e61c589e-76b3-4eef-b717-1262329daba2\") "
Sep 11 23:40:32.013755 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:32.013757    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e61c589e-76b3-4eef-b717-1262329daba2-workload-socket\") pod \"e61c589e-76b3-4eef-b717-1262329daba2\" (UID: \"e61c589e-76b3-4eef-b717-1262329daba2\") "
Sep 11 23:40:32.013979 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:32.013840    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b6dbf\" (UniqueName: \"kubernetes.io/projected/e61c589e-76b3-4eef-b717-1262329daba2-kube-api-access-b6dbf\") pod \"e61c589e-76b3-4eef-b717-1262329daba2\" (UID: \"e61c589e-76b3-4eef-b717-1262329daba2\") "
Sep 11 23:40:32.013979 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:32.013880    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e61c589e-76b3-4eef-b717-1262329daba2-istio-envoy\") pod \"e61c589e-76b3-4eef-b717-1262329daba2\" (UID: \"e61c589e-76b3-4eef-b717-1262329daba2\") "
Sep 11 23:40:32.013979 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:32.013908    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e61c589e-76b3-4eef-b717-1262329daba2-credential-socket\") pod \"e61c589e-76b3-4eef-b717-1262329daba2\" (UID: \"e61c589e-76b3-4eef-b717-1262329daba2\") "
Sep 11 23:40:32.013979 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:32.013932    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e61c589e-76b3-4eef-b717-1262329daba2-istio-token\") pod \"e61c589e-76b3-4eef-b717-1262329daba2\" (UID: \"e61c589e-76b3-4eef-b717-1262329daba2\") "
Sep 11 23:40:32.013979 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:32.013977    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e61c589e-76b3-4eef-b717-1262329daba2-istiod-ca-cert\") pod \"e61c589e-76b3-4eef-b717-1262329daba2\" (UID: \"e61c589e-76b3-4eef-b717-1262329daba2\") "
Sep 11 23:40:32.014292 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:32.014003    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e61c589e-76b3-4eef-b717-1262329daba2-istio-data\") pod \"e61c589e-76b3-4eef-b717-1262329daba2\" (UID: \"e61c589e-76b3-4eef-b717-1262329daba2\") "
Sep 11 23:40:32.014292 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:32.014031    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e61c589e-76b3-4eef-b717-1262329daba2-workload-certs\") pod \"e61c589e-76b3-4eef-b717-1262329daba2\" (UID: \"e61c589e-76b3-4eef-b717-1262329daba2\") "
Sep 11 23:40:32.014292 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:32.014120    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e61c589e-76b3-4eef-b717-1262329daba2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e61c589e-76b3-4eef-b717-1262329daba2" (UID: "e61c589e-76b3-4eef-b717-1262329daba2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:40:32.014292 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:32.014275    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e61c589e-76b3-4eef-b717-1262329daba2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e61c589e-76b3-4eef-b717-1262329daba2" (UID: "e61c589e-76b3-4eef-b717-1262329daba2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:40:32.014492 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:32.014374    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e61c589e-76b3-4eef-b717-1262329daba2-istio-data" (OuterVolumeSpecName: "istio-data") pod "e61c589e-76b3-4eef-b717-1262329daba2" (UID: "e61c589e-76b3-4eef-b717-1262329daba2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:40:32.014628 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:32.014408    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e61c589e-76b3-4eef-b717-1262329daba2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e61c589e-76b3-4eef-b717-1262329daba2" (UID: "e61c589e-76b3-4eef-b717-1262329daba2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 23:40:32.014628 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:32.014486    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e61c589e-76b3-4eef-b717-1262329daba2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e61c589e-76b3-4eef-b717-1262329daba2" (UID: "e61c589e-76b3-4eef-b717-1262329daba2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:40:32.014628 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:32.014500    2546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e61c589e-76b3-4eef-b717-1262329daba2-workload-socket\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:40:32.014758 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:32.014652    2546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e61c589e-76b3-4eef-b717-1262329daba2-credential-socket\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:40:32.014758 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:32.014676    2546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e61c589e-76b3-4eef-b717-1262329daba2-istio-data\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:40:32.016247 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:32.016220    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e61c589e-76b3-4eef-b717-1262329daba2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e61c589e-76b3-4eef-b717-1262329daba2" (UID: "e61c589e-76b3-4eef-b717-1262329daba2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 23:40:32.016486 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:32.016471    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e61c589e-76b3-4eef-b717-1262329daba2-istio-token" (OuterVolumeSpecName: "istio-token") pod "e61c589e-76b3-4eef-b717-1262329daba2" (UID: "e61c589e-76b3-4eef-b717-1262329daba2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 23:40:32.016697 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:32.016666    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e61c589e-76b3-4eef-b717-1262329daba2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e61c589e-76b3-4eef-b717-1262329daba2" (UID: "e61c589e-76b3-4eef-b717-1262329daba2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:40:32.016787 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:32.016674    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e61c589e-76b3-4eef-b717-1262329daba2-kube-api-access-b6dbf" (OuterVolumeSpecName: "kube-api-access-b6dbf") pod "e61c589e-76b3-4eef-b717-1262329daba2" (UID: "e61c589e-76b3-4eef-b717-1262329daba2"). InnerVolumeSpecName "kube-api-access-b6dbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 23:40:32.116304 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:32.116264    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6dbf\" (UniqueName: \"kubernetes.io/projected/e61c589e-76b3-4eef-b717-1262329daba2-kube-api-access-b6dbf\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:40:32.116304 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:32.116295    2546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e61c589e-76b3-4eef-b717-1262329daba2-istio-envoy\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:40:32.116304 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:32.116306    2546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e61c589e-76b3-4eef-b717-1262329daba2-istio-token\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:40:32.116304 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:32.116313    2546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e61c589e-76b3-4eef-b717-1262329daba2-istiod-ca-cert\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:40:32.116581 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:32.116321    2546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e61c589e-76b3-4eef-b717-1262329daba2-workload-certs\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:40:32.116581 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:32.116329    2546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e61c589e-76b3-4eef-b717-1262329daba2-istio-podinfo\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:40:32.206340 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:32.206304    2546 generic.go:356] "Generic (PLEG): container finished" podID="e61c589e-76b3-4eef-b717-1262329daba2" containerID="614ca7d8998a7b0077d25ec359ad8602b43ef3baa69024b88eb2accbfd2c1843" exitCode=0
Sep 11 23:40:32.206568 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:32.206359    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-778566-openshift-default-5f49f5c898-cjpxf"
Sep 11 23:40:32.206568 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:32.206434    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-778566-openshift-default-5f49f5c898-cjpxf" event={"ID":"e61c589e-76b3-4eef-b717-1262329daba2","Type":"ContainerDied","Data":"614ca7d8998a7b0077d25ec359ad8602b43ef3baa69024b88eb2accbfd2c1843"}
Sep 11 23:40:32.206568 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:32.206526    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-778566-openshift-default-5f49f5c898-cjpxf" event={"ID":"e61c589e-76b3-4eef-b717-1262329daba2","Type":"ContainerDied","Data":"9976495b6dc5dedc4031dd2bcbc99d6985415b2b93d6dcce6bd82cf009f0c9de"}
Sep 11 23:40:32.206568 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:32.206558    2546 scope.go:117] "RemoveContainer" containerID="614ca7d8998a7b0077d25ec359ad8602b43ef3baa69024b88eb2accbfd2c1843"
Sep 11 23:40:32.217647 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:32.217629    2546 scope.go:117] "RemoveContainer" containerID="309b53403bef3b77ec46382e97efc39d4e94934996ab31898300094af5da6fb5"
Sep 11 23:40:32.227283 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:32.227254    2546 scope.go:117] "RemoveContainer" containerID="614ca7d8998a7b0077d25ec359ad8602b43ef3baa69024b88eb2accbfd2c1843"
Sep 11 23:40:32.227624 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:40:32.227602    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"614ca7d8998a7b0077d25ec359ad8602b43ef3baa69024b88eb2accbfd2c1843\": container with ID starting with 614ca7d8998a7b0077d25ec359ad8602b43ef3baa69024b88eb2accbfd2c1843 not found: ID does not exist" containerID="614ca7d8998a7b0077d25ec359ad8602b43ef3baa69024b88eb2accbfd2c1843"
Sep 11 23:40:32.227723 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:32.227631    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"614ca7d8998a7b0077d25ec359ad8602b43ef3baa69024b88eb2accbfd2c1843"} err="failed to get container status \"614ca7d8998a7b0077d25ec359ad8602b43ef3baa69024b88eb2accbfd2c1843\": rpc error: code = NotFound desc = could not find container \"614ca7d8998a7b0077d25ec359ad8602b43ef3baa69024b88eb2accbfd2c1843\": container with ID starting with 614ca7d8998a7b0077d25ec359ad8602b43ef3baa69024b88eb2accbfd2c1843 not found: ID does not exist"
Sep 11 23:40:32.227723 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:32.227650    2546 scope.go:117] "RemoveContainer" containerID="309b53403bef3b77ec46382e97efc39d4e94934996ab31898300094af5da6fb5"
Sep 11 23:40:32.228071 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:40:32.228043    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"309b53403bef3b77ec46382e97efc39d4e94934996ab31898300094af5da6fb5\": container with ID starting with 309b53403bef3b77ec46382e97efc39d4e94934996ab31898300094af5da6fb5 not found: ID does not exist" containerID="309b53403bef3b77ec46382e97efc39d4e94934996ab31898300094af5da6fb5"
Sep 11 23:40:32.228217 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:32.228082    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"309b53403bef3b77ec46382e97efc39d4e94934996ab31898300094af5da6fb5"} err="failed to get container status \"309b53403bef3b77ec46382e97efc39d4e94934996ab31898300094af5da6fb5\": rpc error: code = NotFound desc = could not find container \"309b53403bef3b77ec46382e97efc39d4e94934996ab31898300094af5da6fb5\": container with ID starting with 309b53403bef3b77ec46382e97efc39d4e94934996ab31898300094af5da6fb5 not found: ID does not exist"
Sep 11 23:40:32.231983 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:32.231962    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-778566-openshift-default-5f49f5c898-cjpxf"]
Sep 11 23:40:32.236061 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:32.236035    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-778566-openshift-default-5f49f5c898-cjpxf"]
Sep 11 23:40:32.495785 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:32.495707    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e61c589e-76b3-4eef-b717-1262329daba2" path="/var/lib/kubelet/pods/e61c589e-76b3-4eef-b717-1262329daba2/volumes"
Sep 11 23:40:37.825998 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:37.825955    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1c4628-s7gkp"]
Sep 11 23:40:37.828931 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:37.828906    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1c4628-s7gkp"]
Sep 11 23:40:38.494992 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:38.494948    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76c14d3e-8de1-4cdd-b302-4a01c7d22714" path="/var/lib/kubelet/pods/76c14d3e-8de1-4cdd-b302-4a01c7d22714/volumes"
Sep 11 23:40:40.686646 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:40.686609    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"]
Sep 11 23:40:44.414575 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:44.414533    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k"]
Sep 11 23:40:44.414997 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:44.414767    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k" podUID="bb3c41dd-091c-4f4e-94e5-55d389088418" containerName="istio-proxy" containerID="cri-o://44e6e308a812f60817f5b350e9f9975444ab25d327b67e84059aa329a97dd152" gracePeriod=30
Sep 11 23:40:47.484360 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:47.484314    2546 patch_prober.go:28] interesting pod/e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 11 23:40:47.484767 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:47.484391    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k" podUID="bb3c41dd-091c-4f4e-94e5-55d389088418" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 23:40:49.669212 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:49.669176    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k"
Sep 11 23:40:49.681104 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:49.681043    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb3c41dd-091c-4f4e-94e5-55d389088418-istio-data\") pod \"bb3c41dd-091c-4f4e-94e5-55d389088418\" (UID: \"bb3c41dd-091c-4f4e-94e5-55d389088418\") "
Sep 11 23:40:49.681104 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:49.681095    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb3c41dd-091c-4f4e-94e5-55d389088418-istio-token\") pod \"bb3c41dd-091c-4f4e-94e5-55d389088418\" (UID: \"bb3c41dd-091c-4f4e-94e5-55d389088418\") "
Sep 11 23:40:49.681297 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:49.681119    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hnw5p\" (UniqueName: \"kubernetes.io/projected/bb3c41dd-091c-4f4e-94e5-55d389088418-kube-api-access-hnw5p\") pod \"bb3c41dd-091c-4f4e-94e5-55d389088418\" (UID: \"bb3c41dd-091c-4f4e-94e5-55d389088418\") "
Sep 11 23:40:49.681297 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:49.681137    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb3c41dd-091c-4f4e-94e5-55d389088418-credential-socket\") pod \"bb3c41dd-091c-4f4e-94e5-55d389088418\" (UID: \"bb3c41dd-091c-4f4e-94e5-55d389088418\") "
Sep 11 23:40:49.681297 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:49.681229    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb3c41dd-091c-4f4e-94e5-55d389088418-istiod-ca-cert\") pod \"bb3c41dd-091c-4f4e-94e5-55d389088418\" (UID: \"bb3c41dd-091c-4f4e-94e5-55d389088418\") "
Sep 11 23:40:49.681297 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:49.681267    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb3c41dd-091c-4f4e-94e5-55d389088418-workload-socket\") pod \"bb3c41dd-091c-4f4e-94e5-55d389088418\" (UID: \"bb3c41dd-091c-4f4e-94e5-55d389088418\") "
Sep 11 23:40:49.681504 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:49.681307    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb3c41dd-091c-4f4e-94e5-55d389088418-workload-certs\") pod \"bb3c41dd-091c-4f4e-94e5-55d389088418\" (UID: \"bb3c41dd-091c-4f4e-94e5-55d389088418\") "
Sep 11 23:40:49.681504 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:49.681335    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb3c41dd-091c-4f4e-94e5-55d389088418-istio-podinfo\") pod \"bb3c41dd-091c-4f4e-94e5-55d389088418\" (UID: \"bb3c41dd-091c-4f4e-94e5-55d389088418\") "
Sep 11 23:40:49.681504 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:49.681346    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb3c41dd-091c-4f4e-94e5-55d389088418-istio-data" (OuterVolumeSpecName: "istio-data") pod "bb3c41dd-091c-4f4e-94e5-55d389088418" (UID: "bb3c41dd-091c-4f4e-94e5-55d389088418"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:40:49.681504 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:49.681385    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb3c41dd-091c-4f4e-94e5-55d389088418-istio-envoy\") pod \"bb3c41dd-091c-4f4e-94e5-55d389088418\" (UID: \"bb3c41dd-091c-4f4e-94e5-55d389088418\") "
Sep 11 23:40:49.681701 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:49.681542    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb3c41dd-091c-4f4e-94e5-55d389088418-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bb3c41dd-091c-4f4e-94e5-55d389088418" (UID: "bb3c41dd-091c-4f4e-94e5-55d389088418"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:40:49.681701 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:49.681555    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb3c41dd-091c-4f4e-94e5-55d389088418-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bb3c41dd-091c-4f4e-94e5-55d389088418" (UID: "bb3c41dd-091c-4f4e-94e5-55d389088418"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:40:49.681701 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:49.681636    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb3c41dd-091c-4f4e-94e5-55d389088418-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bb3c41dd-091c-4f4e-94e5-55d389088418" (UID: "bb3c41dd-091c-4f4e-94e5-55d389088418"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 23:40:49.681847 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:49.681792    2546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb3c41dd-091c-4f4e-94e5-55d389088418-istiod-ca-cert\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:40:49.681847 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:49.681811    2546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb3c41dd-091c-4f4e-94e5-55d389088418-workload-socket\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:40:49.681847 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:49.681826    2546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb3c41dd-091c-4f4e-94e5-55d389088418-workload-certs\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:40:49.681847 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:49.681840    2546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb3c41dd-091c-4f4e-94e5-55d389088418-istio-data\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:40:49.682132 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:49.682103    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb3c41dd-091c-4f4e-94e5-55d389088418-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bb3c41dd-091c-4f4e-94e5-55d389088418" (UID: "bb3c41dd-091c-4f4e-94e5-55d389088418"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:40:49.683590 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:49.683552    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb3c41dd-091c-4f4e-94e5-55d389088418-istio-token" (OuterVolumeSpecName: "istio-token") pod "bb3c41dd-091c-4f4e-94e5-55d389088418" (UID: "bb3c41dd-091c-4f4e-94e5-55d389088418"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 23:40:49.683684 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:49.683659    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb3c41dd-091c-4f4e-94e5-55d389088418-kube-api-access-hnw5p" (OuterVolumeSpecName: "kube-api-access-hnw5p") pod "bb3c41dd-091c-4f4e-94e5-55d389088418" (UID: "bb3c41dd-091c-4f4e-94e5-55d389088418"). InnerVolumeSpecName "kube-api-access-hnw5p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 23:40:49.684215 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:49.684195    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb3c41dd-091c-4f4e-94e5-55d389088418-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bb3c41dd-091c-4f4e-94e5-55d389088418" (UID: "bb3c41dd-091c-4f4e-94e5-55d389088418"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:40:49.684215 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:49.684206    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bb3c41dd-091c-4f4e-94e5-55d389088418-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bb3c41dd-091c-4f4e-94e5-55d389088418" (UID: "bb3c41dd-091c-4f4e-94e5-55d389088418"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 23:40:49.782562 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:49.782529    2546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb3c41dd-091c-4f4e-94e5-55d389088418-istio-podinfo\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:40:49.782562 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:49.782558    2546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb3c41dd-091c-4f4e-94e5-55d389088418-istio-envoy\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:40:49.782562 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:49.782569    2546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb3c41dd-091c-4f4e-94e5-55d389088418-istio-token\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:40:49.782798 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:49.782578    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnw5p\" (UniqueName: \"kubernetes.io/projected/bb3c41dd-091c-4f4e-94e5-55d389088418-kube-api-access-hnw5p\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:40:49.782798 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:49.782587    2546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb3c41dd-091c-4f4e-94e5-55d389088418-credential-socket\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:40:50.329600 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:50.329567    2546 generic.go:356] "Generic (PLEG): container finished" podID="bb3c41dd-091c-4f4e-94e5-55d389088418" containerID="44e6e308a812f60817f5b350e9f9975444ab25d327b67e84059aa329a97dd152" exitCode=0
Sep 11 23:40:50.329801 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:50.329609    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k" event={"ID":"bb3c41dd-091c-4f4e-94e5-55d389088418","Type":"ContainerDied","Data":"44e6e308a812f60817f5b350e9f9975444ab25d327b67e84059aa329a97dd152"}
Sep 11 23:40:50.329801 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:50.329620    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k"
Sep 11 23:40:50.329801 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:50.329653    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k" event={"ID":"bb3c41dd-091c-4f4e-94e5-55d389088418","Type":"ContainerDied","Data":"130ad9cc4bf416a85d4c916ac4787cf031bc7511c12b12d7f31eb1422633c5f8"}
Sep 11 23:40:50.329801 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:50.329675    2546 scope.go:117] "RemoveContainer" containerID="44e6e308a812f60817f5b350e9f9975444ab25d327b67e84059aa329a97dd152"
Sep 11 23:40:50.340221 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:50.340146    2546 scope.go:117] "RemoveContainer" containerID="5f14e88f52446d1883b5f8e093a7154e7a38ed1b80238ce72f2c461a26eed13b"
Sep 11 23:40:50.348991 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:50.348964    2546 scope.go:117] "RemoveContainer" containerID="44e6e308a812f60817f5b350e9f9975444ab25d327b67e84059aa329a97dd152"
Sep 11 23:40:50.349335 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:40:50.349291    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44e6e308a812f60817f5b350e9f9975444ab25d327b67e84059aa329a97dd152\": container with ID starting with 44e6e308a812f60817f5b350e9f9975444ab25d327b67e84059aa329a97dd152 not found: ID does not exist" containerID="44e6e308a812f60817f5b350e9f9975444ab25d327b67e84059aa329a97dd152"
Sep 11 23:40:50.349335 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:50.349322    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44e6e308a812f60817f5b350e9f9975444ab25d327b67e84059aa329a97dd152"} err="failed to get container status \"44e6e308a812f60817f5b350e9f9975444ab25d327b67e84059aa329a97dd152\": rpc error: code = NotFound desc = could not find container \"44e6e308a812f60817f5b350e9f9975444ab25d327b67e84059aa329a97dd152\": container with ID starting with 44e6e308a812f60817f5b350e9f9975444ab25d327b67e84059aa329a97dd152 not found: ID does not exist"
Sep 11 23:40:50.349335 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:50.349340    2546 scope.go:117] "RemoveContainer" containerID="5f14e88f52446d1883b5f8e093a7154e7a38ed1b80238ce72f2c461a26eed13b"
Sep 11 23:40:50.349697 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:40:50.349669    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f14e88f52446d1883b5f8e093a7154e7a38ed1b80238ce72f2c461a26eed13b\": container with ID starting with 5f14e88f52446d1883b5f8e093a7154e7a38ed1b80238ce72f2c461a26eed13b not found: ID does not exist" containerID="5f14e88f52446d1883b5f8e093a7154e7a38ed1b80238ce72f2c461a26eed13b"
Sep 11 23:40:50.349857 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:50.349706    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f14e88f52446d1883b5f8e093a7154e7a38ed1b80238ce72f2c461a26eed13b"} err="failed to get container status \"5f14e88f52446d1883b5f8e093a7154e7a38ed1b80238ce72f2c461a26eed13b\": rpc error: code = NotFound desc = could not find container \"5f14e88f52446d1883b5f8e093a7154e7a38ed1b80238ce72f2c461a26eed13b\": container with ID starting with 5f14e88f52446d1883b5f8e093a7154e7a38ed1b80238ce72f2c461a26eed13b not found: ID does not exist"
Sep 11 23:40:50.352955 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:50.352932    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k"]
Sep 11 23:40:50.358784 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:50.358763    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1c4628-openshift-default-85667b69c6-zw75k"]
Sep 11 23:40:50.494461 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:50.494423    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb3c41dd-091c-4f4e-94e5-55d389088418" path="/var/lib/kubelet/pods/bb3c41dd-091c-4f4e-94e5-55d389088418/volumes"
Sep 11 23:40:57.996355 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:40:57.996310    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"]
Sep 11 23:41:02.151300 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:41:02.151261    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw"]
Sep 11 23:41:02.151724 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:41:02.151561    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw" podUID="8c437d61-e321-42e7-bb35-9dd7119a09ac" containerName="istio-proxy" containerID="cri-o://e602ccf807868d8aba6da15ef037144f5fc9e69ff8660db0ea2429d1b66ba092" gracePeriod=30
Sep 11 23:41:07.414047 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:41:07.414011    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw"
Sep 11 23:41:07.446739 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:41:07.446659    2546 generic.go:356] "Generic (PLEG): container finished" podID="8c437d61-e321-42e7-bb35-9dd7119a09ac" containerID="e602ccf807868d8aba6da15ef037144f5fc9e69ff8660db0ea2429d1b66ba092" exitCode=0
Sep 11 23:41:07.446739 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:41:07.446712    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw"
Sep 11 23:41:07.446938 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:41:07.446743    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw" event={"ID":"8c437d61-e321-42e7-bb35-9dd7119a09ac","Type":"ContainerDied","Data":"e602ccf807868d8aba6da15ef037144f5fc9e69ff8660db0ea2429d1b66ba092"}
Sep 11 23:41:07.446938 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:41:07.446826    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw" event={"ID":"8c437d61-e321-42e7-bb35-9dd7119a09ac","Type":"ContainerDied","Data":"cdd27f0c367022eb40264c25305ee96e8b24724c0e980af8da3190545be8ebdb"}
Sep 11 23:41:07.446938 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:41:07.446852    2546 scope.go:117] "RemoveContainer" containerID="e602ccf807868d8aba6da15ef037144f5fc9e69ff8660db0ea2429d1b66ba092"
Sep 11 23:41:07.456774 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:41:07.456755    2546 scope.go:117] "RemoveContainer" containerID="266652491fac3416440db75183e30f6877b828bc9b7e33b385644284ed3f89fa"
Sep 11 23:41:07.465779 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:41:07.465760    2546 scope.go:117] "RemoveContainer" containerID="e602ccf807868d8aba6da15ef037144f5fc9e69ff8660db0ea2429d1b66ba092"
Sep 11 23:41:07.466065 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:41:07.466046    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e602ccf807868d8aba6da15ef037144f5fc9e69ff8660db0ea2429d1b66ba092\": container with ID starting with e602ccf807868d8aba6da15ef037144f5fc9e69ff8660db0ea2429d1b66ba092 not found: ID does not exist" containerID="e602ccf807868d8aba6da15ef037144f5fc9e69ff8660db0ea2429d1b66ba092"
Sep 11 23:41:07.466138 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:41:07.466075    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e602ccf807868d8aba6da15ef037144f5fc9e69ff8660db0ea2429d1b66ba092"} err="failed to get container status \"e602ccf807868d8aba6da15ef037144f5fc9e69ff8660db0ea2429d1b66ba092\": rpc error: code = NotFound desc = could not find container \"e602ccf807868d8aba6da15ef037144f5fc9e69ff8660db0ea2429d1b66ba092\": container with ID starting with e602ccf807868d8aba6da15ef037144f5fc9e69ff8660db0ea2429d1b66ba092 not found: ID does not exist"
Sep 11 23:41:07.466138 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:41:07.466092    2546 scope.go:117] "RemoveContainer" containerID="266652491fac3416440db75183e30f6877b828bc9b7e33b385644284ed3f89fa"
Sep 11 23:41:07.466408 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:41:07.466389    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"266652491fac3416440db75183e30f6877b828bc9b7e33b385644284ed3f89fa\": container with ID starting with 266652491fac3416440db75183e30f6877b828bc9b7e33b385644284ed3f89fa not found: ID does not exist" containerID="266652491fac3416440db75183e30f6877b828bc9b7e33b385644284ed3f89fa"
Sep 11 23:41:07.466464 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:41:07.466420    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"266652491fac3416440db75183e30f6877b828bc9b7e33b385644284ed3f89fa"} err="failed to get container status \"266652491fac3416440db75183e30f6877b828bc9b7e33b385644284ed3f89fa\": rpc error: code = NotFound desc = could not find container \"266652491fac3416440db75183e30f6877b828bc9b7e33b385644284ed3f89fa\": container with ID starting with 266652491fac3416440db75183e30f6877b828bc9b7e33b385644284ed3f89fa not found: ID does not exist"
Sep 11 23:41:07.560049 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:41:07.560011    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c437d61-e321-42e7-bb35-9dd7119a09ac-workload-certs\") pod \"8c437d61-e321-42e7-bb35-9dd7119a09ac\" (UID: \"8c437d61-e321-42e7-bb35-9dd7119a09ac\") "
Sep 11 23:41:07.560273 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:41:07.560123    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c437d61-e321-42e7-bb35-9dd7119a09ac-workload-socket\") pod \"8c437d61-e321-42e7-bb35-9dd7119a09ac\" (UID: \"8c437d61-e321-42e7-bb35-9dd7119a09ac\") "
Sep 11 23:41:07.560273 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:41:07.560180    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c437d61-e321-42e7-bb35-9dd7119a09ac-istio-data\") pod \"8c437d61-e321-42e7-bb35-9dd7119a09ac\" (UID: \"8c437d61-e321-42e7-bb35-9dd7119a09ac\") "
Sep 11 23:41:07.560273 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:41:07.560224    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c437d61-e321-42e7-bb35-9dd7119a09ac-credential-socket\") pod \"8c437d61-e321-42e7-bb35-9dd7119a09ac\" (UID: \"8c437d61-e321-42e7-bb35-9dd7119a09ac\") "
Sep 11 23:41:07.560273 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:41:07.560270    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c437d61-e321-42e7-bb35-9dd7119a09ac-istiod-ca-cert\") pod \"8c437d61-e321-42e7-bb35-9dd7119a09ac\" (UID: \"8c437d61-e321-42e7-bb35-9dd7119a09ac\") "
Sep 11 23:41:07.560514 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:41:07.560312    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c437d61-e321-42e7-bb35-9dd7119a09ac-istio-envoy\") pod \"8c437d61-e321-42e7-bb35-9dd7119a09ac\" (UID: \"8c437d61-e321-42e7-bb35-9dd7119a09ac\") "
Sep 11 23:41:07.560514 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:41:07.560340    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-44g9l\" (UniqueName: \"kubernetes.io/projected/8c437d61-e321-42e7-bb35-9dd7119a09ac-kube-api-access-44g9l\") pod \"8c437d61-e321-42e7-bb35-9dd7119a09ac\" (UID: \"8c437d61-e321-42e7-bb35-9dd7119a09ac\") "
Sep 11 23:41:07.560514 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:41:07.560327    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c437d61-e321-42e7-bb35-9dd7119a09ac-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8c437d61-e321-42e7-bb35-9dd7119a09ac" (UID: "8c437d61-e321-42e7-bb35-9dd7119a09ac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:41:07.560514 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:41:07.560366    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c437d61-e321-42e7-bb35-9dd7119a09ac-istio-token\") pod \"8c437d61-e321-42e7-bb35-9dd7119a09ac\" (UID: \"8c437d61-e321-42e7-bb35-9dd7119a09ac\") "
Sep 11 23:41:07.560514 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:41:07.560388    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c437d61-e321-42e7-bb35-9dd7119a09ac-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8c437d61-e321-42e7-bb35-9dd7119a09ac" (UID: "8c437d61-e321-42e7-bb35-9dd7119a09ac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:41:07.560514 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:41:07.560461    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c437d61-e321-42e7-bb35-9dd7119a09ac-istio-podinfo\") pod \"8c437d61-e321-42e7-bb35-9dd7119a09ac\" (UID: \"8c437d61-e321-42e7-bb35-9dd7119a09ac\") "
Sep 11 23:41:07.560514 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:41:07.560488    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c437d61-e321-42e7-bb35-9dd7119a09ac-istio-data" (OuterVolumeSpecName: "istio-data") pod "8c437d61-e321-42e7-bb35-9dd7119a09ac" (UID: "8c437d61-e321-42e7-bb35-9dd7119a09ac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:41:07.560514 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:41:07.560508    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c437d61-e321-42e7-bb35-9dd7119a09ac-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8c437d61-e321-42e7-bb35-9dd7119a09ac" (UID: "8c437d61-e321-42e7-bb35-9dd7119a09ac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:41:07.560930 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:41:07.560726    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c437d61-e321-42e7-bb35-9dd7119a09ac-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8c437d61-e321-42e7-bb35-9dd7119a09ac" (UID: "8c437d61-e321-42e7-bb35-9dd7119a09ac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 23:41:07.560930 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:41:07.560856    2546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c437d61-e321-42e7-bb35-9dd7119a09ac-istiod-ca-cert\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:41:07.560930 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:41:07.560875    2546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c437d61-e321-42e7-bb35-9dd7119a09ac-workload-certs\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:41:07.560930 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:41:07.560888    2546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c437d61-e321-42e7-bb35-9dd7119a09ac-workload-socket\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:41:07.560930 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:41:07.560900    2546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c437d61-e321-42e7-bb35-9dd7119a09ac-istio-data\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:41:07.560930 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:41:07.560912    2546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c437d61-e321-42e7-bb35-9dd7119a09ac-credential-socket\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:41:07.562750 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:41:07.562716    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c437d61-e321-42e7-bb35-9dd7119a09ac-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8c437d61-e321-42e7-bb35-9dd7119a09ac" (UID: "8c437d61-e321-42e7-bb35-9dd7119a09ac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:41:07.562876 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:41:07.562750    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c437d61-e321-42e7-bb35-9dd7119a09ac-kube-api-access-44g9l" (OuterVolumeSpecName: "kube-api-access-44g9l") pod "8c437d61-e321-42e7-bb35-9dd7119a09ac" (UID: "8c437d61-e321-42e7-bb35-9dd7119a09ac"). InnerVolumeSpecName "kube-api-access-44g9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 23:41:07.563066 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:41:07.563046    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8c437d61-e321-42e7-bb35-9dd7119a09ac-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8c437d61-e321-42e7-bb35-9dd7119a09ac" (UID: "8c437d61-e321-42e7-bb35-9dd7119a09ac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 23:41:07.563127 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:41:07.563109    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c437d61-e321-42e7-bb35-9dd7119a09ac-istio-token" (OuterVolumeSpecName: "istio-token") pod "8c437d61-e321-42e7-bb35-9dd7119a09ac" (UID: "8c437d61-e321-42e7-bb35-9dd7119a09ac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 23:41:07.661798 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:41:07.661759    2546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c437d61-e321-42e7-bb35-9dd7119a09ac-istio-envoy\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:41:07.661798 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:41:07.661794    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44g9l\" (UniqueName: \"kubernetes.io/projected/8c437d61-e321-42e7-bb35-9dd7119a09ac-kube-api-access-44g9l\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:41:07.661798 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:41:07.661805    2546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c437d61-e321-42e7-bb35-9dd7119a09ac-istio-token\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:41:07.662046 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:41:07.661816    2546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c437d61-e321-42e7-bb35-9dd7119a09ac-istio-podinfo\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:41:07.773063 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:41:07.773028    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw"]
Sep 11 23:41:07.776752 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:41:07.776726    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-da9451-openshift-default-756d5c4ff8-k5htw"]
Sep 11 23:41:07.788823 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:41:07.788798    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"]
Sep 11 23:41:08.494331 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:41:08.494284    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c437d61-e321-42e7-bb35-9dd7119a09ac" path="/var/lib/kubelet/pods/8c437d61-e321-42e7-bb35-9dd7119a09ac/volumes"
Sep 11 23:41:16.739456 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:41:16.739429    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7244_6b0a8af3-932c-4091-b7a0-180fe1751a3a/ovn-acl-logging/0.log"
Sep 11 23:41:16.744740 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:41:16.744713    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7244_6b0a8af3-932c-4091-b7a0-180fe1751a3a/ovn-acl-logging/0.log"
Sep 11 23:42:27.095448 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:42:27.095407    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"]
Sep 11 23:42:52.183844 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:42:52.183804    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"]
Sep 11 23:43:00.280134 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:43:00.280093    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"]
Sep 11 23:43:07.881560 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:43:07.881523    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"]
Sep 11 23:43:17.302677 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:43:17.302646    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"]
Sep 11 23:43:24.783361 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:43:24.783329    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"]
Sep 11 23:43:32.785221 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:43:32.785182    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"]
Sep 11 23:43:47.791187 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:43:47.791133    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"]
Sep 11 23:43:49.445924 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:43:49.445885    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c56f6"]
Sep 11 23:43:49.446475 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:43:49.446205    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c56f6" podUID="a5a83b12-239d-4342-9f10-dc236f6fbb2b" containerName="manager" containerID="cri-o://30c1843721e461c23442e28b54983c16e083b8003fb63b3dc37bfdf1388c0229" gracePeriod=10
Sep 11 23:43:50.600587 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:43:50.600560    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c56f6"
Sep 11 23:43:50.670562 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:43:50.670525    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a5a83b12-239d-4342-9f10-dc236f6fbb2b-extensions-socket-volume\") pod \"a5a83b12-239d-4342-9f10-dc236f6fbb2b\" (UID: \"a5a83b12-239d-4342-9f10-dc236f6fbb2b\") "
Sep 11 23:43:50.670737 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:43:50.670593    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dpdp8\" (UniqueName: \"kubernetes.io/projected/a5a83b12-239d-4342-9f10-dc236f6fbb2b-kube-api-access-dpdp8\") pod \"a5a83b12-239d-4342-9f10-dc236f6fbb2b\" (UID: \"a5a83b12-239d-4342-9f10-dc236f6fbb2b\") "
Sep 11 23:43:50.670876 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:43:50.670852    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5a83b12-239d-4342-9f10-dc236f6fbb2b-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "a5a83b12-239d-4342-9f10-dc236f6fbb2b" (UID: "a5a83b12-239d-4342-9f10-dc236f6fbb2b"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:43:50.672733 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:43:50.672711    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5a83b12-239d-4342-9f10-dc236f6fbb2b-kube-api-access-dpdp8" (OuterVolumeSpecName: "kube-api-access-dpdp8") pod "a5a83b12-239d-4342-9f10-dc236f6fbb2b" (UID: "a5a83b12-239d-4342-9f10-dc236f6fbb2b"). InnerVolumeSpecName "kube-api-access-dpdp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 23:43:50.772054 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:43:50.772017    2546 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a5a83b12-239d-4342-9f10-dc236f6fbb2b-extensions-socket-volume\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:43:50.772054 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:43:50.772047    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpdp8\" (UniqueName: \"kubernetes.io/projected/a5a83b12-239d-4342-9f10-dc236f6fbb2b-kube-api-access-dpdp8\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:43:51.437185 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:43:51.437132    2546 generic.go:356] "Generic (PLEG): container finished" podID="a5a83b12-239d-4342-9f10-dc236f6fbb2b" containerID="30c1843721e461c23442e28b54983c16e083b8003fb63b3dc37bfdf1388c0229" exitCode=0
Sep 11 23:43:51.437370 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:43:51.437199    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c56f6"
Sep 11 23:43:51.437370 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:43:51.437240    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c56f6" event={"ID":"a5a83b12-239d-4342-9f10-dc236f6fbb2b","Type":"ContainerDied","Data":"30c1843721e461c23442e28b54983c16e083b8003fb63b3dc37bfdf1388c0229"}
Sep 11 23:43:51.437370 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:43:51.437294    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c56f6" event={"ID":"a5a83b12-239d-4342-9f10-dc236f6fbb2b","Type":"ContainerDied","Data":"01559d6b88d4ad2dccf16c1332ba4c425688c85ba2fea216ade7cdf0810135e7"}
Sep 11 23:43:51.437370 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:43:51.437316    2546 scope.go:117] "RemoveContainer" containerID="30c1843721e461c23442e28b54983c16e083b8003fb63b3dc37bfdf1388c0229"
Sep 11 23:43:51.450002 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:43:51.448419    2546 scope.go:117] "RemoveContainer" containerID="489dfc2189f4ab2947303b73fff8bbe6dd1bdc12afe712592429797c4478aa27"
Sep 11 23:43:51.458807 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:43:51.458783    2546 scope.go:117] "RemoveContainer" containerID="30c1843721e461c23442e28b54983c16e083b8003fb63b3dc37bfdf1388c0229"
Sep 11 23:43:51.459250 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:43:51.459216    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30c1843721e461c23442e28b54983c16e083b8003fb63b3dc37bfdf1388c0229\": container with ID starting with 30c1843721e461c23442e28b54983c16e083b8003fb63b3dc37bfdf1388c0229 not found: ID does not exist" containerID="30c1843721e461c23442e28b54983c16e083b8003fb63b3dc37bfdf1388c0229"
Sep 11 23:43:51.459374 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:43:51.459255    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30c1843721e461c23442e28b54983c16e083b8003fb63b3dc37bfdf1388c0229"} err="failed to get container status \"30c1843721e461c23442e28b54983c16e083b8003fb63b3dc37bfdf1388c0229\": rpc error: code = NotFound desc = could not find container \"30c1843721e461c23442e28b54983c16e083b8003fb63b3dc37bfdf1388c0229\": container with ID starting with 30c1843721e461c23442e28b54983c16e083b8003fb63b3dc37bfdf1388c0229 not found: ID does not exist"
Sep 11 23:43:51.459374 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:43:51.459280    2546 scope.go:117] "RemoveContainer" containerID="489dfc2189f4ab2947303b73fff8bbe6dd1bdc12afe712592429797c4478aa27"
Sep 11 23:43:51.459595 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:43:51.459569    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"489dfc2189f4ab2947303b73fff8bbe6dd1bdc12afe712592429797c4478aa27\": container with ID starting with 489dfc2189f4ab2947303b73fff8bbe6dd1bdc12afe712592429797c4478aa27 not found: ID does not exist" containerID="489dfc2189f4ab2947303b73fff8bbe6dd1bdc12afe712592429797c4478aa27"
Sep 11 23:43:51.459705 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:43:51.459601    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"489dfc2189f4ab2947303b73fff8bbe6dd1bdc12afe712592429797c4478aa27"} err="failed to get container status \"489dfc2189f4ab2947303b73fff8bbe6dd1bdc12afe712592429797c4478aa27\": rpc error: code = NotFound desc = could not find container \"489dfc2189f4ab2947303b73fff8bbe6dd1bdc12afe712592429797c4478aa27\": container with ID starting with 489dfc2189f4ab2947303b73fff8bbe6dd1bdc12afe712592429797c4478aa27 not found: ID does not exist"
Sep 11 23:43:51.462318 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:43:51.462267    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c56f6"]
Sep 11 23:43:51.465632 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:43:51.465609    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c56f6"]
Sep 11 23:43:52.495010 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:43:52.494972    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5a83b12-239d-4342-9f10-dc236f6fbb2b" path="/var/lib/kubelet/pods/a5a83b12-239d-4342-9f10-dc236f6fbb2b/volumes"
Sep 11 23:45:00.131289 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:00.131246    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819505-9xxxk"]
Sep 11 23:45:00.131878 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:00.131819    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c437d61-e321-42e7-bb35-9dd7119a09ac" containerName="istio-proxy"
Sep 11 23:45:00.131878 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:00.131839    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c437d61-e321-42e7-bb35-9dd7119a09ac" containerName="istio-proxy"
Sep 11 23:45:00.131878 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:00.131849    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e61c589e-76b3-4eef-b717-1262329daba2" containerName="istio-proxy"
Sep 11 23:45:00.131878 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:00.131858    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="e61c589e-76b3-4eef-b717-1262329daba2" containerName="istio-proxy"
Sep 11 23:45:00.131878 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:00.131872    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76c14d3e-8de1-4cdd-b302-4a01c7d22714" containerName="revoke-keys"
Sep 11 23:45:00.131878 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:00.131881    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="76c14d3e-8de1-4cdd-b302-4a01c7d22714" containerName="revoke-keys"
Sep 11 23:45:00.132225 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:00.131902    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5a83b12-239d-4342-9f10-dc236f6fbb2b" containerName="manager"
Sep 11 23:45:00.132225 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:00.131910    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5a83b12-239d-4342-9f10-dc236f6fbb2b" containerName="manager"
Sep 11 23:45:00.132225 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:00.131919    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e61c589e-76b3-4eef-b717-1262329daba2" containerName="istio-proxy"
Sep 11 23:45:00.132225 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:00.131929    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="e61c589e-76b3-4eef-b717-1262329daba2" containerName="istio-proxy"
Sep 11 23:45:00.132225 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:00.131950    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb3c41dd-091c-4f4e-94e5-55d389088418" containerName="istio-proxy"
Sep 11 23:45:00.132225 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:00.131958    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb3c41dd-091c-4f4e-94e5-55d389088418" containerName="istio-proxy"
Sep 11 23:45:00.132225 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:00.131965    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5a83b12-239d-4342-9f10-dc236f6fbb2b" containerName="manager"
Sep 11 23:45:00.132225 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:00.131973    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5a83b12-239d-4342-9f10-dc236f6fbb2b" containerName="manager"
Sep 11 23:45:00.132225 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:00.131986    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb3c41dd-091c-4f4e-94e5-55d389088418" containerName="istio-proxy"
Sep 11 23:45:00.132225 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:00.131994    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb3c41dd-091c-4f4e-94e5-55d389088418" containerName="istio-proxy"
Sep 11 23:45:00.132225 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:00.132003    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5" containerName="istio-proxy"
Sep 11 23:45:00.132225 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:00.132011    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddd94cab-ad4d-4204-90b3-ba5d48d3e3c5" containerName="istio-proxy"
Sep 11 23:45:00.132225 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:00.132124    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c437d61-e321-42e7-bb35-9dd7119a09ac" containerName="istio-proxy"
Sep 11 23:45:00.132225 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:00.132138    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb3c41dd-091c-4f4e-94e5-55d389088418" containerName="istio-proxy"
Sep 11 23:45:00.132225 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:00.132146    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5a83b12-239d-4342-9f10-dc236f6fbb2b" containerName="manager"
Sep 11 23:45:00.132225 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:00.132175    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="e61c589e-76b3-4eef-b717-1262329daba2" containerName="istio-proxy"
Sep 11 23:45:00.132225 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:00.132185    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="76c14d3e-8de1-4cdd-b302-4a01c7d22714" containerName="revoke-keys"
Sep 11 23:45:00.132225 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:00.132198    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5a83b12-239d-4342-9f10-dc236f6fbb2b" containerName="manager"
Sep 11 23:45:00.135555 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:00.135535    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819505-9xxxk"
Sep 11 23:45:00.138462 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:00.138426    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vrx7m\""
Sep 11 23:45:00.140052 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:00.140027    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819505-9xxxk"]
Sep 11 23:45:00.146908 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:00.146892    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819505-9xxxk"
Sep 11 23:45:00.421470 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:00.421440    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819505-9xxxk"]
Sep 11 23:45:00.422826 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:45:00.422787    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3853d1b_2934_45ce_b67e_d167d927b854.slice/crio-407f4c8007e7c9ada145bb4d08fddaa9816c49b4a5674fa57afdebd67d00a321 WatchSource:0}: Error finding container 407f4c8007e7c9ada145bb4d08fddaa9816c49b4a5674fa57afdebd67d00a321: Status 404 returned error can't find the container with id 407f4c8007e7c9ada145bb4d08fddaa9816c49b4a5674fa57afdebd67d00a321
Sep 11 23:45:00.852615 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:00.852576    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819505-9xxxk" event={"ID":"a3853d1b-2934-45ce-b67e-d167d927b854","Type":"ContainerStarted","Data":"1dddcab0e9c7c7e9e1d9f0bd3889ca5c32094114e84516463fbc86ae351d41f5"}
Sep 11 23:45:00.852802 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:00.852636    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819505-9xxxk" event={"ID":"a3853d1b-2934-45ce-b67e-d167d927b854","Type":"ContainerStarted","Data":"407f4c8007e7c9ada145bb4d08fddaa9816c49b4a5674fa57afdebd67d00a321"}
Sep 11 23:45:00.869606 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:00.869546    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819505-9xxxk" podStartSLOduration=0.869529684 podStartE2EDuration="869.529684ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:45:00.868313513 +0000 UTC m=+2624.901442731" watchObservedRunningTime="2026-09-11 23:45:00.869529684 +0000 UTC m=+2624.902658900"
Sep 11 23:45:01.859775 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:01.859730    2546 generic.go:356] "Generic (PLEG): container finished" podID="a3853d1b-2934-45ce-b67e-d167d927b854" containerID="1dddcab0e9c7c7e9e1d9f0bd3889ca5c32094114e84516463fbc86ae351d41f5" exitCode=0
Sep 11 23:45:01.860189 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:01.859822    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819505-9xxxk" event={"ID":"a3853d1b-2934-45ce-b67e-d167d927b854","Type":"ContainerDied","Data":"1dddcab0e9c7c7e9e1d9f0bd3889ca5c32094114e84516463fbc86ae351d41f5"}
Sep 11 23:45:02.992807 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:02.992774    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819505-9xxxk"
Sep 11 23:45:03.871855 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:03.871823    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819505-9xxxk"
Sep 11 23:45:03.872033 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:03.871857    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819505-9xxxk" event={"ID":"a3853d1b-2934-45ce-b67e-d167d927b854","Type":"ContainerDied","Data":"407f4c8007e7c9ada145bb4d08fddaa9816c49b4a5674fa57afdebd67d00a321"}
Sep 11 23:45:03.872033 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:03.871902    2546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="407f4c8007e7c9ada145bb4d08fddaa9816c49b4a5674fa57afdebd67d00a321"
Sep 11 23:45:13.128531 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:13.128502    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"]
Sep 11 23:45:17.277563 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:17.277532    2546 scope.go:117] "RemoveContainer" containerID="9b1b70be734d74c421ce077ecae27fca28e4309417aa554ee7db1e585846311c"
Sep 11 23:45:20.223935 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:20.223902    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"]
Sep 11 23:45:28.822630 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:28.822596    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"]
Sep 11 23:45:38.519438 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:38.519399    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-a3c98c31-kserve-d9b9855cf-bfxbx"]
Sep 11 23:45:38.519997 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:38.519974    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c437d61-e321-42e7-bb35-9dd7119a09ac" containerName="istio-proxy"
Sep 11 23:45:38.520050 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:38.520003    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c437d61-e321-42e7-bb35-9dd7119a09ac" containerName="istio-proxy"
Sep 11 23:45:38.520050 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:38.520021    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3853d1b-2934-45ce-b67e-d167d927b854" containerName="cleanup"
Sep 11 23:45:38.520050 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:38.520029    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3853d1b-2934-45ce-b67e-d167d927b854" containerName="cleanup"
Sep 11 23:45:38.520177 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:38.520132    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="e61c589e-76b3-4eef-b717-1262329daba2" containerName="istio-proxy"
Sep 11 23:45:38.520177 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:38.520145    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c437d61-e321-42e7-bb35-9dd7119a09ac" containerName="istio-proxy"
Sep 11 23:45:38.520177 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:38.520169    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3853d1b-2934-45ce-b67e-d167d927b854" containerName="cleanup"
Sep 11 23:45:38.520177 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:38.520177    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb3c41dd-091c-4f4e-94e5-55d389088418" containerName="istio-proxy"
Sep 11 23:45:38.523476 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:38.523456    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-a3c98c31-kserve-d9b9855cf-bfxbx"
Sep 11 23:45:38.526317 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:38.526295    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-a3c98c31-kserve-self-signed-certs\""
Sep 11 23:45:38.534477 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:38.534455    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-a3c98c31-kserve-d9b9855cf-bfxbx"]
Sep 11 23:45:38.538815 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:38.538787    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/deb1a639-8342-4227-8b82-9a180cb13e47-tls-certs\") pod \"e2e-same-id-a-a3c98c31-kserve-d9b9855cf-bfxbx\" (UID: \"deb1a639-8342-4227-8b82-9a180cb13e47\") " pod="llm/e2e-same-id-a-a3c98c31-kserve-d9b9855cf-bfxbx"
Sep 11 23:45:38.538904 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:38.538874    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/deb1a639-8342-4227-8b82-9a180cb13e47-dshm\") pod \"e2e-same-id-a-a3c98c31-kserve-d9b9855cf-bfxbx\" (UID: \"deb1a639-8342-4227-8b82-9a180cb13e47\") " pod="llm/e2e-same-id-a-a3c98c31-kserve-d9b9855cf-bfxbx"
Sep 11 23:45:38.538959 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:38.538909    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgg7c\" (UniqueName: \"kubernetes.io/projected/deb1a639-8342-4227-8b82-9a180cb13e47-kube-api-access-kgg7c\") pod \"e2e-same-id-a-a3c98c31-kserve-d9b9855cf-bfxbx\" (UID: \"deb1a639-8342-4227-8b82-9a180cb13e47\") " pod="llm/e2e-same-id-a-a3c98c31-kserve-d9b9855cf-bfxbx"
Sep 11 23:45:38.539020 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:38.538981    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/deb1a639-8342-4227-8b82-9a180cb13e47-tmp-dir\") pod \"e2e-same-id-a-a3c98c31-kserve-d9b9855cf-bfxbx\" (UID: \"deb1a639-8342-4227-8b82-9a180cb13e47\") " pod="llm/e2e-same-id-a-a3c98c31-kserve-d9b9855cf-bfxbx"
Sep 11 23:45:38.539080 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:38.539028    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/deb1a639-8342-4227-8b82-9a180cb13e47-home\") pod \"e2e-same-id-a-a3c98c31-kserve-d9b9855cf-bfxbx\" (UID: \"deb1a639-8342-4227-8b82-9a180cb13e47\") " pod="llm/e2e-same-id-a-a3c98c31-kserve-d9b9855cf-bfxbx"
Sep 11 23:45:38.539132 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:38.539079    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/deb1a639-8342-4227-8b82-9a180cb13e47-model-cache\") pod \"e2e-same-id-a-a3c98c31-kserve-d9b9855cf-bfxbx\" (UID: \"deb1a639-8342-4227-8b82-9a180cb13e47\") " pod="llm/e2e-same-id-a-a3c98c31-kserve-d9b9855cf-bfxbx"
Sep 11 23:45:38.639918 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:38.639892    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/deb1a639-8342-4227-8b82-9a180cb13e47-tmp-dir\") pod \"e2e-same-id-a-a3c98c31-kserve-d9b9855cf-bfxbx\" (UID: \"deb1a639-8342-4227-8b82-9a180cb13e47\") " pod="llm/e2e-same-id-a-a3c98c31-kserve-d9b9855cf-bfxbx"
Sep 11 23:45:38.639918 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:38.639922    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/deb1a639-8342-4227-8b82-9a180cb13e47-home\") pod \"e2e-same-id-a-a3c98c31-kserve-d9b9855cf-bfxbx\" (UID: \"deb1a639-8342-4227-8b82-9a180cb13e47\") " pod="llm/e2e-same-id-a-a3c98c31-kserve-d9b9855cf-bfxbx"
Sep 11 23:45:38.640132 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:38.639959    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/deb1a639-8342-4227-8b82-9a180cb13e47-model-cache\") pod \"e2e-same-id-a-a3c98c31-kserve-d9b9855cf-bfxbx\" (UID: \"deb1a639-8342-4227-8b82-9a180cb13e47\") " pod="llm/e2e-same-id-a-a3c98c31-kserve-d9b9855cf-bfxbx"
Sep 11 23:45:38.640132 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:38.640019    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/deb1a639-8342-4227-8b82-9a180cb13e47-tls-certs\") pod \"e2e-same-id-a-a3c98c31-kserve-d9b9855cf-bfxbx\" (UID: \"deb1a639-8342-4227-8b82-9a180cb13e47\") " pod="llm/e2e-same-id-a-a3c98c31-kserve-d9b9855cf-bfxbx"
Sep 11 23:45:38.640132 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:38.640074    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/deb1a639-8342-4227-8b82-9a180cb13e47-dshm\") pod \"e2e-same-id-a-a3c98c31-kserve-d9b9855cf-bfxbx\" (UID: \"deb1a639-8342-4227-8b82-9a180cb13e47\") " pod="llm/e2e-same-id-a-a3c98c31-kserve-d9b9855cf-bfxbx"
Sep 11 23:45:38.640132 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:38.640104    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgg7c\" (UniqueName: \"kubernetes.io/projected/deb1a639-8342-4227-8b82-9a180cb13e47-kube-api-access-kgg7c\") pod \"e2e-same-id-a-a3c98c31-kserve-d9b9855cf-bfxbx\" (UID: \"deb1a639-8342-4227-8b82-9a180cb13e47\") " pod="llm/e2e-same-id-a-a3c98c31-kserve-d9b9855cf-bfxbx"
Sep 11 23:45:38.640371 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:38.640348    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/deb1a639-8342-4227-8b82-9a180cb13e47-tmp-dir\") pod \"e2e-same-id-a-a3c98c31-kserve-d9b9855cf-bfxbx\" (UID: \"deb1a639-8342-4227-8b82-9a180cb13e47\") " pod="llm/e2e-same-id-a-a3c98c31-kserve-d9b9855cf-bfxbx"
Sep 11 23:45:38.640431 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:38.640401    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/deb1a639-8342-4227-8b82-9a180cb13e47-home\") pod \"e2e-same-id-a-a3c98c31-kserve-d9b9855cf-bfxbx\" (UID: \"deb1a639-8342-4227-8b82-9a180cb13e47\") " pod="llm/e2e-same-id-a-a3c98c31-kserve-d9b9855cf-bfxbx"
Sep 11 23:45:38.640474 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:38.640430    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/deb1a639-8342-4227-8b82-9a180cb13e47-model-cache\") pod \"e2e-same-id-a-a3c98c31-kserve-d9b9855cf-bfxbx\" (UID: \"deb1a639-8342-4227-8b82-9a180cb13e47\") " pod="llm/e2e-same-id-a-a3c98c31-kserve-d9b9855cf-bfxbx"
Sep 11 23:45:38.642453 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:38.642428    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/deb1a639-8342-4227-8b82-9a180cb13e47-dshm\") pod \"e2e-same-id-a-a3c98c31-kserve-d9b9855cf-bfxbx\" (UID: \"deb1a639-8342-4227-8b82-9a180cb13e47\") " pod="llm/e2e-same-id-a-a3c98c31-kserve-d9b9855cf-bfxbx"
Sep 11 23:45:38.642615 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:38.642598    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/deb1a639-8342-4227-8b82-9a180cb13e47-tls-certs\") pod \"e2e-same-id-a-a3c98c31-kserve-d9b9855cf-bfxbx\" (UID: \"deb1a639-8342-4227-8b82-9a180cb13e47\") " pod="llm/e2e-same-id-a-a3c98c31-kserve-d9b9855cf-bfxbx"
Sep 11 23:45:38.649424 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:38.649401    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgg7c\" (UniqueName: \"kubernetes.io/projected/deb1a639-8342-4227-8b82-9a180cb13e47-kube-api-access-kgg7c\") pod \"e2e-same-id-a-a3c98c31-kserve-d9b9855cf-bfxbx\" (UID: \"deb1a639-8342-4227-8b82-9a180cb13e47\") " pod="llm/e2e-same-id-a-a3c98c31-kserve-d9b9855cf-bfxbx"
Sep 11 23:45:38.835364 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:38.835273    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-a3c98c31-kserve-d9b9855cf-bfxbx"
Sep 11 23:45:39.109323 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:39.109280    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-a3c98c31-kserve-d9b9855cf-bfxbx"]
Sep 11 23:45:39.111542 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:45:39.111514    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddeb1a639_8342_4227_8b82_9a180cb13e47.slice/crio-5aeea312e9449a2c425524002d3503452a7497216dc2f3549d79656aaf957d44 WatchSource:0}: Error finding container 5aeea312e9449a2c425524002d3503452a7497216dc2f3549d79656aaf957d44: Status 404 returned error can't find the container with id 5aeea312e9449a2c425524002d3503452a7497216dc2f3549d79656aaf957d44
Sep 11 23:45:40.084174 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:40.084113    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-a3c98c31-kserve-d9b9855cf-bfxbx" event={"ID":"deb1a639-8342-4227-8b82-9a180cb13e47","Type":"ContainerStarted","Data":"ef0ea1256f3c868e97ed25433e6e227a55570fec1b47cf84de17cd7db0c7dcb1"}
Sep 11 23:45:40.084567 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:40.084188    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-a3c98c31-kserve-d9b9855cf-bfxbx" event={"ID":"deb1a639-8342-4227-8b82-9a180cb13e47","Type":"ContainerStarted","Data":"5aeea312e9449a2c425524002d3503452a7497216dc2f3549d79656aaf957d44"}
Sep 11 23:45:40.103017 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:40.102967    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-a3c98c31-kserve-d9b9855cf-bfxbx" podStartSLOduration=2.102953153 podStartE2EDuration="2.102953153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 23:45:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:45:40.10101432 +0000 UTC m=+2664.134143527" watchObservedRunningTime="2026-09-11 23:45:40.102953153 +0000 UTC m=+2664.136082414"
Sep 11 23:45:40.127729 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:40.127701    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"]
Sep 11 23:45:48.835546 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:48.835495    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-a3c98c31-kserve-d9b9855cf-bfxbx"
Sep 11 23:45:48.835546 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:48.835557    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-a3c98c31-kserve-d9b9855cf-bfxbx"
Sep 11 23:45:48.848223 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:48.848196    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-a3c98c31-kserve-d9b9855cf-bfxbx"
Sep 11 23:45:49.146388 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:49.146300    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-a3c98c31-kserve-d9b9855cf-bfxbx"
Sep 11 23:45:55.326124 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:55.326073    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-a3c98c31-kserve-bff9c879-h5x5m"]
Sep 11 23:45:55.329774 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:55.329753    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-a3c98c31-kserve-bff9c879-h5x5m"
Sep 11 23:45:55.332436 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:55.332415    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-a3c98c31-kserve-self-signed-certs\""
Sep 11 23:45:55.339664 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:55.339643    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-a3c98c31-kserve-bff9c879-h5x5m"]
Sep 11 23:45:55.382752 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:55.382723    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/517e687e-3f3f-4e0a-b08d-3dd66cdcf835-model-cache\") pod \"e2e-same-id-b-a3c98c31-kserve-bff9c879-h5x5m\" (UID: \"517e687e-3f3f-4e0a-b08d-3dd66cdcf835\") " pod="llm/e2e-same-id-b-a3c98c31-kserve-bff9c879-h5x5m"
Sep 11 23:45:55.382923 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:55.382818    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/517e687e-3f3f-4e0a-b08d-3dd66cdcf835-tls-certs\") pod \"e2e-same-id-b-a3c98c31-kserve-bff9c879-h5x5m\" (UID: \"517e687e-3f3f-4e0a-b08d-3dd66cdcf835\") " pod="llm/e2e-same-id-b-a3c98c31-kserve-bff9c879-h5x5m"
Sep 11 23:45:55.382923 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:55.382850    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/517e687e-3f3f-4e0a-b08d-3dd66cdcf835-tmp-dir\") pod \"e2e-same-id-b-a3c98c31-kserve-bff9c879-h5x5m\" (UID: \"517e687e-3f3f-4e0a-b08d-3dd66cdcf835\") " pod="llm/e2e-same-id-b-a3c98c31-kserve-bff9c879-h5x5m"
Sep 11 23:45:55.382923 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:55.382877    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/517e687e-3f3f-4e0a-b08d-3dd66cdcf835-home\") pod \"e2e-same-id-b-a3c98c31-kserve-bff9c879-h5x5m\" (UID: \"517e687e-3f3f-4e0a-b08d-3dd66cdcf835\") " pod="llm/e2e-same-id-b-a3c98c31-kserve-bff9c879-h5x5m"
Sep 11 23:45:55.382923 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:55.382900    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/517e687e-3f3f-4e0a-b08d-3dd66cdcf835-dshm\") pod \"e2e-same-id-b-a3c98c31-kserve-bff9c879-h5x5m\" (UID: \"517e687e-3f3f-4e0a-b08d-3dd66cdcf835\") " pod="llm/e2e-same-id-b-a3c98c31-kserve-bff9c879-h5x5m"
Sep 11 23:45:55.383127 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:55.382935    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpb42\" (UniqueName: \"kubernetes.io/projected/517e687e-3f3f-4e0a-b08d-3dd66cdcf835-kube-api-access-wpb42\") pod \"e2e-same-id-b-a3c98c31-kserve-bff9c879-h5x5m\" (UID: \"517e687e-3f3f-4e0a-b08d-3dd66cdcf835\") " pod="llm/e2e-same-id-b-a3c98c31-kserve-bff9c879-h5x5m"
Sep 11 23:45:55.484284 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:55.484251    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/517e687e-3f3f-4e0a-b08d-3dd66cdcf835-tls-certs\") pod \"e2e-same-id-b-a3c98c31-kserve-bff9c879-h5x5m\" (UID: \"517e687e-3f3f-4e0a-b08d-3dd66cdcf835\") " pod="llm/e2e-same-id-b-a3c98c31-kserve-bff9c879-h5x5m"
Sep 11 23:45:55.484475 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:55.484298    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/517e687e-3f3f-4e0a-b08d-3dd66cdcf835-tmp-dir\") pod \"e2e-same-id-b-a3c98c31-kserve-bff9c879-h5x5m\" (UID: \"517e687e-3f3f-4e0a-b08d-3dd66cdcf835\") " pod="llm/e2e-same-id-b-a3c98c31-kserve-bff9c879-h5x5m"
Sep 11 23:45:55.484475 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:55.484322    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/517e687e-3f3f-4e0a-b08d-3dd66cdcf835-home\") pod \"e2e-same-id-b-a3c98c31-kserve-bff9c879-h5x5m\" (UID: \"517e687e-3f3f-4e0a-b08d-3dd66cdcf835\") " pod="llm/e2e-same-id-b-a3c98c31-kserve-bff9c879-h5x5m"
Sep 11 23:45:55.484475 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:55.484350    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/517e687e-3f3f-4e0a-b08d-3dd66cdcf835-dshm\") pod \"e2e-same-id-b-a3c98c31-kserve-bff9c879-h5x5m\" (UID: \"517e687e-3f3f-4e0a-b08d-3dd66cdcf835\") " pod="llm/e2e-same-id-b-a3c98c31-kserve-bff9c879-h5x5m"
Sep 11 23:45:55.484475 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:55.484367    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpb42\" (UniqueName: \"kubernetes.io/projected/517e687e-3f3f-4e0a-b08d-3dd66cdcf835-kube-api-access-wpb42\") pod \"e2e-same-id-b-a3c98c31-kserve-bff9c879-h5x5m\" (UID: \"517e687e-3f3f-4e0a-b08d-3dd66cdcf835\") " pod="llm/e2e-same-id-b-a3c98c31-kserve-bff9c879-h5x5m"
Sep 11 23:45:55.484475 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:55.484398    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/517e687e-3f3f-4e0a-b08d-3dd66cdcf835-model-cache\") pod \"e2e-same-id-b-a3c98c31-kserve-bff9c879-h5x5m\" (UID: \"517e687e-3f3f-4e0a-b08d-3dd66cdcf835\") " pod="llm/e2e-same-id-b-a3c98c31-kserve-bff9c879-h5x5m"
Sep 11 23:45:55.484736 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:55.484685    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/517e687e-3f3f-4e0a-b08d-3dd66cdcf835-tmp-dir\") pod \"e2e-same-id-b-a3c98c31-kserve-bff9c879-h5x5m\" (UID: \"517e687e-3f3f-4e0a-b08d-3dd66cdcf835\") " pod="llm/e2e-same-id-b-a3c98c31-kserve-bff9c879-h5x5m"
Sep 11 23:45:55.484784 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:55.484750    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/517e687e-3f3f-4e0a-b08d-3dd66cdcf835-home\") pod \"e2e-same-id-b-a3c98c31-kserve-bff9c879-h5x5m\" (UID: \"517e687e-3f3f-4e0a-b08d-3dd66cdcf835\") " pod="llm/e2e-same-id-b-a3c98c31-kserve-bff9c879-h5x5m"
Sep 11 23:45:55.484885 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:55.484863    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/517e687e-3f3f-4e0a-b08d-3dd66cdcf835-model-cache\") pod \"e2e-same-id-b-a3c98c31-kserve-bff9c879-h5x5m\" (UID: \"517e687e-3f3f-4e0a-b08d-3dd66cdcf835\") " pod="llm/e2e-same-id-b-a3c98c31-kserve-bff9c879-h5x5m"
Sep 11 23:45:55.486808 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:55.486785    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/517e687e-3f3f-4e0a-b08d-3dd66cdcf835-dshm\") pod \"e2e-same-id-b-a3c98c31-kserve-bff9c879-h5x5m\" (UID: \"517e687e-3f3f-4e0a-b08d-3dd66cdcf835\") " pod="llm/e2e-same-id-b-a3c98c31-kserve-bff9c879-h5x5m"
Sep 11 23:45:55.486900 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:55.486880    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/517e687e-3f3f-4e0a-b08d-3dd66cdcf835-tls-certs\") pod \"e2e-same-id-b-a3c98c31-kserve-bff9c879-h5x5m\" (UID: \"517e687e-3f3f-4e0a-b08d-3dd66cdcf835\") " pod="llm/e2e-same-id-b-a3c98c31-kserve-bff9c879-h5x5m"
Sep 11 23:45:55.493322 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:55.493303    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpb42\" (UniqueName: \"kubernetes.io/projected/517e687e-3f3f-4e0a-b08d-3dd66cdcf835-kube-api-access-wpb42\") pod \"e2e-same-id-b-a3c98c31-kserve-bff9c879-h5x5m\" (UID: \"517e687e-3f3f-4e0a-b08d-3dd66cdcf835\") " pod="llm/e2e-same-id-b-a3c98c31-kserve-bff9c879-h5x5m"
Sep 11 23:45:55.640467 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:55.640378    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-a3c98c31-kserve-bff9c879-h5x5m"
Sep 11 23:45:55.915299 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:55.915271    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-a3c98c31-kserve-bff9c879-h5x5m"]
Sep 11 23:45:55.917035 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:45:55.917009    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod517e687e_3f3f_4e0a_b08d_3dd66cdcf835.slice/crio-230b8bab5e4a5fbd977a11da93013ebe44d2cd933e7ecaec9971833a87bd86d0 WatchSource:0}: Error finding container 230b8bab5e4a5fbd977a11da93013ebe44d2cd933e7ecaec9971833a87bd86d0: Status 404 returned error can't find the container with id 230b8bab5e4a5fbd977a11da93013ebe44d2cd933e7ecaec9971833a87bd86d0
Sep 11 23:45:56.176680 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:56.176592    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-a3c98c31-kserve-bff9c879-h5x5m" event={"ID":"517e687e-3f3f-4e0a-b08d-3dd66cdcf835","Type":"ContainerStarted","Data":"28ac70e202bb4696c12e98572844a735efef505a270a2c7bcbb063e20824db4f"}
Sep 11 23:45:56.176680 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:56.176642    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-a3c98c31-kserve-bff9c879-h5x5m" event={"ID":"517e687e-3f3f-4e0a-b08d-3dd66cdcf835","Type":"ContainerStarted","Data":"230b8bab5e4a5fbd977a11da93013ebe44d2cd933e7ecaec9971833a87bd86d0"}
Sep 11 23:45:56.194778 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:56.194730    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-a3c98c31-kserve-bff9c879-h5x5m" podStartSLOduration=1.194712395 podStartE2EDuration="1.194712395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 23:45:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:45:56.193052917 +0000 UTC m=+2680.226182140" watchObservedRunningTime="2026-09-11 23:45:56.194712395 +0000 UTC m=+2680.227841610"
Sep 11 23:45:58.323351 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:45:58.323314    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"]
Sep 11 23:46:05.641097 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:46:05.641063    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-a3c98c31-kserve-bff9c879-h5x5m"
Sep 11 23:46:05.641507 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:46:05.641119    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-a3c98c31-kserve-bff9c879-h5x5m"
Sep 11 23:46:05.653257 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:46:05.653236    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-a3c98c31-kserve-bff9c879-h5x5m"
Sep 11 23:46:06.246226 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:46:06.246195    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-a3c98c31-kserve-bff9c879-h5x5m"
Sep 11 23:46:16.782923 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:46:16.782886    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7244_6b0a8af3-932c-4091-b7a0-180fe1751a3a/ovn-acl-logging/0.log"
Sep 11 23:46:16.788777 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:46:16.788755    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7244_6b0a8af3-932c-4091-b7a0-180fe1751a3a/ovn-acl-logging/0.log"
Sep 11 23:46:17.299002 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:46:17.298971    2546 scope.go:117] "RemoveContainer" containerID="6768ae84b1352bd083d48da30a81cf471fe1ae38093047eb04c2c237ffd11eb8"
Sep 11 23:46:41.501543 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:46:41.501500    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-a3c98c31-kserve-d9b9855cf-bfxbx"]
Sep 11 23:46:42.659924 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:46:42.659893    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-a3c98c31-kserve-bff9c879-h5x5m"]
Sep 11 23:46:44.727060 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:46:44.727021    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"]
Sep 11 23:46:52.425180 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:46:52.425123    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"]
Sep 11 23:46:56.521568 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:46:56.521525    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-a3c98c31-kserve-d9b9855cf-bfxbx" podUID="deb1a639-8342-4227-8b82-9a180cb13e47" containerName="main" containerID="cri-o://ef0ea1256f3c868e97ed25433e6e227a55570fec1b47cf84de17cd7db0c7dcb1" gracePeriod=45
Sep 11 23:46:56.773828 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:46:56.773763    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-a3c98c31-kserve-d9b9855cf-bfxbx"
Sep 11 23:46:56.860997 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:46:56.860960    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/deb1a639-8342-4227-8b82-9a180cb13e47-tmp-dir\") pod \"deb1a639-8342-4227-8b82-9a180cb13e47\" (UID: \"deb1a639-8342-4227-8b82-9a180cb13e47\") "
Sep 11 23:46:56.861208 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:46:56.861047    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kgg7c\" (UniqueName: \"kubernetes.io/projected/deb1a639-8342-4227-8b82-9a180cb13e47-kube-api-access-kgg7c\") pod \"deb1a639-8342-4227-8b82-9a180cb13e47\" (UID: \"deb1a639-8342-4227-8b82-9a180cb13e47\") "
Sep 11 23:46:56.861208 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:46:56.861094    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/deb1a639-8342-4227-8b82-9a180cb13e47-dshm\") pod \"deb1a639-8342-4227-8b82-9a180cb13e47\" (UID: \"deb1a639-8342-4227-8b82-9a180cb13e47\") "
Sep 11 23:46:56.861208 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:46:56.861122    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/deb1a639-8342-4227-8b82-9a180cb13e47-home\") pod \"deb1a639-8342-4227-8b82-9a180cb13e47\" (UID: \"deb1a639-8342-4227-8b82-9a180cb13e47\") "
Sep 11 23:46:56.861385 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:46:56.861232    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/deb1a639-8342-4227-8b82-9a180cb13e47-model-cache\") pod \"deb1a639-8342-4227-8b82-9a180cb13e47\" (UID: \"deb1a639-8342-4227-8b82-9a180cb13e47\") "
Sep 11 23:46:56.861385 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:46:56.861273    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/deb1a639-8342-4227-8b82-9a180cb13e47-tls-certs\") pod \"deb1a639-8342-4227-8b82-9a180cb13e47\" (UID: \"deb1a639-8342-4227-8b82-9a180cb13e47\") "
Sep 11 23:46:56.861385 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:46:56.861319    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deb1a639-8342-4227-8b82-9a180cb13e47-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "deb1a639-8342-4227-8b82-9a180cb13e47" (UID: "deb1a639-8342-4227-8b82-9a180cb13e47"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:46:56.861559 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:46:56.861403    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deb1a639-8342-4227-8b82-9a180cb13e47-home" (OuterVolumeSpecName: "home") pod "deb1a639-8342-4227-8b82-9a180cb13e47" (UID: "deb1a639-8342-4227-8b82-9a180cb13e47"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:46:56.861559 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:46:56.861480    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deb1a639-8342-4227-8b82-9a180cb13e47-model-cache" (OuterVolumeSpecName: "model-cache") pod "deb1a639-8342-4227-8b82-9a180cb13e47" (UID: "deb1a639-8342-4227-8b82-9a180cb13e47"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:46:56.861654 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:46:56.861586    2546 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/deb1a639-8342-4227-8b82-9a180cb13e47-home\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:46:56.861654 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:46:56.861601    2546 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/deb1a639-8342-4227-8b82-9a180cb13e47-model-cache\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:46:56.861654 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:46:56.861628    2546 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/deb1a639-8342-4227-8b82-9a180cb13e47-tmp-dir\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:46:56.863561 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:46:56.863530    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deb1a639-8342-4227-8b82-9a180cb13e47-dshm" (OuterVolumeSpecName: "dshm") pod "deb1a639-8342-4227-8b82-9a180cb13e47" (UID: "deb1a639-8342-4227-8b82-9a180cb13e47"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:46:56.863667 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:46:56.863608    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/deb1a639-8342-4227-8b82-9a180cb13e47-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "deb1a639-8342-4227-8b82-9a180cb13e47" (UID: "deb1a639-8342-4227-8b82-9a180cb13e47"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 23:46:56.863667 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:46:56.863645    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/deb1a639-8342-4227-8b82-9a180cb13e47-kube-api-access-kgg7c" (OuterVolumeSpecName: "kube-api-access-kgg7c") pod "deb1a639-8342-4227-8b82-9a180cb13e47" (UID: "deb1a639-8342-4227-8b82-9a180cb13e47"). InnerVolumeSpecName "kube-api-access-kgg7c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 23:46:56.962640 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:46:56.962582    2546 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/deb1a639-8342-4227-8b82-9a180cb13e47-tls-certs\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:46:56.962640 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:46:56.962634    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgg7c\" (UniqueName: \"kubernetes.io/projected/deb1a639-8342-4227-8b82-9a180cb13e47-kube-api-access-kgg7c\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:46:56.962640 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:46:56.962646    2546 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/deb1a639-8342-4227-8b82-9a180cb13e47-dshm\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:46:57.538148 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:46:57.538107    2546 generic.go:356] "Generic (PLEG): container finished" podID="deb1a639-8342-4227-8b82-9a180cb13e47" containerID="ef0ea1256f3c868e97ed25433e6e227a55570fec1b47cf84de17cd7db0c7dcb1" exitCode=0
Sep 11 23:46:57.538640 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:46:57.538195    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-a3c98c31-kserve-d9b9855cf-bfxbx"
Sep 11 23:46:57.538640 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:46:57.538204    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-a3c98c31-kserve-d9b9855cf-bfxbx" event={"ID":"deb1a639-8342-4227-8b82-9a180cb13e47","Type":"ContainerDied","Data":"ef0ea1256f3c868e97ed25433e6e227a55570fec1b47cf84de17cd7db0c7dcb1"}
Sep 11 23:46:57.538640 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:46:57.538255    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-a3c98c31-kserve-d9b9855cf-bfxbx" event={"ID":"deb1a639-8342-4227-8b82-9a180cb13e47","Type":"ContainerDied","Data":"5aeea312e9449a2c425524002d3503452a7497216dc2f3549d79656aaf957d44"}
Sep 11 23:46:57.538640 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:46:57.538282    2546 scope.go:117] "RemoveContainer" containerID="ef0ea1256f3c868e97ed25433e6e227a55570fec1b47cf84de17cd7db0c7dcb1"
Sep 11 23:46:57.549031 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:46:57.549004    2546 scope.go:117] "RemoveContainer" containerID="ef0ea1256f3c868e97ed25433e6e227a55570fec1b47cf84de17cd7db0c7dcb1"
Sep 11 23:46:57.549356 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:46:57.549336    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef0ea1256f3c868e97ed25433e6e227a55570fec1b47cf84de17cd7db0c7dcb1\": container with ID starting with ef0ea1256f3c868e97ed25433e6e227a55570fec1b47cf84de17cd7db0c7dcb1 not found: ID does not exist" containerID="ef0ea1256f3c868e97ed25433e6e227a55570fec1b47cf84de17cd7db0c7dcb1"
Sep 11 23:46:57.549424 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:46:57.549365    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef0ea1256f3c868e97ed25433e6e227a55570fec1b47cf84de17cd7db0c7dcb1"} err="failed to get container status \"ef0ea1256f3c868e97ed25433e6e227a55570fec1b47cf84de17cd7db0c7dcb1\": rpc error: code = NotFound desc = could not find container \"ef0ea1256f3c868e97ed25433e6e227a55570fec1b47cf84de17cd7db0c7dcb1\": container with ID starting with ef0ea1256f3c868e97ed25433e6e227a55570fec1b47cf84de17cd7db0c7dcb1 not found: ID does not exist"
Sep 11 23:46:57.561774 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:46:57.561744    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-a3c98c31-kserve-d9b9855cf-bfxbx"]
Sep 11 23:46:57.564162 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:46:57.564128    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-a3c98c31-kserve-d9b9855cf-bfxbx"]
Sep 11 23:46:57.680501 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:46:57.680456    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-a3c98c31-kserve-bff9c879-h5x5m" podUID="517e687e-3f3f-4e0a-b08d-3dd66cdcf835" containerName="main" containerID="cri-o://28ac70e202bb4696c12e98572844a735efef505a270a2c7bcbb063e20824db4f" gracePeriod=45
Sep 11 23:46:57.933051 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:46:57.933026    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-a3c98c31-kserve-bff9c879-h5x5m"
Sep 11 23:46:58.074029 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:46:58.073939    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wpb42\" (UniqueName: \"kubernetes.io/projected/517e687e-3f3f-4e0a-b08d-3dd66cdcf835-kube-api-access-wpb42\") pod \"517e687e-3f3f-4e0a-b08d-3dd66cdcf835\" (UID: \"517e687e-3f3f-4e0a-b08d-3dd66cdcf835\") "
Sep 11 23:46:58.074029 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:46:58.073976    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/517e687e-3f3f-4e0a-b08d-3dd66cdcf835-tmp-dir\") pod \"517e687e-3f3f-4e0a-b08d-3dd66cdcf835\" (UID: \"517e687e-3f3f-4e0a-b08d-3dd66cdcf835\") "
Sep 11 23:46:58.074029 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:46:58.074034    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/517e687e-3f3f-4e0a-b08d-3dd66cdcf835-tls-certs\") pod \"517e687e-3f3f-4e0a-b08d-3dd66cdcf835\" (UID: \"517e687e-3f3f-4e0a-b08d-3dd66cdcf835\") "
Sep 11 23:46:58.074355 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:46:58.074203    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/517e687e-3f3f-4e0a-b08d-3dd66cdcf835-home\") pod \"517e687e-3f3f-4e0a-b08d-3dd66cdcf835\" (UID: \"517e687e-3f3f-4e0a-b08d-3dd66cdcf835\") "
Sep 11 23:46:58.074355 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:46:58.074305    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/517e687e-3f3f-4e0a-b08d-3dd66cdcf835-dshm\") pod \"517e687e-3f3f-4e0a-b08d-3dd66cdcf835\" (UID: \"517e687e-3f3f-4e0a-b08d-3dd66cdcf835\") "
Sep 11 23:46:58.074355 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:46:58.074308    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/517e687e-3f3f-4e0a-b08d-3dd66cdcf835-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "517e687e-3f3f-4e0a-b08d-3dd66cdcf835" (UID: "517e687e-3f3f-4e0a-b08d-3dd66cdcf835"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:46:58.074355 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:46:58.074344    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/517e687e-3f3f-4e0a-b08d-3dd66cdcf835-model-cache\") pod \"517e687e-3f3f-4e0a-b08d-3dd66cdcf835\" (UID: \"517e687e-3f3f-4e0a-b08d-3dd66cdcf835\") "
Sep 11 23:46:58.074542 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:46:58.074429    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/517e687e-3f3f-4e0a-b08d-3dd66cdcf835-home" (OuterVolumeSpecName: "home") pod "517e687e-3f3f-4e0a-b08d-3dd66cdcf835" (UID: "517e687e-3f3f-4e0a-b08d-3dd66cdcf835"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:46:58.074673 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:46:58.074635    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/517e687e-3f3f-4e0a-b08d-3dd66cdcf835-model-cache" (OuterVolumeSpecName: "model-cache") pod "517e687e-3f3f-4e0a-b08d-3dd66cdcf835" (UID: "517e687e-3f3f-4e0a-b08d-3dd66cdcf835"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:46:58.074799 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:46:58.074697    2546 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/517e687e-3f3f-4e0a-b08d-3dd66cdcf835-tmp-dir\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:46:58.074799 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:46:58.074719    2546 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/517e687e-3f3f-4e0a-b08d-3dd66cdcf835-home\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:46:58.076368 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:46:58.076340    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/517e687e-3f3f-4e0a-b08d-3dd66cdcf835-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "517e687e-3f3f-4e0a-b08d-3dd66cdcf835" (UID: "517e687e-3f3f-4e0a-b08d-3dd66cdcf835"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 23:46:58.076458 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:46:58.076368    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/517e687e-3f3f-4e0a-b08d-3dd66cdcf835-dshm" (OuterVolumeSpecName: "dshm") pod "517e687e-3f3f-4e0a-b08d-3dd66cdcf835" (UID: "517e687e-3f3f-4e0a-b08d-3dd66cdcf835"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 23:46:58.076458 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:46:58.076402    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/517e687e-3f3f-4e0a-b08d-3dd66cdcf835-kube-api-access-wpb42" (OuterVolumeSpecName: "kube-api-access-wpb42") pod "517e687e-3f3f-4e0a-b08d-3dd66cdcf835" (UID: "517e687e-3f3f-4e0a-b08d-3dd66cdcf835"). InnerVolumeSpecName "kube-api-access-wpb42". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 23:46:58.175489 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:46:58.175446    2546 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/517e687e-3f3f-4e0a-b08d-3dd66cdcf835-dshm\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:46:58.175489 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:46:58.175480    2546 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/517e687e-3f3f-4e0a-b08d-3dd66cdcf835-model-cache\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:46:58.175489 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:46:58.175491    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpb42\" (UniqueName: \"kubernetes.io/projected/517e687e-3f3f-4e0a-b08d-3dd66cdcf835-kube-api-access-wpb42\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:46:58.175489 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:46:58.175500    2546 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/517e687e-3f3f-4e0a-b08d-3dd66cdcf835-tls-certs\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:46:58.494819 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:46:58.494781    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="deb1a639-8342-4227-8b82-9a180cb13e47" path="/var/lib/kubelet/pods/deb1a639-8342-4227-8b82-9a180cb13e47/volumes"
Sep 11 23:46:58.546860 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:46:58.546819    2546 generic.go:356] "Generic (PLEG): container finished" podID="517e687e-3f3f-4e0a-b08d-3dd66cdcf835" containerID="28ac70e202bb4696c12e98572844a735efef505a270a2c7bcbb063e20824db4f" exitCode=0
Sep 11 23:46:58.546860 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:46:58.546862    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-a3c98c31-kserve-bff9c879-h5x5m"
Sep 11 23:46:58.547440 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:46:58.546897    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-a3c98c31-kserve-bff9c879-h5x5m" event={"ID":"517e687e-3f3f-4e0a-b08d-3dd66cdcf835","Type":"ContainerDied","Data":"28ac70e202bb4696c12e98572844a735efef505a270a2c7bcbb063e20824db4f"}
Sep 11 23:46:58.547440 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:46:58.546942    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-a3c98c31-kserve-bff9c879-h5x5m" event={"ID":"517e687e-3f3f-4e0a-b08d-3dd66cdcf835","Type":"ContainerDied","Data":"230b8bab5e4a5fbd977a11da93013ebe44d2cd933e7ecaec9971833a87bd86d0"}
Sep 11 23:46:58.547440 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:46:58.546960    2546 scope.go:117] "RemoveContainer" containerID="28ac70e202bb4696c12e98572844a735efef505a270a2c7bcbb063e20824db4f"
Sep 11 23:46:58.556671 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:46:58.556650    2546 scope.go:117] "RemoveContainer" containerID="28ac70e202bb4696c12e98572844a735efef505a270a2c7bcbb063e20824db4f"
Sep 11 23:46:58.557023 ip-10-0-128-161 kubenswrapper[2546]: E0911 23:46:58.557004    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28ac70e202bb4696c12e98572844a735efef505a270a2c7bcbb063e20824db4f\": container with ID starting with 28ac70e202bb4696c12e98572844a735efef505a270a2c7bcbb063e20824db4f not found: ID does not exist" containerID="28ac70e202bb4696c12e98572844a735efef505a270a2c7bcbb063e20824db4f"
Sep 11 23:46:58.557100 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:46:58.557033    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28ac70e202bb4696c12e98572844a735efef505a270a2c7bcbb063e20824db4f"} err="failed to get container status \"28ac70e202bb4696c12e98572844a735efef505a270a2c7bcbb063e20824db4f\": rpc error: code = NotFound desc = could not find container \"28ac70e202bb4696c12e98572844a735efef505a270a2c7bcbb063e20824db4f\": container with ID starting with 28ac70e202bb4696c12e98572844a735efef505a270a2c7bcbb063e20824db4f not found: ID does not exist"
Sep 11 23:46:58.566044 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:46:58.566014    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-a3c98c31-kserve-bff9c879-h5x5m"]
Sep 11 23:46:58.570121 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:46:58.570094    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-a3c98c31-kserve-bff9c879-h5x5m"]
Sep 11 23:46:59.224273 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:46:59.224234    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"]
Sep 11 23:47:00.495368 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:00.495326    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="517e687e-3f3f-4e0a-b08d-3dd66cdcf835" path="/var/lib/kubelet/pods/517e687e-3f3f-4e0a-b08d-3dd66cdcf835/volumes"
Sep 11 23:47:05.710990 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:05.710957    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5759ff8db-zhptw"]
Sep 11 23:47:05.711368 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:05.711360    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="517e687e-3f3f-4e0a-b08d-3dd66cdcf835" containerName="main"
Sep 11 23:47:05.711411 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:05.711371    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="517e687e-3f3f-4e0a-b08d-3dd66cdcf835" containerName="main"
Sep 11 23:47:05.711411 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:05.711385    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="deb1a639-8342-4227-8b82-9a180cb13e47" containerName="main"
Sep 11 23:47:05.711411 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:05.711391    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="deb1a639-8342-4227-8b82-9a180cb13e47" containerName="main"
Sep 11 23:47:05.711502 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:05.711455    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="517e687e-3f3f-4e0a-b08d-3dd66cdcf835" containerName="main"
Sep 11 23:47:05.711502 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:05.711466    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="deb1a639-8342-4227-8b82-9a180cb13e47" containerName="main"
Sep 11 23:47:05.716794 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:05.716772    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5759ff8db-zhptw"
Sep 11 23:47:05.719728 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:05.719707    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 23:47:05.721204 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:05.721185    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 11 23:47:05.721320 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:05.721212    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 11 23:47:05.721320 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:05.721212    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 11 23:47:05.721320 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:05.721212    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-224sg\""
Sep 11 23:47:05.728410 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:05.728381    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5759ff8db-zhptw"]
Sep 11 23:47:05.845004 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:05.844961    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5830329c-8118-4fa1-9894-4318fcfcea13-maas-api-tls\") pod \"maas-api-5759ff8db-zhptw\" (UID: \"5830329c-8118-4fa1-9894-4318fcfcea13\") " pod="odh-ai-gateway-infra/maas-api-5759ff8db-zhptw"
Sep 11 23:47:05.845209 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:05.845015    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5830329c-8118-4fa1-9894-4318fcfcea13-openshift-service-ca\") pod \"maas-api-5759ff8db-zhptw\" (UID: \"5830329c-8118-4fa1-9894-4318fcfcea13\") " pod="odh-ai-gateway-infra/maas-api-5759ff8db-zhptw"
Sep 11 23:47:05.845209 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:05.845034    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkgbf\" (UniqueName: \"kubernetes.io/projected/5830329c-8118-4fa1-9894-4318fcfcea13-kube-api-access-rkgbf\") pod \"maas-api-5759ff8db-zhptw\" (UID: \"5830329c-8118-4fa1-9894-4318fcfcea13\") " pod="odh-ai-gateway-infra/maas-api-5759ff8db-zhptw"
Sep 11 23:47:05.845209 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:05.845140    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5830329c-8118-4fa1-9894-4318fcfcea13-metrics-certs\") pod \"maas-api-5759ff8db-zhptw\" (UID: \"5830329c-8118-4fa1-9894-4318fcfcea13\") " pod="odh-ai-gateway-infra/maas-api-5759ff8db-zhptw"
Sep 11 23:47:05.945850 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:05.945811    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5830329c-8118-4fa1-9894-4318fcfcea13-openshift-service-ca\") pod \"maas-api-5759ff8db-zhptw\" (UID: \"5830329c-8118-4fa1-9894-4318fcfcea13\") " pod="odh-ai-gateway-infra/maas-api-5759ff8db-zhptw"
Sep 11 23:47:05.945850 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:05.945848    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkgbf\" (UniqueName: \"kubernetes.io/projected/5830329c-8118-4fa1-9894-4318fcfcea13-kube-api-access-rkgbf\") pod \"maas-api-5759ff8db-zhptw\" (UID: \"5830329c-8118-4fa1-9894-4318fcfcea13\") " pod="odh-ai-gateway-infra/maas-api-5759ff8db-zhptw"
Sep 11 23:47:05.946043 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:05.945899    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5830329c-8118-4fa1-9894-4318fcfcea13-metrics-certs\") pod \"maas-api-5759ff8db-zhptw\" (UID: \"5830329c-8118-4fa1-9894-4318fcfcea13\") " pod="odh-ai-gateway-infra/maas-api-5759ff8db-zhptw"
Sep 11 23:47:05.946043 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:05.945964    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5830329c-8118-4fa1-9894-4318fcfcea13-maas-api-tls\") pod \"maas-api-5759ff8db-zhptw\" (UID: \"5830329c-8118-4fa1-9894-4318fcfcea13\") " pod="odh-ai-gateway-infra/maas-api-5759ff8db-zhptw"
Sep 11 23:47:05.946518 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:05.946484    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5830329c-8118-4fa1-9894-4318fcfcea13-openshift-service-ca\") pod \"maas-api-5759ff8db-zhptw\" (UID: \"5830329c-8118-4fa1-9894-4318fcfcea13\") " pod="odh-ai-gateway-infra/maas-api-5759ff8db-zhptw"
Sep 11 23:47:05.948493 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:05.948471    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5830329c-8118-4fa1-9894-4318fcfcea13-metrics-certs\") pod \"maas-api-5759ff8db-zhptw\" (UID: \"5830329c-8118-4fa1-9894-4318fcfcea13\") " pod="odh-ai-gateway-infra/maas-api-5759ff8db-zhptw"
Sep 11 23:47:05.948657 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:05.948637    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5830329c-8118-4fa1-9894-4318fcfcea13-maas-api-tls\") pod \"maas-api-5759ff8db-zhptw\" (UID: \"5830329c-8118-4fa1-9894-4318fcfcea13\") " pod="odh-ai-gateway-infra/maas-api-5759ff8db-zhptw"
Sep 11 23:47:05.953727 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:05.953709    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkgbf\" (UniqueName: \"kubernetes.io/projected/5830329c-8118-4fa1-9894-4318fcfcea13-kube-api-access-rkgbf\") pod \"maas-api-5759ff8db-zhptw\" (UID: \"5830329c-8118-4fa1-9894-4318fcfcea13\") " pod="odh-ai-gateway-infra/maas-api-5759ff8db-zhptw"
Sep 11 23:47:06.029716 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:06.029634    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5759ff8db-zhptw"
Sep 11 23:47:06.304464 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:06.304437    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5759ff8db-zhptw"]
Sep 11 23:47:06.306456 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:47:06.306427    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5830329c_8118_4fa1_9894_4318fcfcea13.slice/crio-8bb5ad50430ffc6468668de5b840ae5bf0dba86445bbe8c9e247d4a3ead7c1a0 WatchSource:0}: Error finding container 8bb5ad50430ffc6468668de5b840ae5bf0dba86445bbe8c9e247d4a3ead7c1a0: Status 404 returned error can't find the container with id 8bb5ad50430ffc6468668de5b840ae5bf0dba86445bbe8c9e247d4a3ead7c1a0
Sep 11 23:47:06.307729 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:06.307711    2546 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 23:47:06.322985 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:06.322955    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"]
Sep 11 23:47:06.596924 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:06.596839    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5759ff8db-zhptw" event={"ID":"5830329c-8118-4fa1-9894-4318fcfcea13","Type":"ContainerStarted","Data":"8bb5ad50430ffc6468668de5b840ae5bf0dba86445bbe8c9e247d4a3ead7c1a0"}
Sep 11 23:47:09.616440 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:09.616395    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5759ff8db-zhptw" event={"ID":"5830329c-8118-4fa1-9894-4318fcfcea13","Type":"ContainerStarted","Data":"93cb5981a2d4ae8e5bc026f67088e908d4372df172dae8026f92ae153fdb9992"}
Sep 11 23:47:09.616854 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:09.616556    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5759ff8db-zhptw"
Sep 11 23:47:09.634997 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:09.634943    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5759ff8db-zhptw" podStartSLOduration=2.134891773 podStartE2EDuration="4.634926871s" totalImagesPullingTime="2.500035098s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 23:47:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:47:09.634726041 +0000 UTC m=+2753.667855252" watchObservedRunningTime="2026-09-11 23:47:09.634926871 +0000 UTC m=+2753.668056088"
Sep 11 23:47:13.223664 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:13.223628    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"]
Sep 11 23:47:13.514639 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:13.514548    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5759ff8db-zhptw"]
Sep 11 23:47:13.514806 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:13.514779    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5759ff8db-zhptw" podUID="5830329c-8118-4fa1-9894-4318fcfcea13" containerName="maas-api" containerID="cri-o://93cb5981a2d4ae8e5bc026f67088e908d4372df172dae8026f92ae153fdb9992" gracePeriod=30
Sep 11 23:47:13.520341 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:13.520315    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5759ff8db-zhptw"
Sep 11 23:47:13.549603 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:13.549571    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6f85f54558-7l8hl"]
Sep 11 23:47:13.553104 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:13.553083    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6f85f54558-7l8hl"
Sep 11 23:47:13.564369 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:13.564338    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6f85f54558-7l8hl"]
Sep 11 23:47:13.604793 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:13.604755    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0c6d2d0-4287-43fb-a39c-539765b74e68-metrics-certs\") pod \"maas-api-6f85f54558-7l8hl\" (UID: \"e0c6d2d0-4287-43fb-a39c-539765b74e68\") " pod="odh-ai-gateway-infra/maas-api-6f85f54558-7l8hl"
Sep 11 23:47:13.604793 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:13.604795    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq4kd\" (UniqueName: \"kubernetes.io/projected/e0c6d2d0-4287-43fb-a39c-539765b74e68-kube-api-access-gq4kd\") pod \"maas-api-6f85f54558-7l8hl\" (UID: \"e0c6d2d0-4287-43fb-a39c-539765b74e68\") " pod="odh-ai-gateway-infra/maas-api-6f85f54558-7l8hl"
Sep 11 23:47:13.605014 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:13.604880    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0c6d2d0-4287-43fb-a39c-539765b74e68-openshift-service-ca\") pod \"maas-api-6f85f54558-7l8hl\" (UID: \"e0c6d2d0-4287-43fb-a39c-539765b74e68\") " pod="odh-ai-gateway-infra/maas-api-6f85f54558-7l8hl"
Sep 11 23:47:13.605014 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:13.604967    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0c6d2d0-4287-43fb-a39c-539765b74e68-maas-api-tls\") pod \"maas-api-6f85f54558-7l8hl\" (UID: \"e0c6d2d0-4287-43fb-a39c-539765b74e68\") " pod="odh-ai-gateway-infra/maas-api-6f85f54558-7l8hl"
Sep 11 23:47:13.641767 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:13.641722    2546 generic.go:356] "Generic (PLEG): container finished" podID="5830329c-8118-4fa1-9894-4318fcfcea13" containerID="93cb5981a2d4ae8e5bc026f67088e908d4372df172dae8026f92ae153fdb9992" exitCode=0
Sep 11 23:47:13.641767 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:13.641749    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5759ff8db-zhptw" event={"ID":"5830329c-8118-4fa1-9894-4318fcfcea13","Type":"ContainerDied","Data":"93cb5981a2d4ae8e5bc026f67088e908d4372df172dae8026f92ae153fdb9992"}
Sep 11 23:47:13.705920 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:13.705595    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0c6d2d0-4287-43fb-a39c-539765b74e68-maas-api-tls\") pod \"maas-api-6f85f54558-7l8hl\" (UID: \"e0c6d2d0-4287-43fb-a39c-539765b74e68\") " pod="odh-ai-gateway-infra/maas-api-6f85f54558-7l8hl"
Sep 11 23:47:13.705920 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:13.705667    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0c6d2d0-4287-43fb-a39c-539765b74e68-metrics-certs\") pod \"maas-api-6f85f54558-7l8hl\" (UID: \"e0c6d2d0-4287-43fb-a39c-539765b74e68\") " pod="odh-ai-gateway-infra/maas-api-6f85f54558-7l8hl"
Sep 11 23:47:13.705920 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:13.705696    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq4kd\" (UniqueName: \"kubernetes.io/projected/e0c6d2d0-4287-43fb-a39c-539765b74e68-kube-api-access-gq4kd\") pod \"maas-api-6f85f54558-7l8hl\" (UID: \"e0c6d2d0-4287-43fb-a39c-539765b74e68\") " pod="odh-ai-gateway-infra/maas-api-6f85f54558-7l8hl"
Sep 11 23:47:13.705920 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:13.705755    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0c6d2d0-4287-43fb-a39c-539765b74e68-openshift-service-ca\") pod \"maas-api-6f85f54558-7l8hl\" (UID: \"e0c6d2d0-4287-43fb-a39c-539765b74e68\") " pod="odh-ai-gateway-infra/maas-api-6f85f54558-7l8hl"
Sep 11 23:47:13.707032 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:13.707000    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0c6d2d0-4287-43fb-a39c-539765b74e68-openshift-service-ca\") pod \"maas-api-6f85f54558-7l8hl\" (UID: \"e0c6d2d0-4287-43fb-a39c-539765b74e68\") " pod="odh-ai-gateway-infra/maas-api-6f85f54558-7l8hl"
Sep 11 23:47:13.709085 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:13.709060    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0c6d2d0-4287-43fb-a39c-539765b74e68-maas-api-tls\") pod \"maas-api-6f85f54558-7l8hl\" (UID: \"e0c6d2d0-4287-43fb-a39c-539765b74e68\") " pod="odh-ai-gateway-infra/maas-api-6f85f54558-7l8hl"
Sep 11 23:47:13.710238 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:13.710210    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0c6d2d0-4287-43fb-a39c-539765b74e68-metrics-certs\") pod \"maas-api-6f85f54558-7l8hl\" (UID: \"e0c6d2d0-4287-43fb-a39c-539765b74e68\") " pod="odh-ai-gateway-infra/maas-api-6f85f54558-7l8hl"
Sep 11 23:47:13.715320 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:13.715289    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq4kd\" (UniqueName: \"kubernetes.io/projected/e0c6d2d0-4287-43fb-a39c-539765b74e68-kube-api-access-gq4kd\") pod \"maas-api-6f85f54558-7l8hl\" (UID: \"e0c6d2d0-4287-43fb-a39c-539765b74e68\") " pod="odh-ai-gateway-infra/maas-api-6f85f54558-7l8hl"
Sep 11 23:47:13.773115 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:13.773049    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5759ff8db-zhptw"
Sep 11 23:47:13.806957 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:13.806910    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5830329c-8118-4fa1-9894-4318fcfcea13-maas-api-tls\") pod \"5830329c-8118-4fa1-9894-4318fcfcea13\" (UID: \"5830329c-8118-4fa1-9894-4318fcfcea13\") "
Sep 11 23:47:13.806957 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:13.806964    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5830329c-8118-4fa1-9894-4318fcfcea13-metrics-certs\") pod \"5830329c-8118-4fa1-9894-4318fcfcea13\" (UID: \"5830329c-8118-4fa1-9894-4318fcfcea13\") "
Sep 11 23:47:13.807253 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:13.807009    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5830329c-8118-4fa1-9894-4318fcfcea13-openshift-service-ca\") pod \"5830329c-8118-4fa1-9894-4318fcfcea13\" (UID: \"5830329c-8118-4fa1-9894-4318fcfcea13\") "
Sep 11 23:47:13.807253 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:13.807069    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rkgbf\" (UniqueName: \"kubernetes.io/projected/5830329c-8118-4fa1-9894-4318fcfcea13-kube-api-access-rkgbf\") pod \"5830329c-8118-4fa1-9894-4318fcfcea13\" (UID: \"5830329c-8118-4fa1-9894-4318fcfcea13\") "
Sep 11 23:47:13.807402 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:13.807378    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5830329c-8118-4fa1-9894-4318fcfcea13-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5830329c-8118-4fa1-9894-4318fcfcea13" (UID: "5830329c-8118-4fa1-9894-4318fcfcea13"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 23:47:13.809466 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:13.809437    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5830329c-8118-4fa1-9894-4318fcfcea13-kube-api-access-rkgbf" (OuterVolumeSpecName: "kube-api-access-rkgbf") pod "5830329c-8118-4fa1-9894-4318fcfcea13" (UID: "5830329c-8118-4fa1-9894-4318fcfcea13"). InnerVolumeSpecName "kube-api-access-rkgbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 23:47:13.809466 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:13.809458    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5830329c-8118-4fa1-9894-4318fcfcea13-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5830329c-8118-4fa1-9894-4318fcfcea13" (UID: "5830329c-8118-4fa1-9894-4318fcfcea13"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 23:47:13.809612 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:13.809482    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5830329c-8118-4fa1-9894-4318fcfcea13-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5830329c-8118-4fa1-9894-4318fcfcea13" (UID: "5830329c-8118-4fa1-9894-4318fcfcea13"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 23:47:13.890379 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:13.890338    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6f85f54558-7l8hl"
Sep 11 23:47:13.907930 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:13.907886    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkgbf\" (UniqueName: \"kubernetes.io/projected/5830329c-8118-4fa1-9894-4318fcfcea13-kube-api-access-rkgbf\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:47:13.907930 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:13.907926    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5830329c-8118-4fa1-9894-4318fcfcea13-maas-api-tls\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:47:13.907930 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:13.907937    2546 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5830329c-8118-4fa1-9894-4318fcfcea13-metrics-certs\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:47:13.907930 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:13.907945    2546 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5830329c-8118-4fa1-9894-4318fcfcea13-openshift-service-ca\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 11 23:47:14.168929 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:14.168900    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6f85f54558-7l8hl"]
Sep 11 23:47:14.170304 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:47:14.170274    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0c6d2d0_4287_43fb_a39c_539765b74e68.slice/crio-9ceaa0921fdc7491bd7abf61e6cb3c9f5891306ec75f8ceb27fc505022a30ddd WatchSource:0}: Error finding container 9ceaa0921fdc7491bd7abf61e6cb3c9f5891306ec75f8ceb27fc505022a30ddd: Status 404 returned error can't find the container with id 9ceaa0921fdc7491bd7abf61e6cb3c9f5891306ec75f8ceb27fc505022a30ddd
Sep 11 23:47:14.650211 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:14.650172    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6f85f54558-7l8hl" event={"ID":"e0c6d2d0-4287-43fb-a39c-539765b74e68","Type":"ContainerStarted","Data":"8fb304222b72193ba183fb89f620e35bb6412a49a155f214f2fb6070515296a8"}
Sep 11 23:47:14.650710 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:14.650237    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6f85f54558-7l8hl" event={"ID":"e0c6d2d0-4287-43fb-a39c-539765b74e68","Type":"ContainerStarted","Data":"9ceaa0921fdc7491bd7abf61e6cb3c9f5891306ec75f8ceb27fc505022a30ddd"}
Sep 11 23:47:14.650710 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:14.650391    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6f85f54558-7l8hl"
Sep 11 23:47:14.651758 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:14.651733    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5759ff8db-zhptw"
Sep 11 23:47:14.651898 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:14.651770    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5759ff8db-zhptw" event={"ID":"5830329c-8118-4fa1-9894-4318fcfcea13","Type":"ContainerDied","Data":"8bb5ad50430ffc6468668de5b840ae5bf0dba86445bbe8c9e247d4a3ead7c1a0"}
Sep 11 23:47:14.651898 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:14.651822    2546 scope.go:117] "RemoveContainer" containerID="93cb5981a2d4ae8e5bc026f67088e908d4372df172dae8026f92ae153fdb9992"
Sep 11 23:47:14.669093 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:14.669040    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6f85f54558-7l8hl" podStartSLOduration=1.314007811 podStartE2EDuration="1.669021065s" totalImagesPullingTime="355.013254ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 23:47:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:47:14.667758465 +0000 UTC m=+2758.700887681" watchObservedRunningTime="2026-09-11 23:47:14.669021065 +0000 UTC m=+2758.702150333"
Sep 11 23:47:14.683561 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:14.683516    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5759ff8db-zhptw"]
Sep 11 23:47:14.686974 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:14.686941    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5759ff8db-zhptw"]
Sep 11 23:47:16.494511 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:16.494480    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5830329c-8118-4fa1-9894-4318fcfcea13" path="/var/lib/kubelet/pods/5830329c-8118-4fa1-9894-4318fcfcea13/volumes"
Sep 11 23:47:17.320803 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:17.320761    2546 scope.go:117] "RemoveContainer" containerID="1cb931065690e4abcdb8e0e9d638fdbfefc01b6e1f83c48febc72cd36bd56bb8"
Sep 11 23:47:20.021061 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:20.021020    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"]
Sep 11 23:47:20.664678 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:20.664650    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6f85f54558-7l8hl"
Sep 11 23:47:26.123141 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:26.123102    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"]
Sep 11 23:47:33.122457 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:33.122417    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwbzk"]
Sep 11 23:47:40.918054 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:40.917973    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-v8jrh_f4aa47f4-30c0-4e92-94d9-3b9b00889ca9/server/0.log"
Sep 11 23:47:41.028995 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:41.028959    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-wtlq2_66a2b731-2067-4a0d-99bd-29385ad13e94/manager/0.log"
Sep 11 23:47:41.271067 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:41.270966    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6857545767-t5kb6_d6280fd5-65fd-4c95-a616-19794456d570/copy-manifests/0.log"
Sep 11 23:47:41.278176 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:41.278126    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6857545767-t5kb6_d6280fd5-65fd-4c95-a616-19794456d570/manager/0.log"
Sep 11 23:47:41.815212 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:41.815142    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6f85f54558-7l8hl_e0c6d2d0-4287-43fb-a39c-539765b74e68/maas-api/0.log"
Sep 11 23:47:41.926780 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:41.926745    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29819490-kwbtx_d55d452e-62be-494c-a30f-ee01a215943a/cleanup/0.log"
Sep 11 23:47:42.037213 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:42.037183    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29819505-9xxxk_a3853d1b-2934-45ce-b67e-d167d927b854/cleanup/0.log"
Sep 11 23:47:43.412799 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:43.412759    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-pwbzk_c13e18e8-e890-44b5-988e-049d0b5cc447/limitador/0.log"
Sep 11 23:47:43.529786 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:43.529749    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-lf82j_5644f03d-9297-4d55-86fc-9db18e38cdae/manager/0.log"
Sep 11 23:47:43.859682 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:43.859591    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-zz8lz_92056a3e-aacd-45ed-9f05-a360ad3c8b21/istio-proxy/0.log"
Sep 11 23:47:43.971228 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:43.971186    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-qxgqr_f445a382-8507-45a5-bc72-5d760e31efe7/discovery/0.log"
Sep 11 23:47:44.184795 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:44.184764    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c75bd8f97-csrr2_a6196e75-6ca1-42e7-9846-8ad5850e4020/kube-auth-proxy/0.log"
Sep 11 23:47:44.296489 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:44.296452    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-vdztn_4a09df99-b62e-4670-9852-a452fa1c6cba/istio-proxy/1.log"
Sep 11 23:47:44.648808 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:44.648706    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-c8fff6cd4-tr742_42864299-d484-4b6b-a2c7-dea808adb9f6/router/0.log"
Sep 11 23:47:45.499093 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:45.499065    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-n87mz_a6c49bb6-490d-4570-8f86-212c566c5b90/main/0.log"
Sep 11 23:47:53.831583 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:53.831540    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-5n55v_3ad2cd03-8eae-49c4-8c94-ea45de3d3c7a/global-pull-secret-syncer/0.log"
Sep 11 23:47:54.007753 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:54.007719    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-mnljs_d2174c19-0971-419a-bc04-0c4662d5ff3c/konnectivity-agent/0.log"
Sep 11 23:47:54.059520 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:54.059483    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-161.ec2.internal_fbf860b5b0097ab5f709024d70f9c3d5/haproxy/0.log"
Sep 11 23:47:58.643705 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:58.643673    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-pwbzk_c13e18e8-e890-44b5-988e-049d0b5cc447/limitador/0.log"
Sep 11 23:47:58.696712 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:47:58.696672    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-lf82j_5644f03d-9297-4d55-86fc-9db18e38cdae/manager/0.log"
Sep 11 23:48:00.249593 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:00.249559    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cd954b897-h6w6f_a6dd2771-bd9e-4e48-976e-d2f4c14eba11/cluster-monitoring-operator/0.log"
Sep 11 23:48:00.285578 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:00.285546    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-p4wm5_c04c75b9-4bcf-4e64-9e21-f017aa288b6f/kube-state-metrics/0.log"
Sep 11 23:48:00.313895 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:00.313860    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-p4wm5_c04c75b9-4bcf-4e64-9e21-f017aa288b6f/kube-rbac-proxy-main/0.log"
Sep 11 23:48:00.338294 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:00.338239    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-p4wm5_c04c75b9-4bcf-4e64-9e21-f017aa288b6f/kube-rbac-proxy-self/0.log"
Sep 11 23:48:00.396677 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:00.396643    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-89z9j_df384b1d-9bce-49ed-814d-4a83c3c4c15c/monitoring-plugin/0.log"
Sep 11 23:48:00.491436 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:00.491407    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kwrgh_2668c2c0-570b-4b37-b41a-6baddf95128b/node-exporter/0.log"
Sep 11 23:48:00.510195 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:00.510048    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kwrgh_2668c2c0-570b-4b37-b41a-6baddf95128b/kube-rbac-proxy/0.log"
Sep 11 23:48:00.529974 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:00.529950    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kwrgh_2668c2c0-570b-4b37-b41a-6baddf95128b/init-textfile/0.log"
Sep 11 23:48:00.982214 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:00.982176    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-c5b4b88f5-b89zk_9e92a8c5-2c4c-4853-9934-d1034caac9a2/telemeter-client/0.log"
Sep 11 23:48:01.006257 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:01.006212    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-c5b4b88f5-b89zk_9e92a8c5-2c4c-4853-9934-d1034caac9a2/reload/0.log"
Sep 11 23:48:01.073392 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:01.073346    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-c5b4b88f5-b89zk_9e92a8c5-2c4c-4853-9934-d1034caac9a2/kube-rbac-proxy/0.log"
Sep 11 23:48:01.123590 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:01.123562    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8559bbc876-crxkz_f0617d4e-a6e6-47a9-9ba8-8902f4adda81/thanos-query/0.log"
Sep 11 23:48:01.146998 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:01.146971    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8559bbc876-crxkz_f0617d4e-a6e6-47a9-9ba8-8902f4adda81/kube-rbac-proxy-web/0.log"
Sep 11 23:48:01.171224 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:01.171193    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8559bbc876-crxkz_f0617d4e-a6e6-47a9-9ba8-8902f4adda81/kube-rbac-proxy/0.log"
Sep 11 23:48:01.197001 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:01.196976    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8559bbc876-crxkz_f0617d4e-a6e6-47a9-9ba8-8902f4adda81/prom-label-proxy/0.log"
Sep 11 23:48:01.218918 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:01.218888    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8559bbc876-crxkz_f0617d4e-a6e6-47a9-9ba8-8902f4adda81/kube-rbac-proxy-rules/0.log"
Sep 11 23:48:01.239246 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:01.239127    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8559bbc876-crxkz_f0617d4e-a6e6-47a9-9ba8-8902f4adda81/kube-rbac-proxy-metrics/0.log"
Sep 11 23:48:02.840122 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:02.840092    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-7q8hx_9783f5ec-7c57-4a7a-a688-3afad6cda55d/networking-console-plugin/0.log"
Sep 11 23:48:04.766112 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:04.766076    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-5tht9_8b564021-1e62-4eac-a86f-7a3a92980aa3/volume-data-source-validator/0.log"
Sep 11 23:48:05.002644 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:05.002602    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-m6z6g/perf-node-gather-daemonset-46t99"]
Sep 11 23:48:05.003118 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:05.003083    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5830329c-8118-4fa1-9894-4318fcfcea13" containerName="maas-api"
Sep 11 23:48:05.003118 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:05.003106    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="5830329c-8118-4fa1-9894-4318fcfcea13" containerName="maas-api"
Sep 11 23:48:05.003365 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:05.003276    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="5830329c-8118-4fa1-9894-4318fcfcea13" containerName="maas-api"
Sep 11 23:48:05.006730 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:05.006706    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-m6z6g/perf-node-gather-daemonset-46t99"
Sep 11 23:48:05.009862 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:05.009841    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-m6z6g\"/\"default-dockercfg-j5mj9\""
Sep 11 23:48:05.009974 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:05.009867    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-m6z6g\"/\"openshift-service-ca.crt\""
Sep 11 23:48:05.009974 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:05.009850    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-m6z6g\"/\"kube-root-ca.crt\""
Sep 11 23:48:05.015276 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:05.015250    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-m6z6g/perf-node-gather-daemonset-46t99"]
Sep 11 23:48:05.101800 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:05.101697    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nlqs\" (UniqueName: \"kubernetes.io/projected/33b73000-418b-4f4a-99b9-d9881c70d710-kube-api-access-4nlqs\") pod \"perf-node-gather-daemonset-46t99\" (UID: \"33b73000-418b-4f4a-99b9-d9881c70d710\") " pod="openshift-must-gather-m6z6g/perf-node-gather-daemonset-46t99"
Sep 11 23:48:05.101800 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:05.101751    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/33b73000-418b-4f4a-99b9-d9881c70d710-sys\") pod \"perf-node-gather-daemonset-46t99\" (UID: \"33b73000-418b-4f4a-99b9-d9881c70d710\") " pod="openshift-must-gather-m6z6g/perf-node-gather-daemonset-46t99"
Sep 11 23:48:05.101997 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:05.101810    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/33b73000-418b-4f4a-99b9-d9881c70d710-lib-modules\") pod \"perf-node-gather-daemonset-46t99\" (UID: \"33b73000-418b-4f4a-99b9-d9881c70d710\") " pod="openshift-must-gather-m6z6g/perf-node-gather-daemonset-46t99"
Sep 11 23:48:05.101997 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:05.101849    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/33b73000-418b-4f4a-99b9-d9881c70d710-proc\") pod \"perf-node-gather-daemonset-46t99\" (UID: \"33b73000-418b-4f4a-99b9-d9881c70d710\") " pod="openshift-must-gather-m6z6g/perf-node-gather-daemonset-46t99"
Sep 11 23:48:05.101997 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:05.101875    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/33b73000-418b-4f4a-99b9-d9881c70d710-podres\") pod \"perf-node-gather-daemonset-46t99\" (UID: \"33b73000-418b-4f4a-99b9-d9881c70d710\") " pod="openshift-must-gather-m6z6g/perf-node-gather-daemonset-46t99"
Sep 11 23:48:05.202950 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:05.202894    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/33b73000-418b-4f4a-99b9-d9881c70d710-lib-modules\") pod \"perf-node-gather-daemonset-46t99\" (UID: \"33b73000-418b-4f4a-99b9-d9881c70d710\") " pod="openshift-must-gather-m6z6g/perf-node-gather-daemonset-46t99"
Sep 11 23:48:05.202950 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:05.202960    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/33b73000-418b-4f4a-99b9-d9881c70d710-proc\") pod \"perf-node-gather-daemonset-46t99\" (UID: \"33b73000-418b-4f4a-99b9-d9881c70d710\") " pod="openshift-must-gather-m6z6g/perf-node-gather-daemonset-46t99"
Sep 11 23:48:05.203146 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:05.202982    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/33b73000-418b-4f4a-99b9-d9881c70d710-podres\") pod \"perf-node-gather-daemonset-46t99\" (UID: \"33b73000-418b-4f4a-99b9-d9881c70d710\") " pod="openshift-must-gather-m6z6g/perf-node-gather-daemonset-46t99"
Sep 11 23:48:05.203146 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:05.203042    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nlqs\" (UniqueName: \"kubernetes.io/projected/33b73000-418b-4f4a-99b9-d9881c70d710-kube-api-access-4nlqs\") pod \"perf-node-gather-daemonset-46t99\" (UID: \"33b73000-418b-4f4a-99b9-d9881c70d710\") " pod="openshift-must-gather-m6z6g/perf-node-gather-daemonset-46t99"
Sep 11 23:48:05.203146 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:05.203060    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/33b73000-418b-4f4a-99b9-d9881c70d710-sys\") pod \"perf-node-gather-daemonset-46t99\" (UID: \"33b73000-418b-4f4a-99b9-d9881c70d710\") " pod="openshift-must-gather-m6z6g/perf-node-gather-daemonset-46t99"
Sep 11 23:48:05.203146 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:05.203108    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/33b73000-418b-4f4a-99b9-d9881c70d710-lib-modules\") pod \"perf-node-gather-daemonset-46t99\" (UID: \"33b73000-418b-4f4a-99b9-d9881c70d710\") " pod="openshift-must-gather-m6z6g/perf-node-gather-daemonset-46t99"
Sep 11 23:48:05.203146 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:05.203108    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/33b73000-418b-4f4a-99b9-d9881c70d710-proc\") pod \"perf-node-gather-daemonset-46t99\" (UID: \"33b73000-418b-4f4a-99b9-d9881c70d710\") " pod="openshift-must-gather-m6z6g/perf-node-gather-daemonset-46t99"
Sep 11 23:48:05.203365 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:05.203182    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/33b73000-418b-4f4a-99b9-d9881c70d710-podres\") pod \"perf-node-gather-daemonset-46t99\" (UID: \"33b73000-418b-4f4a-99b9-d9881c70d710\") " pod="openshift-must-gather-m6z6g/perf-node-gather-daemonset-46t99"
Sep 11 23:48:05.203365 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:05.203187    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/33b73000-418b-4f4a-99b9-d9881c70d710-sys\") pod \"perf-node-gather-daemonset-46t99\" (UID: \"33b73000-418b-4f4a-99b9-d9881c70d710\") " pod="openshift-must-gather-m6z6g/perf-node-gather-daemonset-46t99"
Sep 11 23:48:05.211510 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:05.211489    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nlqs\" (UniqueName: \"kubernetes.io/projected/33b73000-418b-4f4a-99b9-d9881c70d710-kube-api-access-4nlqs\") pod \"perf-node-gather-daemonset-46t99\" (UID: \"33b73000-418b-4f4a-99b9-d9881c70d710\") " pod="openshift-must-gather-m6z6g/perf-node-gather-daemonset-46t99"
Sep 11 23:48:05.318002 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:05.317956    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-m6z6g/perf-node-gather-daemonset-46t99"
Sep 11 23:48:05.595730 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:05.595692    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-m6z6g/perf-node-gather-daemonset-46t99"]
Sep 11 23:48:05.597336 ip-10-0-128-161 kubenswrapper[2546]: W0911 23:48:05.597301    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod33b73000_418b_4f4a_99b9_d9881c70d710.slice/crio-125c04141781625cfb05db09d979e5021113a52e28df5dc48ca106a87e016ee3 WatchSource:0}: Error finding container 125c04141781625cfb05db09d979e5021113a52e28df5dc48ca106a87e016ee3: Status 404 returned error can't find the container with id 125c04141781625cfb05db09d979e5021113a52e28df5dc48ca106a87e016ee3
Sep 11 23:48:05.973420 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:05.973380    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-m6z6g/perf-node-gather-daemonset-46t99" event={"ID":"33b73000-418b-4f4a-99b9-d9881c70d710","Type":"ContainerStarted","Data":"d6658369501bdff39250d9d8581976353f0c4e365e13ccf3e6ca90ecee94c5ac"}
Sep 11 23:48:05.973911 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:05.973435    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-m6z6g/perf-node-gather-daemonset-46t99" event={"ID":"33b73000-418b-4f4a-99b9-d9881c70d710","Type":"ContainerStarted","Data":"125c04141781625cfb05db09d979e5021113a52e28df5dc48ca106a87e016ee3"}
Sep 11 23:48:05.973911 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:05.973589    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-m6z6g/perf-node-gather-daemonset-46t99"
Sep 11 23:48:05.991552 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:05.991497    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-m6z6g/perf-node-gather-daemonset-46t99" podStartSLOduration=1.991477637 podStartE2EDuration="1.991477637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 23:48:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 23:48:05.989102241 +0000 UTC m=+2810.022231458" watchObservedRunningTime="2026-09-11 23:48:05.991477637 +0000 UTC m=+2810.024606853"
Sep 11 23:48:06.029337 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:06.029307    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fzfcr_98962ebe-ac82-4d92-a979-022ad4873bc9/dns/0.log"
Sep 11 23:48:06.050814 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:06.050761    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fzfcr_98962ebe-ac82-4d92-a979-022ad4873bc9/kube-rbac-proxy/0.log"
Sep 11 23:48:06.160674 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:06.160634    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vqs5t_9dcaac8a-b529-4ba2-a3f3-a617817741c2/dns-node-resolver/0.log"
Sep 11 23:48:06.751608 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:06.751582    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zgtd5_aa21ad55-513d-46a1-89ce-78e88c3e0efa/node-ca/0.log"
Sep 11 23:48:07.647765 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:07.647735    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-zz8lz_92056a3e-aacd-45ed-9f05-a360ad3c8b21/istio-proxy/0.log"
Sep 11 23:48:08.097494 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:08.097458    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-qxgqr_f445a382-8507-45a5-bc72-5d760e31efe7/discovery/0.log"
Sep 11 23:48:08.142319 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:08.142291    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c75bd8f97-csrr2_a6196e75-6ca1-42e7-9846-8ad5850e4020/kube-auth-proxy/0.log"
Sep 11 23:48:08.170718 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:08.170687    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-vdztn_4a09df99-b62e-4670-9852-a452fa1c6cba/istio-proxy/0.log"
Sep 11 23:48:08.173395 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:08.173372    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-vdztn_4a09df99-b62e-4670-9852-a452fa1c6cba/istio-proxy/1.log"
Sep 11 23:48:08.263178 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:08.263127    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-c8fff6cd4-tr742_42864299-d484-4b6b-a2c7-dea808adb9f6/router/0.log"
Sep 11 23:48:09.125726 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:09.125692    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-glt22_efcb43e4-c096-422f-bc0b-2c42795b0adc/serve-healthcheck-canary/0.log"
Sep 11 23:48:09.614295 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:09.614244    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-xr6z6_68e4123a-25ea-4c58-a934-d44495740042/insights-operator/0.log"
Sep 11 23:48:09.614850 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:09.614829    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-xr6z6_68e4123a-25ea-4c58-a934-d44495740042/insights-operator/1.log"
Sep 11 23:48:09.638215 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:09.638187    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-t2g8v_b24f52c2-9cad-4c69-9b65-f6752ecf1afb/kube-rbac-proxy/0.log"
Sep 11 23:48:09.660303 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:09.660275    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-t2g8v_b24f52c2-9cad-4c69-9b65-f6752ecf1afb/exporter/0.log"
Sep 11 23:48:09.681244 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:09.681211    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-t2g8v_b24f52c2-9cad-4c69-9b65-f6752ecf1afb/extractor/0.log"
Sep 11 23:48:11.993543 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:11.993512    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-m6z6g/perf-node-gather-daemonset-46t99"
Sep 11 23:48:12.290232 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:12.290140    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-v8jrh_f4aa47f4-30c0-4e92-94d9-3b9b00889ca9/server/0.log"
Sep 11 23:48:12.389736 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:12.389701    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-wtlq2_66a2b731-2067-4a0d-99bd-29385ad13e94/manager/0.log"
Sep 11 23:48:12.557548 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:12.557467    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6857545767-t5kb6_d6280fd5-65fd-4c95-a616-19794456d570/manager/0.log"
Sep 11 23:48:12.576871 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:12.576846    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6857545767-t5kb6_d6280fd5-65fd-4c95-a616-19794456d570/copy-manifests/0.log"
Sep 11 23:48:14.505204 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:14.505174    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-zz8lz_92056a3e-aacd-45ed-9f05-a360ad3c8b21/istio-proxy/0.log"
Sep 11 23:48:14.922830 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:14.922791    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-qxgqr_f445a382-8507-45a5-bc72-5d760e31efe7/discovery/0.log"
Sep 11 23:48:14.947949 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:14.947916    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c75bd8f97-csrr2_a6196e75-6ca1-42e7-9846-8ad5850e4020/kube-auth-proxy/0.log"
Sep 11 23:48:14.970955 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:14.970924    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-vdztn_4a09df99-b62e-4670-9852-a452fa1c6cba/istio-proxy/0.log"
Sep 11 23:48:14.973369 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:14.973348    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-vdztn_4a09df99-b62e-4670-9852-a452fa1c6cba/istio-proxy/1.log"
Sep 11 23:48:15.040554 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:15.040517    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-c8fff6cd4-tr742_42864299-d484-4b6b-a2c7-dea808adb9f6/router/0.log"
Sep 11 23:48:16.697075 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:16.697047    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-zz8lz_92056a3e-aacd-45ed-9f05-a360ad3c8b21/istio-proxy/0.log"
Sep 11 23:48:17.144575 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:17.144468    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-qxgqr_f445a382-8507-45a5-bc72-5d760e31efe7/discovery/0.log"
Sep 11 23:48:17.188072 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:17.188037    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c75bd8f97-csrr2_a6196e75-6ca1-42e7-9846-8ad5850e4020/kube-auth-proxy/0.log"
Sep 11 23:48:17.209722 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:17.209685    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-vdztn_4a09df99-b62e-4670-9852-a452fa1c6cba/istio-proxy/0.log"
Sep 11 23:48:17.212730 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:17.212707    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-vdztn_4a09df99-b62e-4670-9852-a452fa1c6cba/istio-proxy/1.log"
Sep 11 23:48:17.288958 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:17.288920    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-c8fff6cd4-tr742_42864299-d484-4b6b-a2c7-dea808adb9f6/router/0.log"
Sep 11 23:48:19.274607 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:19.274572    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-4tdps_51cae370-f275-4cc9-8c5e-5f5e6998b6a1/migrator/0.log"
Sep 11 23:48:19.296668 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:19.296628    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-4tdps_51cae370-f275-4cc9-8c5e-5f5e6998b6a1/graceful-termination/0.log"
Sep 11 23:48:20.960071 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:20.960041    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bv5gf_0155dec9-f8e2-4007-9861-b8c870223adb/kube-multus-additional-cni-plugins/0.log"
Sep 11 23:48:20.979952 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:20.979922    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bv5gf_0155dec9-f8e2-4007-9861-b8c870223adb/egress-router-binary-copy/0.log"
Sep 11 23:48:20.999808 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:20.999777    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bv5gf_0155dec9-f8e2-4007-9861-b8c870223adb/cni-plugins/0.log"
Sep 11 23:48:21.019542 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:21.019516    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bv5gf_0155dec9-f8e2-4007-9861-b8c870223adb/bond-cni-plugin/0.log"
Sep 11 23:48:21.038508 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:21.038478    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bv5gf_0155dec9-f8e2-4007-9861-b8c870223adb/routeoverride-cni/0.log"
Sep 11 23:48:21.057217 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:21.057182    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bv5gf_0155dec9-f8e2-4007-9861-b8c870223adb/whereabouts-cni-bincopy/0.log"
Sep 11 23:48:21.075552 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:21.075519    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bv5gf_0155dec9-f8e2-4007-9861-b8c870223adb/whereabouts-cni/0.log"
Sep 11 23:48:21.123303 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:21.123268    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-lhmpr_a489f10b-bf16-40dc-aa5f-dc1d8228aa32/kube-multus/0.log"
Sep 11 23:48:21.332549 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:21.332457    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-vbgsz_50607ccd-47f9-463a-bdc7-181484a00caa/network-metrics-daemon/0.log"
Sep 11 23:48:21.348489 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:21.348454    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-vbgsz_50607ccd-47f9-463a-bdc7-181484a00caa/kube-rbac-proxy/0.log"
Sep 11 23:48:22.418265 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:22.418230    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7244_6b0a8af3-932c-4091-b7a0-180fe1751a3a/ovn-controller/0.log"
Sep 11 23:48:22.435603 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:22.435569    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7244_6b0a8af3-932c-4091-b7a0-180fe1751a3a/ovn-acl-logging/0.log"
Sep 11 23:48:22.435603 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:22.435601    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7244_6b0a8af3-932c-4091-b7a0-180fe1751a3a/ovn-acl-logging/1.log"
Sep 11 23:48:22.456078 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:22.456049    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7244_6b0a8af3-932c-4091-b7a0-180fe1751a3a/kube-rbac-proxy-node/0.log"
Sep 11 23:48:22.477001 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:22.476966    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7244_6b0a8af3-932c-4091-b7a0-180fe1751a3a/kube-rbac-proxy-ovn-metrics/0.log"
Sep 11 23:48:22.494030 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:22.493998    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7244_6b0a8af3-932c-4091-b7a0-180fe1751a3a/northd/0.log"
Sep 11 23:48:22.513996 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:22.513962    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7244_6b0a8af3-932c-4091-b7a0-180fe1751a3a/nbdb/0.log"
Sep 11 23:48:22.533624 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:22.533596    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7244_6b0a8af3-932c-4091-b7a0-180fe1751a3a/sbdb/0.log"
Sep 11 23:48:22.598740 ip-10-0-128-161 kubenswrapper[2546]: I0911 23:48:22.598706    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7244_6b0a8af3-932c-4091-b7a0-180fe1751a3a/ovnkube-controller/0.log"