Sep 18 19:54:50.168405 ip-10-0-140-178 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 18 19:54:50.168419 ip-10-0-140-178 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 18 19:54:50.168428 ip-10-0-140-178 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 18 19:54:50.168749 ip-10-0-140-178 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 18 19:55:00.245235 ip-10-0-140-178 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 18 19:55:00.245251 ip-10-0-140-178 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot aeedd41f6038472e9792ddcebf99a2fd --
Sep 18 19:56:54.945665 ip-10-0-140-178 systemd[1]: Starting Kubernetes Kubelet...
Sep 18 19:56:55.294559 ip-10-0-140-178 kubenswrapper[2549]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 19:56:55.294559 ip-10-0-140-178 kubenswrapper[2549]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 18 19:56:55.294559 ip-10-0-140-178 kubenswrapper[2549]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 19:56:55.294559 ip-10-0-140-178 kubenswrapper[2549]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 18 19:56:55.294559 ip-10-0-140-178 kubenswrapper[2549]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 19:56:55.295211 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.294576    2549 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 18 19:56:55.298256 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298240    2549 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:56:55.298256 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298255    2549 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:56:55.298316 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298259    2549 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:56:55.298316 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298263    2549 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:56:55.298316 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298266    2549 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:56:55.298316 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298270    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:56:55.298316 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298274    2549 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:56:55.298316 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298277    2549 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:56:55.298316 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298280    2549 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:56:55.298316 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298283    2549 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:56:55.298316 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298285    2549 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:56:55.298316 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298292    2549 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:56:55.298316 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298295    2549 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:56:55.298316 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298297    2549 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:56:55.298316 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298300    2549 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:56:55.298316 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298303    2549 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:56:55.298316 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298306    2549 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:56:55.298316 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298309    2549 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:56:55.298316 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298312    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:56:55.298316 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298314    2549 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:56:55.298316 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298317    2549 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:56:55.298316 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298319    2549 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:56:55.298803 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298322    2549 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:56:55.298803 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298325    2549 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:56:55.298803 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298328    2549 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:56:55.298803 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298331    2549 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:56:55.298803 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298334    2549 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:56:55.298803 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298336    2549 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:56:55.298803 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298339    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:56:55.298803 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298342    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:56:55.298803 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298344    2549 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:56:55.298803 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298347    2549 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:56:55.298803 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298349    2549 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:56:55.298803 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298352    2549 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:56:55.298803 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298355    2549 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:56:55.298803 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298358    2549 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:56:55.298803 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298360    2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:56:55.298803 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298363    2549 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:56:55.298803 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298365    2549 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:56:55.298803 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298368    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:56:55.298803 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298370    2549 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:56:55.298803 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298373    2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:56:55.299614 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298375    2549 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:56:55.299614 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298378    2549 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:56:55.299614 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298391    2549 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:56:55.299614 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298393    2549 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:56:55.299614 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298396    2549 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:56:55.299614 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298399    2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:56:55.299614 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298401    2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:56:55.299614 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298404    2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:56:55.299614 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298406    2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:56:55.299614 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298409    2549 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:56:55.299614 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298411    2549 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:56:55.299614 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298414    2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:56:55.299614 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298417    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:56:55.299614 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298429    2549 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:56:55.299614 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298433    2549 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:56:55.299614 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298436    2549 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:56:55.299614 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298439    2549 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:56:55.299614 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298442    2549 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:56:55.299614 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298445    2549 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:56:55.300094 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298448    2549 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:56:55.300094 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298451    2549 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:56:55.300094 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298454    2549 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:56:55.300094 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298457    2549 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:56:55.300094 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298460    2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:56:55.300094 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298464    2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:56:55.300094 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298466    2549 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:56:55.300094 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298469    2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:56:55.300094 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298472    2549 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:56:55.300094 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298474    2549 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:56:55.300094 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298477    2549 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:56:55.300094 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298480    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:56:55.300094 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298482    2549 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:56:55.300094 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298485    2549 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:56:55.300094 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298488    2549 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:56:55.300094 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298490    2549 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:56:55.300094 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298493    2549 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:56:55.300094 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298495    2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:56:55.300094 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298500    2549 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:56:55.300094 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298504    2549 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:56:55.300588 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298507    2549 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:56:55.300588 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298509    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:56:55.300588 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298512    2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:56:55.300588 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298514    2549 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:56:55.300588 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298517    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:56:55.300588 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298520    2549 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:56:55.300588 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298523    2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:56:55.300588 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298526    2549 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:56:55.300588 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298528    2549 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:56:55.300588 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298531    2549 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:56:55.300588 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298533    2549 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:56:55.300588 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298536    2549 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:56:55.300588 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298539    2549 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:56:55.300588 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298542    2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:56:55.300588 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298544    2549 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:56:55.300588 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298548    2549 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:56:55.300588 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298551    2549 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:56:55.300588 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.298554    2549 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:56:55.301706 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301690    2549 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:56:55.301706 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301706    2549 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:56:55.301767 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301710    2549 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:56:55.301767 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301714    2549 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:56:55.301767 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301716    2549 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:56:55.301767 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301719    2549 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:56:55.301767 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301722    2549 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:56:55.301767 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301724    2549 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:56:55.301767 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301727    2549 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:56:55.301767 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301730    2549 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:56:55.301767 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301733    2549 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:56:55.301767 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301735    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:56:55.301767 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301738    2549 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:56:55.301767 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301740    2549 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:56:55.301767 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301743    2549 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:56:55.301767 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301746    2549 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:56:55.301767 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301749    2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:56:55.301767 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301752    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:56:55.301767 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301755    2549 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:56:55.301767 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301757    2549 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:56:55.301767 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301760    2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:56:55.301767 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301763    2549 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:56:55.302310 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301765    2549 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:56:55.302310 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301768    2549 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:56:55.302310 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301770    2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:56:55.302310 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301773    2549 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:56:55.302310 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301776    2549 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:56:55.302310 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301779    2549 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:56:55.302310 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301782    2549 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:56:55.302310 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301785    2549 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:56:55.302310 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301787    2549 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:56:55.302310 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301799    2549 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:56:55.302310 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301802    2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:56:55.302310 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301805    2549 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:56:55.302310 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301807    2549 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:56:55.302310 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301810    2549 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:56:55.302310 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301812    2549 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:56:55.302310 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301815    2549 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:56:55.302310 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301818    2549 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:56:55.302310 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301820    2549 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:56:55.302310 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301823    2549 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:56:55.302310 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301826    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:56:55.302807 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301828    2549 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:56:55.302807 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301833    2549 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:56:55.302807 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301836    2549 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:56:55.302807 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301839    2549 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:56:55.302807 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301842    2549 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:56:55.302807 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301845    2549 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:56:55.302807 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301847    2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:56:55.302807 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301850    2549 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:56:55.302807 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301852    2549 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:56:55.302807 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301856    2549 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:56:55.302807 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301860    2549 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:56:55.302807 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301863    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:56:55.302807 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301866    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:56:55.302807 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301869    2549 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:56:55.302807 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301871    2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:56:55.302807 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301874    2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:56:55.302807 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301877    2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:56:55.302807 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301880    2549 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:56:55.302807 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301883    2549 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:56:55.303272 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301886    2549 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:56:55.303272 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301890    2549 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:56:55.303272 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301894    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:56:55.303272 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301897    2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:56:55.303272 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301900    2549 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:56:55.303272 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301903    2549 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:56:55.303272 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301906    2549 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:56:55.303272 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301909    2549 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:56:55.303272 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301911    2549 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:56:55.303272 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301914    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:56:55.303272 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301917    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:56:55.303272 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301919    2549 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:56:55.303272 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301922    2549 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:56:55.303272 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301926    2549 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:56:55.303272 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301929    2549 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:56:55.303272 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301932    2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:56:55.303272 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301934    2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:56:55.303272 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301937    2549 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:56:55.303272 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301940    2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:56:55.303751 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301942    2549 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:56:55.303751 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301945    2549 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:56:55.303751 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301947    2549 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:56:55.303751 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301951    2549 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:56:55.303751 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301954    2549 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:56:55.303751 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301957    2549 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:56:55.303751 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301960    2549 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:56:55.303751 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301963    2549 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:56:55.303751 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301965    2549 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:56:55.303751 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301968    2549 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:56:55.303751 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301971    2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:56:55.303751 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301973    2549 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:56:55.303751 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301976    2549 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:56:55.303751 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301978    2549 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:56:55.303751 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301981    2549 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:56:55.303751 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301988    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:56:55.303751 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301991    2549 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:56:55.303751 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301993    2549 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:56:55.303751 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.301996    2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:56:55.303751 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302413    2549 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:56:55.304238 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302420    2549 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:56:55.304238 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302423    2549 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:56:55.304238 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302427    2549 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:56:55.304238 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302429    2549 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:56:55.304238 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302432    2549 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:56:55.304238 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302435    2549 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:56:55.304238 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302437    2549 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:56:55.304238 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302440    2549 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:56:55.304238 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302442    2549 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:56:55.304238 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302445    2549 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:56:55.304238 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302447    2549 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:56:55.304238 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302450    2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:56:55.304238 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302455    2549 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:56:55.304238 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302458    2549 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:56:55.304238 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302461    2549 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:56:55.304238 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302470    2549 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:56:55.304238 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302473    2549 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:56:55.304238 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302476    2549 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:56:55.304238 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302478    2549 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:56:55.304238 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302482    2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:56:55.304736 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302485    2549 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:56:55.304736 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302489    2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:56:55.304736 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302492    2549 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:56:55.304736 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302494    2549 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:56:55.304736 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302498    2549 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:56:55.304736 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302502    2549 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:56:55.304736 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302505    2549 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:56:55.304736 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302508    2549 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:56:55.304736 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302511    2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:56:55.304736 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302513    2549 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:56:55.304736 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302516    2549 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:56:55.304736 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302518    2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:56:55.304736 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302521    2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:56:55.304736 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302524    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:56:55.304736 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302527    2549 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:56:55.304736 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302530    2549 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:56:55.304736 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302532    2549 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:56:55.304736 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302535    2549 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:56:55.304736 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302537    2549 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:56:55.305195 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302540    2549 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:56:55.305195 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302543    2549 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:56:55.305195 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302545    2549 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:56:55.305195 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302548    2549 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:56:55.305195 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302550    2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:56:55.305195 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302555    2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:56:55.305195 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302558    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:56:55.305195 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302560    2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:56:55.305195 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302563    2549 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:56:55.305195 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302567    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:56:55.305195 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302570    2549 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:56:55.305195 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302572    2549 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:56:55.305195 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302575    2549 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:56:55.305195 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302578    2549 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:56:55.305195 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302580    2549 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:56:55.305195 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302583    2549 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:56:55.305195 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302585    2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:56:55.305195 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302588    2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:56:55.305195 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302590    2549 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:56:55.305195 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302593    2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:56:55.305692 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302595    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:56:55.305692 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302598    2549 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:56:55.305692 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302601    2549 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:56:55.305692 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302603    2549 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:56:55.305692 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302606    2549 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:56:55.305692 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302608    2549 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:56:55.305692 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302611    2549 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:56:55.305692 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302613    2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:56:55.305692 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302616    2549 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:56:55.305692 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302618    2549 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:56:55.305692 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302621    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:56:55.305692 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302623    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:56:55.305692 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302626    2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:56:55.305692 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302628    2549 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:56:55.305692 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302631    2549 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:56:55.305692 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302633    2549 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:56:55.305692 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302635    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:56:55.305692 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302639    2549 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:56:55.305692 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302641    2549 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:56:55.306237 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302644    2549 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:56:55.306237 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302646    2549 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:56:55.306237 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302649    2549 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:56:55.306237 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302652    2549 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:56:55.306237 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302655    2549 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:56:55.306237 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302658    2549 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:56:55.306237 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302661    2549 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:56:55.306237 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302664    2549 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:56:55.306237 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302666    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:56:55.306237 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302669    2549 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:56:55.306237 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302671    2549 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:56:55.306237 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302674    2549 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:56:55.306237 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302676    2549 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:56:55.306237 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302679    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:56:55.306237 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302681    2549 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:56:55.306237 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302684    2549 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:56:55.306237 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302686    2549 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:56:55.306237 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302689    2549 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:56:55.306237 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302691    2549 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:56:55.306237 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.302694    2549 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:56:55.306237 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.302764    2549 flags.go:64] FLAG: --address="0.0.0.0"
Sep 18 19:56:55.306765 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.302799    2549 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 18 19:56:55.306765 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.302805    2549 flags.go:64] FLAG: --anonymous-auth="true"
Sep 18 19:56:55.306765 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.302809    2549 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 18 19:56:55.306765 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.302814    2549 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 18 19:56:55.306765 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.302818    2549 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 18 19:56:55.306765 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.302827    2549 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 18 19:56:55.306765 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.302831    2549 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 18 19:56:55.306765 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.302834    2549 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 18 19:56:55.306765 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.302837    2549 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 18 19:56:55.306765 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.302842    2549 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 18 19:56:55.306765 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.302845    2549 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 18 19:56:55.306765 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.302848    2549 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 18 19:56:55.306765 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.302851    2549 flags.go:64] FLAG: --cgroup-root=""
Sep 18 19:56:55.306765 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.302854    2549 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 18 19:56:55.306765 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.302857    2549 flags.go:64] FLAG: --client-ca-file=""
Sep 18 19:56:55.306765 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.302861    2549 flags.go:64] FLAG: --cloud-provider="external"
Sep 18 19:56:55.306765 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.302864    2549 flags.go:64] FLAG: --cluster-dns="[]"
Sep 18 19:56:55.306765 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.302869    2549 flags.go:64] FLAG: --cluster-domain=""
Sep 18 19:56:55.306765 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.302872    2549 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 18 19:56:55.306765 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.302875    2549 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 18 19:56:55.306765 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.302878    2549 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 18 19:56:55.306765 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.302882    2549 flags.go:64] FLAG: --container-log-max-files="5"
Sep 18 19:56:55.306765 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.302885    2549 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 18 19:56:55.306765 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.302888    2549 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 18 19:56:55.307339 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.302891    2549 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 18 19:56:55.307339 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.302894    2549 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 18 19:56:55.307339 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.302897    2549 flags.go:64] FLAG: --contention-profiling="false"
Sep 18 19:56:55.307339 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.302901    2549 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 18 19:56:55.307339 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.302904    2549 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 18 19:56:55.307339 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.302907    2549 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 18 19:56:55.307339 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.302910    2549 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 18 19:56:55.307339 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.302913    2549 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 18 19:56:55.307339 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.302916    2549 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 18 19:56:55.307339 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.302919    2549 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 18 19:56:55.307339 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.302922    2549 flags.go:64] FLAG: --enable-load-reader="false"
Sep 18 19:56:55.307339 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.302925    2549 flags.go:64] FLAG: --enable-server="true"
Sep 18 19:56:55.307339 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.302928    2549 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 18 19:56:55.307339 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.302932    2549 flags.go:64] FLAG: --event-burst="100"
Sep 18 19:56:55.307339 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.302936    2549 flags.go:64] FLAG: --event-qps="50"
Sep 18 19:56:55.307339 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.302938    2549 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 18 19:56:55.307339 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.302942    2549 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 18 19:56:55.307339 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.302946    2549 flags.go:64] FLAG: --eviction-hard=""
Sep 18 19:56:55.307339 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.302949    2549 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 18 19:56:55.307339 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.302952    2549 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 18 19:56:55.307339 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.302955    2549 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 18 19:56:55.307339 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.302958    2549 flags.go:64] FLAG: --eviction-soft=""
Sep 18 19:56:55.307339 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.302961    2549 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 18 19:56:55.307339 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.302964    2549 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 18 19:56:55.307339 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.302968    2549 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 18 19:56:55.308003 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.302971    2549 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 18 19:56:55.308003 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.302974    2549 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 18 19:56:55.308003 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.302976    2549 flags.go:64] FLAG: --fail-swap-on="true"
Sep 18 19:56:55.308003 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.302979    2549 flags.go:64] FLAG: --feature-gates=""
Sep 18 19:56:55.308003 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.302983    2549 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 18 19:56:55.308003 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.302986    2549 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 18 19:56:55.308003 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.302989    2549 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 18 19:56:55.308003 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.302992    2549 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 18 19:56:55.308003 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.302995    2549 flags.go:64] FLAG: --healthz-port="10248"
Sep 18 19:56:55.308003 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.302998    2549 flags.go:64] FLAG: --help="false"
Sep 18 19:56:55.308003 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303001    2549 flags.go:64] FLAG: --hostname-override="ip-10-0-140-178.ec2.internal"
Sep 18 19:56:55.308003 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303004    2549 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 18 19:56:55.308003 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303008    2549 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 18 19:56:55.308003 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303011    2549 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 18 19:56:55.308003 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303014    2549 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 18 19:56:55.308003 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303018    2549 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 18 19:56:55.308003 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303021    2549 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 18 19:56:55.308003 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303024    2549 flags.go:64] FLAG: --image-service-endpoint=""
Sep 18 19:56:55.308003 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303026    2549 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 18 19:56:55.308003 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303029    2549 flags.go:64] FLAG: --kube-api-burst="100"
Sep 18 19:56:55.308003 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303032    2549 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 18 19:56:55.308003 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303035    2549 flags.go:64] FLAG: --kube-api-qps="50"
Sep 18 19:56:55.308003 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303038    2549 flags.go:64] FLAG: --kube-reserved=""
Sep 18 19:56:55.308003 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303041    2549 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 18 19:56:55.308614 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303045    2549 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 18 19:56:55.308614 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303048    2549 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 18 19:56:55.308614 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303051    2549 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 18 19:56:55.308614 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303053    2549 flags.go:64] FLAG: --lock-file=""
Sep 18 19:56:55.308614 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303056    2549 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 18 19:56:55.308614 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303059    2549 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 18 19:56:55.308614 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303062    2549 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 18 19:56:55.308614 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303068    2549 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 18 19:56:55.308614 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303072    2549 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 18 19:56:55.308614 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303075    2549 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 18 19:56:55.308614 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303078    2549 flags.go:64] FLAG: --logging-format="text"
Sep 18 19:56:55.308614 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303081    2549 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 18 19:56:55.308614 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303084    2549 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 18 19:56:55.308614 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303087    2549 flags.go:64] FLAG: --manifest-url=""
Sep 18 19:56:55.308614 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303089    2549 flags.go:64] FLAG: --manifest-url-header=""
Sep 18 19:56:55.308614 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303093    2549 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 18 19:56:55.308614 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303096    2549 flags.go:64] FLAG: --max-open-files="1000000"
Sep 18 19:56:55.308614 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303100    2549 flags.go:64] FLAG: --max-pods="110"
Sep 18 19:56:55.308614 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303103    2549 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 18 19:56:55.308614 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303106    2549 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 18 19:56:55.308614 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303109    2549 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 18 19:56:55.308614 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303112    2549 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 18 19:56:55.308614 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303115    2549 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 18 19:56:55.308614 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303118    2549 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 18 19:56:55.308614 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303121    2549 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 18 19:56:55.309221 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303128    2549 flags.go:64] FLAG: --node-status-max-images="50"
Sep 18 19:56:55.309221 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303131    2549 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 18 19:56:55.309221 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303135    2549 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 18 19:56:55.309221 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303138    2549 flags.go:64] FLAG: --pod-cidr=""
Sep 18 19:56:55.309221 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303141    2549 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 18 19:56:55.309221 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303153    2549 flags.go:64] FLAG: --pod-manifest-path=""
Sep 18 19:56:55.309221 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303157    2549 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 18 19:56:55.309221 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303160    2549 flags.go:64] FLAG: --pods-per-core="0"
Sep 18 19:56:55.309221 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303164    2549 flags.go:64] FLAG: --port="10250"
Sep 18 19:56:55.309221 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303167    2549 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 18 19:56:55.309221 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303170    2549 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-01bd75737a3d1d92f"
Sep 18 19:56:55.309221 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303173    2549 flags.go:64] FLAG: --qos-reserved=""
Sep 18 19:56:55.309221 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303176    2549 flags.go:64] FLAG: --read-only-port="10255"
Sep 18 19:56:55.309221 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303179    2549 flags.go:64] FLAG: --register-node="true"
Sep 18 19:56:55.309221 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303182    2549 flags.go:64] FLAG: --register-with-taints=""
Sep 18 19:56:55.309221 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303186    2549 flags.go:64] FLAG: --registry-burst="10"
Sep 18 19:56:55.309221 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303190    2549 flags.go:64] FLAG: --registry-qps="5"
Sep 18 19:56:55.309221 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303193    2549 flags.go:64] FLAG: --reserved-cpus=""
Sep 18 19:56:55.309221 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303196    2549 flags.go:64] FLAG: --reserved-memory=""
Sep 18 19:56:55.309221 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303199    2549 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 18 19:56:55.309221 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303202    2549 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 18 19:56:55.309221 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303205    2549 flags.go:64] FLAG: --rotate-certificates="false"
Sep 18 19:56:55.309221 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303208    2549 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 18 19:56:55.309221 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303211    2549 flags.go:64] FLAG: --runonce="false"
Sep 18 19:56:55.309221 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303214    2549 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 18 19:56:55.309845 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303217    2549 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 18 19:56:55.309845 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303220    2549 flags.go:64] FLAG: --seccomp-default="false"
Sep 18 19:56:55.309845 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303223    2549 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 18 19:56:55.309845 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303226    2549 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 18 19:56:55.309845 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303229    2549 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 18 19:56:55.309845 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303232    2549 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 18 19:56:55.309845 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303236    2549 flags.go:64] FLAG: --storage-driver-password="root"
Sep 18 19:56:55.309845 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303238    2549 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 18 19:56:55.309845 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303241    2549 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 18 19:56:55.309845 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303244    2549 flags.go:64] FLAG: --storage-driver-user="root"
Sep 18 19:56:55.309845 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303247    2549 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 18 19:56:55.309845 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303250    2549 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 18 19:56:55.309845 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303253    2549 flags.go:64] FLAG: --system-cgroups=""
Sep 18 19:56:55.309845 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303256    2549 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 18 19:56:55.309845 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303261    2549 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 18 19:56:55.309845 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303265    2549 flags.go:64] FLAG: --tls-cert-file=""
Sep 18 19:56:55.309845 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303268    2549 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 18 19:56:55.309845 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303272    2549 flags.go:64] FLAG: --tls-min-version=""
Sep 18 19:56:55.309845 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303275    2549 flags.go:64] FLAG: --tls-private-key-file=""
Sep 18 19:56:55.309845 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303279    2549 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 18 19:56:55.309845 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303282    2549 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 18 19:56:55.309845 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303286    2549 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 18 19:56:55.309845 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303289    2549 flags.go:64] FLAG: --v="2"
Sep 18 19:56:55.309845 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303292    2549 flags.go:64] FLAG: --version="false"
Sep 18 19:56:55.309845 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303300    2549 flags.go:64] FLAG: --vmodule=""
Sep 18 19:56:55.310506 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303304    2549 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 18 19:56:55.310506 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303307    2549 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 18 19:56:55.310506 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303449    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:56:55.310506 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303453    2549 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:56:55.310506 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303456    2549 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:56:55.310506 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303459    2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:56:55.310506 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303462    2549 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:56:55.310506 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303465    2549 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:56:55.310506 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303468    2549 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:56:55.310506 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303471    2549 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:56:55.310506 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303473    2549 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:56:55.310506 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303476    2549 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:56:55.310506 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303479    2549 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:56:55.310506 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303481    2549 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:56:55.310506 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303485    2549 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:56:55.310506 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303487    2549 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:56:55.310506 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303491    2549 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:56:55.310506 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303494    2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:56:55.310506 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303497    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:56:55.310966 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303500    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:56:55.310966 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303503    2549 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:56:55.310966 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303506    2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:56:55.310966 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303510    2549 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:56:55.310966 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303513    2549 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:56:55.310966 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303516    2549 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:56:55.310966 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303518    2549 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:56:55.310966 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303521    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:56:55.310966 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303524    2549 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:56:55.310966 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303527    2549 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:56:55.310966 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303529    2549 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:56:55.310966 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303532    2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:56:55.310966 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303535    2549 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:56:55.310966 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303538    2549 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:56:55.310966 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303540    2549 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:56:55.310966 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303543    2549 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:56:55.310966 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303546    2549 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:56:55.310966 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303548    2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:56:55.310966 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303551    2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:56:55.311484 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303554    2549 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:56:55.311484 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303556    2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:56:55.311484 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303559    2549 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:56:55.311484 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303562    2549 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:56:55.311484 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303564    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:56:55.311484 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303567    2549 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:56:55.311484 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303570    2549 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:56:55.311484 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303573    2549 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:56:55.311484 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303576    2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:56:55.311484 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303578    2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:56:55.311484 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303581    2549 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:56:55.311484 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303583    2549 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:56:55.311484 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303586    2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:56:55.311484 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303588    2549 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:56:55.311484 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303591    2549 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:56:55.311484 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303593    2549 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:56:55.311484 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303599    2549 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:56:55.311484 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303602    2549 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:56:55.311484 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303604    2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:56:55.311484 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303607    2549 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:56:55.311983 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303609    2549 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:56:55.311983 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303612    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:56:55.311983 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303615    2549 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:56:55.311983 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303618    2549 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:56:55.311983 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303620    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:56:55.311983 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303623    2549 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:56:55.311983 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303625    2549 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:56:55.311983 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303628    2549 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:56:55.311983 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303631    2549 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:56:55.311983 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303633    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:56:55.311983 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303636    2549 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:56:55.311983 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303639    2549 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:56:55.311983 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303641    2549 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:56:55.311983 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303644    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:56:55.311983 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303647    2549 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:56:55.311983 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303649    2549 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:56:55.311983 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303652    2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:56:55.311983 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303654    2549 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:56:55.311983 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303657    2549 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:56:55.311983 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303660    2549 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:56:55.312603 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303663    2549 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:56:55.312603 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303665    2549 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:56:55.312603 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303668    2549 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:56:55.312603 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303670    2549 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:56:55.312603 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303673    2549 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:56:55.312603 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303676    2549 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:56:55.312603 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303678    2549 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:56:55.312603 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303681    2549 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:56:55.312603 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303685    2549 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:56:55.312603 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303687    2549 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:56:55.312603 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303690    2549 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:56:55.312603 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303693    2549 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:56:55.312603 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303695    2549 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:56:55.312603 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303698    2549 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:56:55.312603 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303701    2549 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:56:55.312603 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303704    2549 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:56:55.312603 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303706    2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:56:55.312603 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303711    2549 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:56:55.312603 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303714    2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:56:55.312603 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303717    2549 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:56:55.313104 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303721    2549 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:56:55.313104 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303724    2549 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:56:55.313104 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.303728    2549 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:56:55.313104 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.303733    2549 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 19:56:55.313104 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.311731    2549 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 18 19:56:55.313104 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.311750    2549 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 18 19:56:55.313104 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.312435    2549 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 18 19:56:55.313104 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.312443    2549 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 18 19:56:55.313104 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312501    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:56:55.313104 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312506    2549 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:56:55.313104 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312510    2549 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:56:55.313104 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312513    2549 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:56:55.313104 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312517    2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:56:55.313104 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312521    2549 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:56:55.313465 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312525    2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:56:55.313465 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312529    2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:56:55.313465 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312532    2549 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:56:55.313465 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312535    2549 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:56:55.313465 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312539    2549 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:56:55.313465 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312543    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:56:55.313465 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312546    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:56:55.313465 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312549    2549 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:56:55.313465 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312553    2549 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:56:55.313465 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312555    2549 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:56:55.313465 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312558    2549 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:56:55.313465 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312561    2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:56:55.313465 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312564    2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:56:55.313465 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312566    2549 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:56:55.313465 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312569    2549 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:56:55.313465 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312572    2549 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:56:55.313465 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312575    2549 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:56:55.313465 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312577    2549 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:56:55.313465 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312580    2549 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:56:55.313465 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312583    2549 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:56:55.313959 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312585    2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:56:55.313959 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312588    2549 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:56:55.313959 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312591    2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:56:55.313959 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312595    2549 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:56:55.313959 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312598    2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:56:55.313959 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312602    2549 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:56:55.313959 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312605    2549 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:56:55.313959 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312608    2549 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:56:55.313959 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312611    2549 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:56:55.313959 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312613    2549 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:56:55.313959 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312616    2549 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:56:55.313959 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312619    2549 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:56:55.313959 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312622    2549 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:56:55.313959 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312626    2549 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:56:55.313959 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312628    2549 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:56:55.313959 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312631    2549 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:56:55.313959 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312633    2549 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:56:55.313959 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312636    2549 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:56:55.313959 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312639    2549 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:56:55.313959 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312642    2549 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:56:55.314463 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312645    2549 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:56:55.314463 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312648    2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:56:55.314463 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312650    2549 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:56:55.314463 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312653    2549 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:56:55.314463 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312656    2549 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:56:55.314463 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312658    2549 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:56:55.314463 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312661    2549 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:56:55.314463 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312664    2549 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:56:55.314463 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312667    2549 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:56:55.314463 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312671    2549 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:56:55.314463 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312674    2549 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:56:55.314463 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312676    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:56:55.314463 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312679    2549 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:56:55.314463 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312682    2549 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:56:55.314463 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312684    2549 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:56:55.314463 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312687    2549 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:56:55.314463 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312690    2549 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:56:55.314463 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312693    2549 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:56:55.314463 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312696    2549 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:56:55.314953 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312699    2549 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:56:55.314953 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312701    2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:56:55.314953 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312704    2549 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:56:55.314953 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312706    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:56:55.314953 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312709    2549 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:56:55.314953 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312712    2549 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:56:55.314953 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312714    2549 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:56:55.314953 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312717    2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:56:55.314953 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312719    2549 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:56:55.314953 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312722    2549 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:56:55.314953 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312725    2549 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:56:55.314953 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312727    2549 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:56:55.314953 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312730    2549 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:56:55.314953 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312732    2549 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:56:55.314953 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312735    2549 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:56:55.314953 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312738    2549 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:56:55.314953 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312740    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:56:55.314953 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312743    2549 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:56:55.314953 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312745    2549 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:56:55.314953 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312748    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:56:55.315453 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312751    2549 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:56:55.315453 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312753    2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:56:55.315453 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312756    2549 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:56:55.315453 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312758    2549 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:56:55.315453 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312761    2549 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:56:55.315453 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312763    2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:56:55.315453 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312766    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:56:55.315453 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312769    2549 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:56:55.315453 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312772    2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:56:55.315453 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312775    2549 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:56:55.315453 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312778    2549 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:56:55.315453 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312780    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:56:55.315453 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312783    2549 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:56:55.315453 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312786    2549 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:56:55.315453 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.312791    2549 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 19:56:55.315859 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312903    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:56:55.315859 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312908    2549 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:56:55.315859 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312911    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:56:55.315859 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312914    2549 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:56:55.315859 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312917    2549 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:56:55.315859 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312920    2549 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:56:55.315859 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312923    2549 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:56:55.315859 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312926    2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:56:55.315859 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312929    2549 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:56:55.315859 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312932    2549 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:56:55.315859 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312935    2549 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:56:55.315859 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312937    2549 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:56:55.315859 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312940    2549 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:56:55.315859 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312943    2549 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:56:55.315859 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312946    2549 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:56:55.315859 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312949    2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:56:55.315859 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312952    2549 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:56:55.315859 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312954    2549 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:56:55.315859 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312957    2549 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:56:55.316333 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312959    2549 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:56:55.316333 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312962    2549 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:56:55.316333 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312965    2549 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:56:55.316333 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312967    2549 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:56:55.316333 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312971    2549 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:56:55.316333 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312974    2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:56:55.316333 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312978    2549 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:56:55.316333 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312981    2549 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:56:55.316333 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312984    2549 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:56:55.316333 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312987    2549 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:56:55.316333 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312990    2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:56:55.316333 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312993    2549 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:56:55.316333 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312995    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:56:55.316333 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.312998    2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:56:55.316333 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.313000    2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:56:55.316333 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.313003    2549 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:56:55.316333 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.313005    2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:56:55.316333 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.313008    2549 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:56:55.316333 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.313012    2549 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:56:55.317099 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.313015    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:56:55.317099 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.313018    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:56:55.317099 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.313020    2549 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:56:55.317099 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.313023    2549 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:56:55.317099 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.313025    2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:56:55.317099 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.313028    2549 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:56:55.317099 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.313030    2549 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:56:55.317099 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.313033    2549 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:56:55.317099 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.313035    2549 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:56:55.317099 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.313038    2549 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:56:55.317099 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.313040    2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:56:55.317099 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.313043    2549 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:56:55.317099 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.313046    2549 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:56:55.317099 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.313048    2549 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:56:55.317099 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.313051    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:56:55.317099 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.313053    2549 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:56:55.317099 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.313056    2549 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:56:55.317099 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.313058    2549 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:56:55.317099 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.313061    2549 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:56:55.317099 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.313064    2549 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:56:55.317777 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.313067    2549 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:56:55.317777 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.313070    2549 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:56:55.317777 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.313072    2549 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:56:55.317777 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.313075    2549 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:56:55.317777 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.313078    2549 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:56:55.317777 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.313080    2549 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:56:55.317777 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.313083    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:56:55.317777 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.313085    2549 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:56:55.317777 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.313088    2549 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:56:55.317777 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.313091    2549 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:56:55.317777 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.313093    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:56:55.317777 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.313096    2549 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:56:55.317777 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.313098    2549 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:56:55.317777 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.313101    2549 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:56:55.317777 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.313104    2549 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:56:55.317777 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.313106    2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:56:55.317777 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.313109    2549 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:56:55.317777 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.313112    2549 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:56:55.317777 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.313114    2549 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:56:55.317777 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.313117    2549 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:56:55.318252 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.313120    2549 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:56:55.318252 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.313122    2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:56:55.318252 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.313125    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:56:55.318252 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.313127    2549 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:56:55.318252 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.313131    2549 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:56:55.318252 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.313135    2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:56:55.318252 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.313138    2549 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:56:55.318252 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.313141    2549 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:56:55.318252 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.313143    2549 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:56:55.318252 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.313146    2549 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:56:55.318252 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.313148    2549 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:56:55.318252 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.313152    2549 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:56:55.318252 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.313154    2549 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:56:55.318252 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.313157    2549 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:56:55.318252 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.313159    2549 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:56:55.318252 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.313162    2549 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:56:55.318252 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.313165    2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:56:55.318252 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.313168    2549 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:56:55.318252 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.313170    2549 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:56:55.318820 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.313173    2549 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:56:55.318820 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:55.313175    2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:56:55.318820 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.313180    2549 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 19:56:55.318820 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.313289    2549 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 18 19:56:55.318820 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.315485    2549 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 18 19:56:55.318820 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.316296    2549 server.go:1019] "Starting client certificate rotation"
Sep 18 19:56:55.318820 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.316420    2549 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 19:56:55.318820 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.316469    2549 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 19:56:55.318820 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.318743    2549 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 19:56:55.322180 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.322162    2549 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 19:56:55.333150 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.333135    2549 log.go:25] "Validated CRI v1 runtime API"
Sep 18 19:56:55.336621 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.336607    2549 log.go:25] "Validated CRI v1 image API"
Sep 18 19:56:55.337593 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.337575    2549 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 18 19:56:55.340755 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.340736    2549 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 80bda45b-6524-4933-8678-a5a845d4500d:/dev/nvme0n1p4 be0d0a5e-6be5-4f1c-810d-1127b0164727:/dev/nvme0n1p3]
Sep 18 19:56:55.340823 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.340754    2549 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 18 19:56:55.346940 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.346826    2549 manager.go:216] Machine: {Timestamp:2026-09-18 19:56:55.345662044 +0000 UTC m=+0.299403203 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098373 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c4521241e2c057aa5a77e5279cec0 SystemUUID:ec2c4521-241e-2c05-7aa5-a77e5279cec0 BootID:aeedd41f-6038-472e-9792-ddcebf99a2fd Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e5:42:35:2f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e5:42:35:2f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:3a:ae:24:a2:70:69 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 18 19:56:55.346940 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.346934    2549 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 18 19:56:55.347057 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.347011    2549 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 19:56:55.347057 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.347014    2549 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 18 19:56:55.347993 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.347968    2549 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 18 19:56:55.348123 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.347995    2549 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-178.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 18 19:56:55.348720 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.348710    2549 topology_manager.go:138] "Creating topology manager with none policy"
Sep 18 19:56:55.348750 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.348723    2549 container_manager_linux.go:309] "Creating device plugin manager"
Sep 18 19:56:55.348750 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.348731    2549 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 19:56:55.349180 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.349171    2549 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 19:56:55.349259 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.349251    2549 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 18 19:56:55.352549 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.352538    2549 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 19:56:55.352657 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.352648    2549 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 18 19:56:55.352689 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.352675    2549 kubelet.go:486] "Attempting to sync node with API server"
Sep 18 19:56:55.352689 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.352685    2549 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 18 19:56:55.352742 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.352696    2549 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 18 19:56:55.352742 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.352705    2549 kubelet.go:398] "Adding apiserver pod source"
Sep 18 19:56:55.352742 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.352714    2549 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 18 19:56:55.353044 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.353033    2549 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 19:56:55.353089 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.353052    2549 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 19:56:55.355100 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.355081    2549 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 18 19:56:55.358065 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.358052    2549 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 18 19:56:55.358113 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.358082    2549 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 18 19:56:55.359441 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.359425    2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 18 19:56:55.359441 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.359445    2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 18 19:56:55.359545 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.359451    2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 18 19:56:55.359545 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.359456    2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 18 19:56:55.359545 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.359471    2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 18 19:56:55.359545 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.359477    2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 18 19:56:55.359545 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.359490    2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 18 19:56:55.359545 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.359495    2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 18 19:56:55.359545 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.359501    2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 18 19:56:55.359545 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.359506    2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 18 19:56:55.359545 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.359518    2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 18 19:56:55.359545 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.359523    2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 18 19:56:55.361314 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.361303    2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 18 19:56:55.361314 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.361313    2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 18 19:56:55.363261 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:56:55.363227    2549 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 18 19:56:55.363317 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:56:55.363241    2549 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-178.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 18 19:56:55.369189 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.369174    2549 server.go:1268] "Started kubelet"
Sep 18 19:56:55.369268 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.369239    2549 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 18 19:56:55.369528 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.369400    2549 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 18 19:56:55.369633 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.369619    2549 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 18 19:56:55.370108 ip-10-0-140-178 systemd[1]: Started Kubernetes Kubelet.
Sep 18 19:56:55.371109 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.371083    2549 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 18 19:56:55.373109 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.373091    2549 server.go:310] "Adding debug handlers to kubelet server"
Sep 18 19:56:55.375545 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.375520    2549 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-178.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 19:56:55.376176 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:56:55.375188    2549 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-178.ec2.internal.18d6828653c70aae  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-178.ec2.internal,UID:ip-10-0-140-178.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-178.ec2.internal,},FirstTimestamp:2026-09-18 19:56:55.369132718 +0000 UTC m=+0.322873878,LastTimestamp:2026-09-18 19:56:55.369132718 +0000 UTC m=+0.322873878,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-178.ec2.internal,}"
Sep 18 19:56:55.377055 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.377037    2549 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wc6p7"
Sep 18 19:56:55.378575 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:56:55.378559    2549 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 18 19:56:55.380549 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.380532    2549 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 18 19:56:55.380549 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.380548    2549 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 18 19:56:55.381210 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:56:55.380817    2549 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-178.ec2.internal\" not found"
Sep 18 19:56:55.381350 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.381327    2549 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 18 19:56:55.381445 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.381339    2549 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 18 19:56:55.382301 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.382261    2549 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 18 19:56:55.383263 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.383243    2549 reconstruct.go:97] "Volume reconstruction finished"
Sep 18 19:56:55.383263 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.383260    2549 reconciler.go:29] "Reconciler: start to sync state"
Sep 18 19:56:55.383618 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.383000    2549 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 18 19:56:55.383712 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.383688    2549 factory.go:55] Registering systemd factory
Sep 18 19:56:55.383712 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.383707    2549 factory.go:223] Registration of the systemd container factory successfully
Sep 18 19:56:55.384101 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.384080    2549 factory.go:153] Registering CRI-O factory
Sep 18 19:56:55.384179 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.384104    2549 factory.go:223] Registration of the crio container factory successfully
Sep 18 19:56:55.384179 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.384122    2549 factory.go:103] Registering Raw factory
Sep 18 19:56:55.384179 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.384136    2549 manager.go:1196] Started watching for new ooms in manager
Sep 18 19:56:55.384993 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.384973    2549 manager.go:349] Starting recovery of all containers
Sep 18 19:56:55.385127 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:56:55.385085    2549 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 18 19:56:55.385944 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:56:55.385215    2549 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-178.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 18 19:56:55.386073 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.386051    2549 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wc6p7"
Sep 18 19:56:55.389493 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.389446    2549 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 18 19:56:55.398599 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.398445    2549 manager.go:354] Recovery completed
Sep 18 19:56:55.402992 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.402979    2549 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:56:55.404724 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.404708    2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-178.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:56:55.404810 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.404739    2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-178.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:56:55.404810 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.404761    2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-178.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:56:55.405342 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.405326    2549 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 18 19:56:55.405342 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.405339    2549 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 18 19:56:55.405474 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.405353    2549 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 19:56:55.405474 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.405401    2549 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:56:55.406165 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.406145    2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-178.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:56:55.406244 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.406172    2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-178.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:56:55.406244 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.406182    2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-178.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:56:55.406487 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.406477    2549 policy_none.go:49] "None policy: Start"
Sep 18 19:56:55.406523 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.406491    2549 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 18 19:56:55.406523 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.406501    2549 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 18 19:56:55.406953 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:56:55.406880    2549 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-178.ec2.internal.18d6828655e61c50  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-178.ec2.internal,UID:ip-10-0-140-178.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-178.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-178.ec2.internal,},FirstTimestamp:2026-09-18 19:56:55.40472328 +0000 UTC m=+0.358464439,LastTimestamp:2026-09-18 19:56:55.40472328 +0000 UTC m=+0.358464439,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-178.ec2.internal,}"
Sep 18 19:56:55.407736 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.407725    2549 policy_none.go:47] "Start"
Sep 18 19:56:55.441014 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.440997    2549 manager.go:337] "Starting Device Plugin manager"
Sep 18 19:56:55.451608 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:56:55.441050    2549 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 18 19:56:55.451608 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.441065    2549 server.go:87] "Starting device plugin registration server"
Sep 18 19:56:55.451608 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.441495    2549 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 18 19:56:55.451608 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.441509    2549 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 18 19:56:55.451608 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.441627    2549 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 18 19:56:55.451608 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.441705    2549 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 18 19:56:55.451608 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.441717    2549 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 18 19:56:55.451608 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:56:55.442289    2549 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 18 19:56:55.451608 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:56:55.442326    2549 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-178.ec2.internal\" not found"
Sep 18 19:56:55.514370 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.514341    2549 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 18 19:56:55.514466 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.514411    2549 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 18 19:56:55.514466 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.514438    2549 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 18 19:56:55.514562 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:56:55.514527    2549 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 18 19:56:55.516925 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.516901    2549 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:56:55.542204 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.542176    2549 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:56:55.543061 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.543042    2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-178.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:56:55.543156 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.543075    2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-178.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:56:55.543156 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.543088    2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-178.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:56:55.543156 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.543109    2549 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-178.ec2.internal"
Sep 18 19:56:55.555582 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.555512    2549 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-178.ec2.internal"
Sep 18 19:56:55.555582 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:56:55.555537    2549 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-178.ec2.internal\": node \"ip-10-0-140-178.ec2.internal\" not found"
Sep 18 19:56:55.580060 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:56:55.580034    2549 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-178.ec2.internal\" not found"
Sep 18 19:56:55.615923 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.615030    2549 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-178.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-178.ec2.internal"]
Sep 18 19:56:55.615992 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.615966    2549 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:56:55.617589 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.617575    2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-178.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:56:55.617644 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.617607    2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-178.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:56:55.617644 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.617621    2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-178.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:56:55.619088 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.619076    2549 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:56:55.619226 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.619211    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-178.ec2.internal"
Sep 18 19:56:55.619266 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.619242    2549 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:56:55.619878 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.619861    2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-178.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:56:55.619964 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.619890    2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-178.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:56:55.619964 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.619907    2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-178.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:56:55.619964 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.619891    2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-178.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:56:55.619964 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.619956    2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-178.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:56:55.620148 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.619974    2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-178.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:56:55.621351 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.621336    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-178.ec2.internal"
Sep 18 19:56:55.621429 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.621362    2549 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:56:55.622094 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.622079    2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-178.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:56:55.622173 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.622105    2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-178.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:56:55.622173 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.622116    2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-178.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:56:55.646996 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:56:55.646974    2549 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-178.ec2.internal\" not found" node="ip-10-0-140-178.ec2.internal"
Sep 18 19:56:55.651476 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:56:55.651462    2549 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-178.ec2.internal\" not found" node="ip-10-0-140-178.ec2.internal"
Sep 18 19:56:55.680652 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:56:55.680628    2549 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-178.ec2.internal\" not found"
Sep 18 19:56:55.684916 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.684900    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e511ba71d033b03f5d71c14df866e732-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-178.ec2.internal\" (UID: \"e511ba71d033b03f5d71c14df866e732\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-178.ec2.internal"
Sep 18 19:56:55.684984 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.684924    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e511ba71d033b03f5d71c14df866e732-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-178.ec2.internal\" (UID: \"e511ba71d033b03f5d71c14df866e732\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-178.ec2.internal"
Sep 18 19:56:55.684984 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.684941    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2748dceba5a1bbaba05c6bb022d69842-config\") pod \"kube-apiserver-proxy-ip-10-0-140-178.ec2.internal\" (UID: \"2748dceba5a1bbaba05c6bb022d69842\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-178.ec2.internal"
Sep 18 19:56:55.781109 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:56:55.781080    2549 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-178.ec2.internal\" not found"
Sep 18 19:56:55.785570 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.785553    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2748dceba5a1bbaba05c6bb022d69842-config\") pod \"kube-apiserver-proxy-ip-10-0-140-178.ec2.internal\" (UID: \"2748dceba5a1bbaba05c6bb022d69842\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-178.ec2.internal"
Sep 18 19:56:55.785677 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.785581    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e511ba71d033b03f5d71c14df866e732-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-178.ec2.internal\" (UID: \"e511ba71d033b03f5d71c14df866e732\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-178.ec2.internal"
Sep 18 19:56:55.785677 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.785604    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e511ba71d033b03f5d71c14df866e732-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-178.ec2.internal\" (UID: \"e511ba71d033b03f5d71c14df866e732\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-178.ec2.internal"
Sep 18 19:56:55.785677 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.785657    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e511ba71d033b03f5d71c14df866e732-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-178.ec2.internal\" (UID: \"e511ba71d033b03f5d71c14df866e732\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-178.ec2.internal"
Sep 18 19:56:55.785791 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.785658    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2748dceba5a1bbaba05c6bb022d69842-config\") pod \"kube-apiserver-proxy-ip-10-0-140-178.ec2.internal\" (UID: \"2748dceba5a1bbaba05c6bb022d69842\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-178.ec2.internal"
Sep 18 19:56:55.785791 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.785719    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e511ba71d033b03f5d71c14df866e732-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-178.ec2.internal\" (UID: \"e511ba71d033b03f5d71c14df866e732\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-178.ec2.internal"
Sep 18 19:56:55.882117 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:56:55.882082    2549 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-178.ec2.internal\" not found"
Sep 18 19:56:55.951439 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.951409    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-178.ec2.internal"
Sep 18 19:56:55.956244 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:55.956217    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-178.ec2.internal"
Sep 18 19:56:55.982259 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:56:55.982236    2549 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-178.ec2.internal\" not found"
Sep 18 19:56:56.082717 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:56:56.082677    2549 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-178.ec2.internal\" not found"
Sep 18 19:56:56.183257 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:56:56.183185    2549 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-178.ec2.internal\" not found"
Sep 18 19:56:56.283784 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:56:56.283755    2549 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-178.ec2.internal\" not found"
Sep 18 19:56:56.292987 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.292958    2549 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:56:56.316914 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.316885    2549 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 18 19:56:56.317390 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.317075    2549 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 18 19:56:56.317390 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.317112    2549 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 18 19:56:56.353555 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.353525    2549 apiserver.go:52] "Watching apiserver"
Sep 18 19:56:56.375840 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.375813    2549 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 18 19:56:56.378427 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.378406    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-tlqj8","openshift-cluster-node-tuning-operator/tuned-ks678","openshift-dns/node-resolver-t26hd","openshift-multus/multus-additional-cni-plugins-8nv28","openshift-multus/network-metrics-daemon-2hw77","openshift-network-diagnostics/network-check-target-rktz8","openshift-ovn-kubernetes/ovnkube-node-scn7p","kube-system/global-pull-secret-syncer-zq878","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfp4z","openshift-image-registry/node-ca-m6qj6","openshift-multus/multus-gk46f","openshift-network-operator/iptables-alerter-5bnj6"]
Sep 18 19:56:56.379976 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.379954    2549 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:56:56.380961 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.380944    2549 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-178.ec2.internal"
Sep 18 19:56:56.381039 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.380981    2549 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 18 19:56:56.381150 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.381129    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-tlqj8"
Sep 18 19:56:56.383351 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.383331    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ks678"
Sep 18 19:56:56.384407 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.384386    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 18 19:56:56.384525 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.384415    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 18 19:56:56.384722 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.384706    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-w29br\""
Sep 18 19:56:56.385752 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.385731    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-t26hd"
Sep 18 19:56:56.386899 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.386880    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ml6zn\""
Sep 18 19:56:56.386986 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.386903    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 18 19:56:56.387155 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.387137    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 18 19:56:56.388093 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.388072    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bfd97804-116b-4355-bf8f-adb95dc143af-run\") pod \"tuned-ks678\" (UID: \"bfd97804-116b-4355-bf8f-adb95dc143af\") " pod="openshift-cluster-node-tuning-operator/tuned-ks678"
Sep 18 19:56:56.388162 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.388093    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8nv28"
Sep 18 19:56:56.388162 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.388107    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bfd97804-116b-4355-bf8f-adb95dc143af-var-lib-kubelet\") pod \"tuned-ks678\" (UID: \"bfd97804-116b-4355-bf8f-adb95dc143af\") " pod="openshift-cluster-node-tuning-operator/tuned-ks678"
Sep 18 19:56:56.388162 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.388138    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bfd97804-116b-4355-bf8f-adb95dc143af-host\") pod \"tuned-ks678\" (UID: \"bfd97804-116b-4355-bf8f-adb95dc143af\") " pod="openshift-cluster-node-tuning-operator/tuned-ks678"
Sep 18 19:56:56.388277 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.388164    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bfd97804-116b-4355-bf8f-adb95dc143af-etc-tuned\") pod \"tuned-ks678\" (UID: \"bfd97804-116b-4355-bf8f-adb95dc143af\") " pod="openshift-cluster-node-tuning-operator/tuned-ks678"
Sep 18 19:56:56.388277 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.388211    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bfd97804-116b-4355-bf8f-adb95dc143af-etc-systemd\") pod \"tuned-ks678\" (UID: \"bfd97804-116b-4355-bf8f-adb95dc143af\") " pod="openshift-cluster-node-tuning-operator/tuned-ks678"
Sep 18 19:56:56.388277 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.388258    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bfd97804-116b-4355-bf8f-adb95dc143af-sys\") pod \"tuned-ks678\" (UID: \"bfd97804-116b-4355-bf8f-adb95dc143af\") " pod="openshift-cluster-node-tuning-operator/tuned-ks678"
Sep 18 19:56:56.388420 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.388283    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bfd97804-116b-4355-bf8f-adb95dc143af-tmp\") pod \"tuned-ks678\" (UID: \"bfd97804-116b-4355-bf8f-adb95dc143af\") " pod="openshift-cluster-node-tuning-operator/tuned-ks678"
Sep 18 19:56:56.388420 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.388322    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2e178884-5bfb-46cb-9305-6170284cb243-konnectivity-ca\") pod \"konnectivity-agent-tlqj8\" (UID: \"2e178884-5bfb-46cb-9305-6170284cb243\") " pod="kube-system/konnectivity-agent-tlqj8"
Sep 18 19:56:56.388420 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.388347    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bfd97804-116b-4355-bf8f-adb95dc143af-etc-modprobe-d\") pod \"tuned-ks678\" (UID: \"bfd97804-116b-4355-bf8f-adb95dc143af\") " pod="openshift-cluster-node-tuning-operator/tuned-ks678"
Sep 18 19:56:56.388420 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.388373    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bfd97804-116b-4355-bf8f-adb95dc143af-etc-kubernetes\") pod \"tuned-ks678\" (UID: \"bfd97804-116b-4355-bf8f-adb95dc143af\") " pod="openshift-cluster-node-tuning-operator/tuned-ks678"
Sep 18 19:56:56.388420 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.388396    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bfd97804-116b-4355-bf8f-adb95dc143af-etc-sysctl-d\") pod \"tuned-ks678\" (UID: \"bfd97804-116b-4355-bf8f-adb95dc143af\") " pod="openshift-cluster-node-tuning-operator/tuned-ks678"
Sep 18 19:56:56.388800 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.388494    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bfd97804-116b-4355-bf8f-adb95dc143af-etc-sysctl-conf\") pod \"tuned-ks678\" (UID: \"bfd97804-116b-4355-bf8f-adb95dc143af\") " pod="openshift-cluster-node-tuning-operator/tuned-ks678"
Sep 18 19:56:56.388800 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.388531    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bfd97804-116b-4355-bf8f-adb95dc143af-lib-modules\") pod \"tuned-ks678\" (UID: \"bfd97804-116b-4355-bf8f-adb95dc143af\") " pod="openshift-cluster-node-tuning-operator/tuned-ks678"
Sep 18 19:56:56.388800 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.388550    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q458c\" (UniqueName: \"kubernetes.io/projected/bfd97804-116b-4355-bf8f-adb95dc143af-kube-api-access-q458c\") pod \"tuned-ks678\" (UID: \"bfd97804-116b-4355-bf8f-adb95dc143af\") " pod="openshift-cluster-node-tuning-operator/tuned-ks678"
Sep 18 19:56:56.388800 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.388615    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 18 19:56:56.388800 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.388615    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2e178884-5bfb-46cb-9305-6170284cb243-agent-certs\") pod \"konnectivity-agent-tlqj8\" (UID: \"2e178884-5bfb-46cb-9305-6170284cb243\") " pod="kube-system/konnectivity-agent-tlqj8"
Sep 18 19:56:56.388800 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.388687    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bfd97804-116b-4355-bf8f-adb95dc143af-etc-sysconfig\") pod \"tuned-ks678\" (UID: \"bfd97804-116b-4355-bf8f-adb95dc143af\") " pod="openshift-cluster-node-tuning-operator/tuned-ks678"
Sep 18 19:56:56.389100 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.388971    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-jb267\""
Sep 18 19:56:56.389175 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.389141    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 18 19:56:56.390856 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.390739    2549 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-17 19:51:56 +0000 UTC" deadline="2028-02-22 10:33:00.638390554 +0000 UTC"
Sep 18 19:56:56.390856 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.390797    2549 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12518h36m4.24759711s"
Sep 18 19:56:56.391025 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.390890    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 18 19:56:56.391025 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.390913    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6zxtn\""
Sep 18 19:56:56.391167 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.391087    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 18 19:56:56.391231 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.391193    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 18 19:56:56.391231 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.391213    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 18 19:56:56.392049 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.392030    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2hw77"
Sep 18 19:56:56.392141 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:56:56.392112    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2hw77" podUID="063d8ee5-35b4-401b-a0ff-d8b416e20175"
Sep 18 19:56:56.393571 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.393552    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 18 19:56:56.394309 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.394291    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rktz8"
Sep 18 19:56:56.394398 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:56:56.394359    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rktz8" podUID="ed609df8-4cad-445d-99a8-2df85ce75ba8"
Sep 18 19:56:56.396058 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.396036    2549 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 19:56:56.396141 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.396125    2549 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-178.ec2.internal"
Sep 18 19:56:56.396902 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.396886    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-scn7p"
Sep 18 19:56:56.399118 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.399099    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zq878"
Sep 18 19:56:56.399212 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:56:56.399179    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zq878" podUID="254868d4-6e40-410f-9a3d-0f994b704438"
Sep 18 19:56:56.399315 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.399298    2549 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 19:56:56.399753 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.399732    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 18 19:56:56.399877 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.399852    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 18 19:56:56.399999 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.399984    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 18 19:56:56.400104 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.400088    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 18 19:56:56.400171 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.400120    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 18 19:56:56.400171 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.400126    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 18 19:56:56.400171 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.400130    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-xk7s5\""
Sep 18 19:56:56.402009 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.401993    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfp4z"
Sep 18 19:56:56.404759 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.404742    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 18 19:56:56.404860 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.404784    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-gdtkw\""
Sep 18 19:56:56.404929 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.404914    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 18 19:56:56.405004 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.404996    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-m6qj6"
Sep 18 19:56:56.405072 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.405003    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 18 19:56:56.407365 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.407345    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gk46f"
Sep 18 19:56:56.407889 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.407872    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 18 19:56:56.407965 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.407947    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 18 19:56:56.408179 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.408163    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 18 19:56:56.408228 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.408204    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-tdcqh\""
Sep 18 19:56:56.408944 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.408931    2549 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 19:56:56.410116 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.410099    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 18 19:56:56.410183 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.410143    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-lfcjv\""
Sep 18 19:56:56.410636 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.410620    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-178.ec2.internal"]
Sep 18 19:56:56.410680 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.410652    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-140-178.ec2.internal"]
Sep 18 19:56:56.410714 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.410708    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5bnj6"
Sep 18 19:56:56.413488 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.413467    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 18 19:56:56.413593 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.413517    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 18 19:56:56.413754 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.413732    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 18 19:56:56.413861 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.413844    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-sb9rw\""
Sep 18 19:56:56.448785 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.448766    2549 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-9v9b8"
Sep 18 19:56:56.457787 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.457769    2549 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-9v9b8"
Sep 18 19:56:56.485426 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.483426    2549 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 18 19:56:56.486681 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.486659    2549 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:56:56.489302 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.489281    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bfd97804-116b-4355-bf8f-adb95dc143af-etc-sysconfig\") pod \"tuned-ks678\" (UID: \"bfd97804-116b-4355-bf8f-adb95dc143af\") " pod="openshift-cluster-node-tuning-operator/tuned-ks678"
Sep 18 19:56:56.489402 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.489313    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bfd97804-116b-4355-bf8f-adb95dc143af-var-lib-kubelet\") pod \"tuned-ks678\" (UID: \"bfd97804-116b-4355-bf8f-adb95dc143af\") " pod="openshift-cluster-node-tuning-operator/tuned-ks678"
Sep 18 19:56:56.489402 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.489337    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bfd97804-116b-4355-bf8f-adb95dc143af-host\") pod \"tuned-ks678\" (UID: \"bfd97804-116b-4355-bf8f-adb95dc143af\") " pod="openshift-cluster-node-tuning-operator/tuned-ks678"
Sep 18 19:56:56.489402 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.489360    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bfd97804-116b-4355-bf8f-adb95dc143af-etc-tuned\") pod \"tuned-ks678\" (UID: \"bfd97804-116b-4355-bf8f-adb95dc143af\") " pod="openshift-cluster-node-tuning-operator/tuned-ks678"
Sep 18 19:56:56.489402 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.489390    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/063d8ee5-35b4-401b-a0ff-d8b416e20175-metrics-certs\") pod \"network-metrics-daemon-2hw77\" (UID: \"063d8ee5-35b4-401b-a0ff-d8b416e20175\") " pod="openshift-multus/network-metrics-daemon-2hw77"
Sep 18 19:56:56.489645 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.489415    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/525400e6-6678-424f-a1ba-0896aac41264-hosts-file\") pod \"node-resolver-t26hd\" (UID: \"525400e6-6678-424f-a1ba-0896aac41264\") " pod="openshift-dns/node-resolver-t26hd"
Sep 18 19:56:56.489645 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.489442    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f4120871-0e94-46f5-bd6d-739f9b0383bf-cnibin\") pod \"multus-gk46f\" (UID: \"f4120871-0e94-46f5-bd6d-739f9b0383bf\") " pod="openshift-multus/multus-gk46f"
Sep 18 19:56:56.489645 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.489441    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bfd97804-116b-4355-bf8f-adb95dc143af-etc-sysconfig\") pod \"tuned-ks678\" (UID: \"bfd97804-116b-4355-bf8f-adb95dc143af\") " pod="openshift-cluster-node-tuning-operator/tuned-ks678"
Sep 18 19:56:56.489645 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.489441    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bfd97804-116b-4355-bf8f-adb95dc143af-var-lib-kubelet\") pod \"tuned-ks678\" (UID: \"bfd97804-116b-4355-bf8f-adb95dc143af\") " pod="openshift-cluster-node-tuning-operator/tuned-ks678"
Sep 18 19:56:56.489645 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.489503    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/56b6948b-89ff-4d48-ba8c-c0c3f63544e1-cnibin\") pod \"multus-additional-cni-plugins-8nv28\" (UID: \"56b6948b-89ff-4d48-ba8c-c0c3f63544e1\") " pod="openshift-multus/multus-additional-cni-plugins-8nv28"
Sep 18 19:56:56.489645 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.489541    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/56b6948b-89ff-4d48-ba8c-c0c3f63544e1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8nv28\" (UID: \"56b6948b-89ff-4d48-ba8c-c0c3f63544e1\") " pod="openshift-multus/multus-additional-cni-plugins-8nv28"
Sep 18 19:56:56.489645 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.489553    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bfd97804-116b-4355-bf8f-adb95dc143af-host\") pod \"tuned-ks678\" (UID: \"bfd97804-116b-4355-bf8f-adb95dc143af\") " pod="openshift-cluster-node-tuning-operator/tuned-ks678"
Sep 18 19:56:56.489645 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.489570    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd8qv\" (UniqueName: \"kubernetes.io/projected/ed609df8-4cad-445d-99a8-2df85ce75ba8-kube-api-access-nd8qv\") pod \"network-check-target-rktz8\" (UID: \"ed609df8-4cad-445d-99a8-2df85ce75ba8\") " pod="openshift-network-diagnostics/network-check-target-rktz8"
Sep 18 19:56:56.489645 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.489601    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48-host-cni-bin\") pod \"ovnkube-node-scn7p\" (UID: \"a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-scn7p"
Sep 18 19:56:56.489645 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.489630    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pckm\" (UniqueName: \"kubernetes.io/projected/a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48-kube-api-access-4pckm\") pod \"ovnkube-node-scn7p\" (UID: \"a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-scn7p"
Sep 18 19:56:56.490149 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.489654    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e5f6d5d2-ce02-4de2-a798-be32d7584248-socket-dir\") pod \"aws-ebs-csi-driver-node-dfp4z\" (UID: \"e5f6d5d2-ce02-4de2-a798-be32d7584248\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfp4z"
Sep 18 19:56:56.490149 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.489677    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f4120871-0e94-46f5-bd6d-739f9b0383bf-system-cni-dir\") pod \"multus-gk46f\" (UID: \"f4120871-0e94-46f5-bd6d-739f9b0383bf\") " pod="openshift-multus/multus-gk46f"
Sep 18 19:56:56.490149 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.489702    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48-etc-openvswitch\") pod \"ovnkube-node-scn7p\" (UID: \"a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-scn7p"
Sep 18 19:56:56.490149 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.489727    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48-log-socket\") pod \"ovnkube-node-scn7p\" (UID: \"a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-scn7p"
Sep 18 19:56:56.490149 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.489752    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/254868d4-6e40-410f-9a3d-0f994b704438-kubelet-config\") pod \"global-pull-secret-syncer-zq878\" (UID: \"254868d4-6e40-410f-9a3d-0f994b704438\") " pod="kube-system/global-pull-secret-syncer-zq878"
Sep 18 19:56:56.490149 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.489755    2549 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 18 19:56:56.490149 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.489784    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqc97\" (UniqueName: \"kubernetes.io/projected/14b84844-b7fa-4a6b-a930-4f40a8230980-kube-api-access-pqc97\") pod \"iptables-alerter-5bnj6\" (UID: \"14b84844-b7fa-4a6b-a930-4f40a8230980\") " pod="openshift-network-operator/iptables-alerter-5bnj6"
Sep 18 19:56:56.490149 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.489809    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/56b6948b-89ff-4d48-ba8c-c0c3f63544e1-cni-binary-copy\") pod \"multus-additional-cni-plugins-8nv28\" (UID: \"56b6948b-89ff-4d48-ba8c-c0c3f63544e1\") " pod="openshift-multus/multus-additional-cni-plugins-8nv28"
Sep 18 19:56:56.490149 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.489841    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bfd97804-116b-4355-bf8f-adb95dc143af-tmp\") pod \"tuned-ks678\" (UID: \"bfd97804-116b-4355-bf8f-adb95dc143af\") " pod="openshift-cluster-node-tuning-operator/tuned-ks678"
Sep 18 19:56:56.490149 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.489865    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48-node-log\") pod \"ovnkube-node-scn7p\" (UID: \"a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-scn7p"
Sep 18 19:56:56.490149 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.489888    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48-env-overrides\") pod \"ovnkube-node-scn7p\" (UID: \"a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-scn7p"
Sep 18 19:56:56.490149 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.489912    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e5f6d5d2-ce02-4de2-a798-be32d7584248-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dfp4z\" (UID: \"e5f6d5d2-ce02-4de2-a798-be32d7584248\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfp4z"
Sep 18 19:56:56.490149 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.489936    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f4120871-0e94-46f5-bd6d-739f9b0383bf-etc-kubernetes\") pod \"multus-gk46f\" (UID: \"f4120871-0e94-46f5-bd6d-739f9b0383bf\") " pod="openshift-multus/multus-gk46f"
Sep 18 19:56:56.490149 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.489960    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48-run-systemd\") pod \"ovnkube-node-scn7p\" (UID: \"a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-scn7p"
Sep 18 19:56:56.490149 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.489994    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f4120871-0e94-46f5-bd6d-739f9b0383bf-host-run-multus-certs\") pod \"multus-gk46f\" (UID: \"f4120871-0e94-46f5-bd6d-739f9b0383bf\") " pod="openshift-multus/multus-gk46f"
Sep 18 19:56:56.490149 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.490019    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/14b84844-b7fa-4a6b-a930-4f40a8230980-iptables-alerter-script\") pod \"iptables-alerter-5bnj6\" (UID: \"14b84844-b7fa-4a6b-a930-4f40a8230980\") " pod="openshift-network-operator/iptables-alerter-5bnj6"
Sep 18 19:56:56.490149 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.490046    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bfd97804-116b-4355-bf8f-adb95dc143af-etc-modprobe-d\") pod \"tuned-ks678\" (UID: \"bfd97804-116b-4355-bf8f-adb95dc143af\") " pod="openshift-cluster-node-tuning-operator/tuned-ks678"
Sep 18 19:56:56.491001 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.490074    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bfd97804-116b-4355-bf8f-adb95dc143af-etc-kubernetes\") pod \"tuned-ks678\" (UID: \"bfd97804-116b-4355-bf8f-adb95dc143af\") " pod="openshift-cluster-node-tuning-operator/tuned-ks678"
Sep 18 19:56:56.491001 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.490099    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f4120871-0e94-46f5-bd6d-739f9b0383bf-host-var-lib-kubelet\") pod \"multus-gk46f\" (UID: \"f4120871-0e94-46f5-bd6d-739f9b0383bf\") " pod="openshift-multus/multus-gk46f"
Sep 18 19:56:56.491001 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.490125    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/254868d4-6e40-410f-9a3d-0f994b704438-original-pull-secret\") pod \"global-pull-secret-syncer-zq878\" (UID: \"254868d4-6e40-410f-9a3d-0f994b704438\") " pod="kube-system/global-pull-secret-syncer-zq878"
Sep 18 19:56:56.491001 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.490149    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f4120871-0e94-46f5-bd6d-739f9b0383bf-host-run-netns\") pod \"multus-gk46f\" (UID: \"f4120871-0e94-46f5-bd6d-739f9b0383bf\") " pod="openshift-multus/multus-gk46f"
Sep 18 19:56:56.491001 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.490173    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/56b6948b-89ff-4d48-ba8c-c0c3f63544e1-system-cni-dir\") pod \"multus-additional-cni-plugins-8nv28\" (UID: \"56b6948b-89ff-4d48-ba8c-c0c3f63544e1\") " pod="openshift-multus/multus-additional-cni-plugins-8nv28"
Sep 18 19:56:56.491001 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.490200    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bfd97804-116b-4355-bf8f-adb95dc143af-run\") pod \"tuned-ks678\" (UID: \"bfd97804-116b-4355-bf8f-adb95dc143af\") " pod="openshift-cluster-node-tuning-operator/tuned-ks678"
Sep 18 19:56:56.491001 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.490225    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48-host-cni-netd\") pod \"ovnkube-node-scn7p\" (UID: \"a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-scn7p"
Sep 18 19:56:56.491001 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.490229    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bfd97804-116b-4355-bf8f-adb95dc143af-etc-modprobe-d\") pod \"tuned-ks678\" (UID: \"bfd97804-116b-4355-bf8f-adb95dc143af\") " pod="openshift-cluster-node-tuning-operator/tuned-ks678"
Sep 18 19:56:56.491001 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.490250    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e5f6d5d2-ce02-4de2-a798-be32d7584248-device-dir\") pod \"aws-ebs-csi-driver-node-dfp4z\" (UID: \"e5f6d5d2-ce02-4de2-a798-be32d7584248\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfp4z"
Sep 18 19:56:56.491001 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.490284    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbszj\" (UniqueName: \"kubernetes.io/projected/cb6d3f0d-bc38-43aa-8810-3b14a624295d-kube-api-access-pbszj\") pod \"node-ca-m6qj6\" (UID: \"cb6d3f0d-bc38-43aa-8810-3b14a624295d\") " pod="openshift-image-registry/node-ca-m6qj6"
Sep 18 19:56:56.491001 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.490309    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f4120871-0e94-46f5-bd6d-739f9b0383bf-multus-cni-dir\") pod \"multus-gk46f\" (UID: \"f4120871-0e94-46f5-bd6d-739f9b0383bf\") " pod="openshift-multus/multus-gk46f"
Sep 18 19:56:56.491001 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.490330    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f4120871-0e94-46f5-bd6d-739f9b0383bf-multus-socket-dir-parent\") pod \"multus-gk46f\" (UID: \"f4120871-0e94-46f5-bd6d-739f9b0383bf\") " pod="openshift-multus/multus-gk46f"
Sep 18 19:56:56.491001 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.490351    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5x6f\" (UniqueName: \"kubernetes.io/projected/f4120871-0e94-46f5-bd6d-739f9b0383bf-kube-api-access-b5x6f\") pod \"multus-gk46f\" (UID: \"f4120871-0e94-46f5-bd6d-739f9b0383bf\") " pod="openshift-multus/multus-gk46f"
Sep 18 19:56:56.491001 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.490359    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bfd97804-116b-4355-bf8f-adb95dc143af-etc-kubernetes\") pod \"tuned-ks678\" (UID: \"bfd97804-116b-4355-bf8f-adb95dc143af\") " pod="openshift-cluster-node-tuning-operator/tuned-ks678"
Sep 18 19:56:56.491001 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.490374    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bfd97804-116b-4355-bf8f-adb95dc143af-etc-systemd\") pod \"tuned-ks678\" (UID: \"bfd97804-116b-4355-bf8f-adb95dc143af\") " pod="openshift-cluster-node-tuning-operator/tuned-ks678"
Sep 18 19:56:56.491001 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.490406    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bfd97804-116b-4355-bf8f-adb95dc143af-sys\") pod \"tuned-ks678\" (UID: \"bfd97804-116b-4355-bf8f-adb95dc143af\") " pod="openshift-cluster-node-tuning-operator/tuned-ks678"
Sep 18 19:56:56.491001 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.490413    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bfd97804-116b-4355-bf8f-adb95dc143af-etc-systemd\") pod \"tuned-ks678\" (UID: \"bfd97804-116b-4355-bf8f-adb95dc143af\") " pod="openshift-cluster-node-tuning-operator/tuned-ks678"
Sep 18 19:56:56.491921 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.490435    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9b27\" (UniqueName: \"kubernetes.io/projected/063d8ee5-35b4-401b-a0ff-d8b416e20175-kube-api-access-n9b27\") pod \"network-metrics-daemon-2hw77\" (UID: \"063d8ee5-35b4-401b-a0ff-d8b416e20175\") " pod="openshift-multus/network-metrics-daemon-2hw77"
Sep 18 19:56:56.491921 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.490478    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e5f6d5d2-ce02-4de2-a798-be32d7584248-sys-fs\") pod \"aws-ebs-csi-driver-node-dfp4z\" (UID: \"e5f6d5d2-ce02-4de2-a798-be32d7584248\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfp4z"
Sep 18 19:56:56.491921 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.490510    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cb6d3f0d-bc38-43aa-8810-3b14a624295d-host\") pod \"node-ca-m6qj6\" (UID: \"cb6d3f0d-bc38-43aa-8810-3b14a624295d\") " pod="openshift-image-registry/node-ca-m6qj6"
Sep 18 19:56:56.491921 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.490542    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/56b6948b-89ff-4d48-ba8c-c0c3f63544e1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8nv28\" (UID: \"56b6948b-89ff-4d48-ba8c-c0c3f63544e1\") " pod="openshift-multus/multus-additional-cni-plugins-8nv28"
Sep 18 19:56:56.491921 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.490573    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bfd97804-116b-4355-bf8f-adb95dc143af-run\") pod \"tuned-ks678\" (UID: \"bfd97804-116b-4355-bf8f-adb95dc143af\") " pod="openshift-cluster-node-tuning-operator/tuned-ks678"
Sep 18 19:56:56.491921 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.490618    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bfd97804-116b-4355-bf8f-adb95dc143af-sys\") pod \"tuned-ks678\" (UID: \"bfd97804-116b-4355-bf8f-adb95dc143af\") " pod="openshift-cluster-node-tuning-operator/tuned-ks678"
Sep 18 19:56:56.491921 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.490630    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2e178884-5bfb-46cb-9305-6170284cb243-konnectivity-ca\") pod \"konnectivity-agent-tlqj8\" (UID: \"2e178884-5bfb-46cb-9305-6170284cb243\") " pod="kube-system/konnectivity-agent-tlqj8"
Sep 18 19:56:56.491921 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.490661    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bfd97804-116b-4355-bf8f-adb95dc143af-etc-sysctl-d\") pod \"tuned-ks678\" (UID: \"bfd97804-116b-4355-bf8f-adb95dc143af\") " pod="openshift-cluster-node-tuning-operator/tuned-ks678"
Sep 18 19:56:56.491921 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.490688    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bfd97804-116b-4355-bf8f-adb95dc143af-etc-sysctl-conf\") pod \"tuned-ks678\" (UID: \"bfd97804-116b-4355-bf8f-adb95dc143af\") " pod="openshift-cluster-node-tuning-operator/tuned-ks678"
Sep 18 19:56:56.491921 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.490712    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bfd97804-116b-4355-bf8f-adb95dc143af-lib-modules\") pod \"tuned-ks678\" (UID: \"bfd97804-116b-4355-bf8f-adb95dc143af\") " pod="openshift-cluster-node-tuning-operator/tuned-ks678"
Sep 18 19:56:56.491921 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.490738    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48-run-ovn\") pod \"ovnkube-node-scn7p\" (UID: \"a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-scn7p"
Sep 18 19:56:56.491921 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.490762    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48-ovn-node-metrics-cert\") pod \"ovnkube-node-scn7p\" (UID: \"a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-scn7p"
Sep 18 19:56:56.491921 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.490790    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e5f6d5d2-ce02-4de2-a798-be32d7584248-registration-dir\") pod \"aws-ebs-csi-driver-node-dfp4z\" (UID: \"e5f6d5d2-ce02-4de2-a798-be32d7584248\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfp4z"
Sep 18 19:56:56.491921 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.490819    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5md6\" (UniqueName: \"kubernetes.io/projected/e5f6d5d2-ce02-4de2-a798-be32d7584248-kube-api-access-g5md6\") pod \"aws-ebs-csi-driver-node-dfp4z\" (UID: \"e5f6d5d2-ce02-4de2-a798-be32d7584248\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfp4z"
Sep 18 19:56:56.491921 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.490845    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e5f6d5d2-ce02-4de2-a798-be32d7584248-etc-selinux\") pod \"aws-ebs-csi-driver-node-dfp4z\" (UID: \"e5f6d5d2-ce02-4de2-a798-be32d7584248\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfp4z"
Sep 18 19:56:56.491921 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.490870    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48-host-run-netns\") pod \"ovnkube-node-scn7p\" (UID: \"a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-scn7p"
Sep 18 19:56:56.491921 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.490895    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48-ovnkube-config\") pod \"ovnkube-node-scn7p\" (UID: \"a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-scn7p"
Sep 18 19:56:56.492825 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.490918    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cb6d3f0d-bc38-43aa-8810-3b14a624295d-serviceca\") pod \"node-ca-m6qj6\" (UID: \"cb6d3f0d-bc38-43aa-8810-3b14a624295d\") " pod="openshift-image-registry/node-ca-m6qj6"
Sep 18 19:56:56.492825 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.490943    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f4120871-0e94-46f5-bd6d-739f9b0383bf-hostroot\") pod \"multus-gk46f\" (UID: \"f4120871-0e94-46f5-bd6d-739f9b0383bf\") " pod="openshift-multus/multus-gk46f"
Sep 18 19:56:56.492825 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.490966    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/14b84844-b7fa-4a6b-a930-4f40a8230980-host-slash\") pod \"iptables-alerter-5bnj6\" (UID: \"14b84844-b7fa-4a6b-a930-4f40a8230980\") " pod="openshift-network-operator/iptables-alerter-5bnj6"
Sep 18 19:56:56.492825 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.490990    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/525400e6-6678-424f-a1ba-0896aac41264-tmp-dir\") pod \"node-resolver-t26hd\" (UID: \"525400e6-6678-424f-a1ba-0896aac41264\") " pod="openshift-dns/node-resolver-t26hd"
Sep 18 19:56:56.492825 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.491027    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn2pt\" (UniqueName: \"kubernetes.io/projected/525400e6-6678-424f-a1ba-0896aac41264-kube-api-access-hn2pt\") pod \"node-resolver-t26hd\" (UID: \"525400e6-6678-424f-a1ba-0896aac41264\") " pod="openshift-dns/node-resolver-t26hd"
Sep 18 19:56:56.492825 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.491056    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2e178884-5bfb-46cb-9305-6170284cb243-agent-certs\") pod \"konnectivity-agent-tlqj8\" (UID: \"2e178884-5bfb-46cb-9305-6170284cb243\") " pod="kube-system/konnectivity-agent-tlqj8"
Sep 18 19:56:56.492825 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.491080    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48-var-lib-openvswitch\") pod \"ovnkube-node-scn7p\" (UID: \"a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-scn7p"
Sep 18 19:56:56.492825 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.491109    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-scn7p\" (UID: \"a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-scn7p"
Sep 18 19:56:56.492825 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.491129    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2e178884-5bfb-46cb-9305-6170284cb243-konnectivity-ca\") pod \"konnectivity-agent-tlqj8\" (UID: \"2e178884-5bfb-46cb-9305-6170284cb243\") " pod="kube-system/konnectivity-agent-tlqj8"
Sep 18 19:56:56.492825 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.491134    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/254868d4-6e40-410f-9a3d-0f994b704438-dbus\") pod \"global-pull-secret-syncer-zq878\" (UID: \"254868d4-6e40-410f-9a3d-0f994b704438\") " pod="kube-system/global-pull-secret-syncer-zq878"
Sep 18 19:56:56.492825 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.491162    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f4120871-0e94-46f5-bd6d-739f9b0383bf-cni-binary-copy\") pod \"multus-gk46f\" (UID: \"f4120871-0e94-46f5-bd6d-739f9b0383bf\") " pod="openshift-multus/multus-gk46f"
Sep 18 19:56:56.492825 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.491203    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f4120871-0e94-46f5-bd6d-739f9b0383bf-host-var-lib-cni-multus\") pod \"multus-gk46f\" (UID: \"f4120871-0e94-46f5-bd6d-739f9b0383bf\") " pod="openshift-multus/multus-gk46f"
Sep 18 19:56:56.492825 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.491276    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f4120871-0e94-46f5-bd6d-739f9b0383bf-multus-daemon-config\") pod \"multus-gk46f\" (UID: \"f4120871-0e94-46f5-bd6d-739f9b0383bf\") " pod="openshift-multus/multus-gk46f"
Sep 18 19:56:56.492825 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.491285    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bfd97804-116b-4355-bf8f-adb95dc143af-etc-sysctl-d\") pod \"tuned-ks678\" (UID: \"bfd97804-116b-4355-bf8f-adb95dc143af\") " pod="openshift-cluster-node-tuning-operator/tuned-ks678"
Sep 18 19:56:56.492825 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.491302    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48-systemd-units\") pod \"ovnkube-node-scn7p\" (UID: \"a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-scn7p"
Sep 18 19:56:56.492825 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.491331    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48-host-run-ovn-kubernetes\") pod \"ovnkube-node-scn7p\" (UID: \"a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-scn7p"
Sep 18 19:56:56.492825 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.491356    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/56b6948b-89ff-4d48-ba8c-c0c3f63544e1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8nv28\" (UID: \"56b6948b-89ff-4d48-ba8c-c0c3f63544e1\") " pod="openshift-multus/multus-additional-cni-plugins-8nv28"
Sep 18 19:56:56.493728 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.491389    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bfd97804-116b-4355-bf8f-adb95dc143af-etc-sysctl-conf\") pod \"tuned-ks678\" (UID: \"bfd97804-116b-4355-bf8f-adb95dc143af\") " pod="openshift-cluster-node-tuning-operator/tuned-ks678"
Sep 18 19:56:56.493728 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.491394    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q458c\" (UniqueName: \"kubernetes.io/projected/bfd97804-116b-4355-bf8f-adb95dc143af-kube-api-access-q458c\") pod \"tuned-ks678\" (UID: \"bfd97804-116b-4355-bf8f-adb95dc143af\") " pod="openshift-cluster-node-tuning-operator/tuned-ks678"
Sep 18 19:56:56.493728 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.491425    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48-host-kubelet\") pod \"ovnkube-node-scn7p\" (UID: \"a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-scn7p"
Sep 18 19:56:56.493728 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.491452    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg7xq\" (UniqueName: \"kubernetes.io/projected/56b6948b-89ff-4d48-ba8c-c0c3f63544e1-kube-api-access-cg7xq\") pod \"multus-additional-cni-plugins-8nv28\" (UID: \"56b6948b-89ff-4d48-ba8c-c0c3f63544e1\") " pod="openshift-multus/multus-additional-cni-plugins-8nv28"
Sep 18 19:56:56.493728 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.491490    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bfd97804-116b-4355-bf8f-adb95dc143af-lib-modules\") pod \"tuned-ks678\" (UID: \"bfd97804-116b-4355-bf8f-adb95dc143af\") " pod="openshift-cluster-node-tuning-operator/tuned-ks678"
Sep 18 19:56:56.493728 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.491494    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48-host-slash\") pod \"ovnkube-node-scn7p\" (UID: \"a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-scn7p"
Sep 18 19:56:56.493728 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.491843    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48-run-openvswitch\") pod \"ovnkube-node-scn7p\" (UID: \"a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-scn7p"
Sep 18 19:56:56.493728 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.491876    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48-ovnkube-script-lib\") pod \"ovnkube-node-scn7p\" (UID: \"a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-scn7p"
Sep 18 19:56:56.493728 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.491908    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f4120871-0e94-46f5-bd6d-739f9b0383bf-host-run-k8s-cni-cncf-io\") pod \"multus-gk46f\" (UID: \"f4120871-0e94-46f5-bd6d-739f9b0383bf\") " pod="openshift-multus/multus-gk46f"
Sep 18 19:56:56.493728 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.491962    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f4120871-0e94-46f5-bd6d-739f9b0383bf-host-var-lib-cni-bin\") pod \"multus-gk46f\" (UID: \"f4120871-0e94-46f5-bd6d-739f9b0383bf\") " pod="openshift-multus/multus-gk46f"
Sep 18 19:56:56.493728 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.491993    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f4120871-0e94-46f5-bd6d-739f9b0383bf-multus-conf-dir\") pod \"multus-gk46f\" (UID: \"f4120871-0e94-46f5-bd6d-739f9b0383bf\") " pod="openshift-multus/multus-gk46f"
Sep 18 19:56:56.493728 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.493707    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bfd97804-116b-4355-bf8f-adb95dc143af-tmp\") pod \"tuned-ks678\" (UID: \"bfd97804-116b-4355-bf8f-adb95dc143af\") " pod="openshift-cluster-node-tuning-operator/tuned-ks678"
Sep 18 19:56:56.494659 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.493786    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bfd97804-116b-4355-bf8f-adb95dc143af-etc-tuned\") pod \"tuned-ks678\" (UID: \"bfd97804-116b-4355-bf8f-adb95dc143af\") " pod="openshift-cluster-node-tuning-operator/tuned-ks678"
Sep 18 19:56:56.495362 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.495337    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2e178884-5bfb-46cb-9305-6170284cb243-agent-certs\") pod \"konnectivity-agent-tlqj8\" (UID: \"2e178884-5bfb-46cb-9305-6170284cb243\") " pod="kube-system/konnectivity-agent-tlqj8"
Sep 18 19:56:56.502941 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.502915    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q458c\" (UniqueName: \"kubernetes.io/projected/bfd97804-116b-4355-bf8f-adb95dc143af-kube-api-access-q458c\") pod \"tuned-ks678\" (UID: \"bfd97804-116b-4355-bf8f-adb95dc143af\") " pod="openshift-cluster-node-tuning-operator/tuned-ks678"
Sep 18 19:56:56.506043 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:56.506014    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2748dceba5a1bbaba05c6bb022d69842.slice/crio-a67ba30450c1c6ccbb3cd71212b1a6b4e2d69344f7490b30f4641595cc7a1c77 WatchSource:0}: Error finding container a67ba30450c1c6ccbb3cd71212b1a6b4e2d69344f7490b30f4641595cc7a1c77: Status 404 returned error can't find the container with id a67ba30450c1c6ccbb3cd71212b1a6b4e2d69344f7490b30f4641595cc7a1c77
Sep 18 19:56:56.506310 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:56.506290    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode511ba71d033b03f5d71c14df866e732.slice/crio-7a6a8dea613a5aaedca8b87dc5a18157de2a28ac70b665ce2487161979fd03b4 WatchSource:0}: Error finding container 7a6a8dea613a5aaedca8b87dc5a18157de2a28ac70b665ce2487161979fd03b4: Status 404 returned error can't find the container with id 7a6a8dea613a5aaedca8b87dc5a18157de2a28ac70b665ce2487161979fd03b4
Sep 18 19:56:56.511359 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.511344    2549 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 19:56:56.517440 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.517403    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-178.ec2.internal" event={"ID":"2748dceba5a1bbaba05c6bb022d69842","Type":"ContainerStarted","Data":"a67ba30450c1c6ccbb3cd71212b1a6b4e2d69344f7490b30f4641595cc7a1c77"}
Sep 18 19:56:56.518563 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.518542    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-178.ec2.internal" event={"ID":"e511ba71d033b03f5d71c14df866e732","Type":"ContainerStarted","Data":"7a6a8dea613a5aaedca8b87dc5a18157de2a28ac70b665ce2487161979fd03b4"}
Sep 18 19:56:56.592532 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.592500    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f4120871-0e94-46f5-bd6d-739f9b0383bf-system-cni-dir\") pod \"multus-gk46f\" (UID: \"f4120871-0e94-46f5-bd6d-739f9b0383bf\") " pod="openshift-multus/multus-gk46f"
Sep 18 19:56:56.592532 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.592528    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48-etc-openvswitch\") pod \"ovnkube-node-scn7p\" (UID: \"a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-scn7p"
Sep 18 19:56:56.592814 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.592545    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48-log-socket\") pod \"ovnkube-node-scn7p\" (UID: \"a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-scn7p"
Sep 18 19:56:56.592814 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.592566    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/254868d4-6e40-410f-9a3d-0f994b704438-kubelet-config\") pod \"global-pull-secret-syncer-zq878\" (UID: \"254868d4-6e40-410f-9a3d-0f994b704438\") " pod="kube-system/global-pull-secret-syncer-zq878"
Sep 18 19:56:56.592814 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.592582    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqc97\" (UniqueName: \"kubernetes.io/projected/14b84844-b7fa-4a6b-a930-4f40a8230980-kube-api-access-pqc97\") pod \"iptables-alerter-5bnj6\" (UID: \"14b84844-b7fa-4a6b-a930-4f40a8230980\") " pod="openshift-network-operator/iptables-alerter-5bnj6"
Sep 18 19:56:56.592814 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.592598    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/56b6948b-89ff-4d48-ba8c-c0c3f63544e1-cni-binary-copy\") pod \"multus-additional-cni-plugins-8nv28\" (UID: \"56b6948b-89ff-4d48-ba8c-c0c3f63544e1\") " pod="openshift-multus/multus-additional-cni-plugins-8nv28"
Sep 18 19:56:56.592814 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.592614    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48-node-log\") pod \"ovnkube-node-scn7p\" (UID: \"a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-scn7p"
Sep 18 19:56:56.592814 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.592617    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48-etc-openvswitch\") pod \"ovnkube-node-scn7p\" (UID: \"a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-scn7p"
Sep 18 19:56:56.592814 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.592632    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48-log-socket\") pod \"ovnkube-node-scn7p\" (UID: \"a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-scn7p"
Sep 18 19:56:56.592814 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.592631    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48-env-overrides\") pod \"ovnkube-node-scn7p\" (UID: \"a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-scn7p"
Sep 18 19:56:56.592814 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.592618    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f4120871-0e94-46f5-bd6d-739f9b0383bf-system-cni-dir\") pod \"multus-gk46f\" (UID: \"f4120871-0e94-46f5-bd6d-739f9b0383bf\") " pod="openshift-multus/multus-gk46f"
Sep 18 19:56:56.592814 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.592656    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/254868d4-6e40-410f-9a3d-0f994b704438-kubelet-config\") pod \"global-pull-secret-syncer-zq878\" (UID: \"254868d4-6e40-410f-9a3d-0f994b704438\") " pod="kube-system/global-pull-secret-syncer-zq878"
Sep 18 19:56:56.592814 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.592681    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48-node-log\") pod \"ovnkube-node-scn7p\" (UID: \"a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-scn7p"
Sep 18 19:56:56.592814 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.592722    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e5f6d5d2-ce02-4de2-a798-be32d7584248-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dfp4z\" (UID: \"e5f6d5d2-ce02-4de2-a798-be32d7584248\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfp4z"
Sep 18 19:56:56.593298 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.592846    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f4120871-0e94-46f5-bd6d-739f9b0383bf-etc-kubernetes\") pod \"multus-gk46f\" (UID: \"f4120871-0e94-46f5-bd6d-739f9b0383bf\") " pod="openshift-multus/multus-gk46f"
Sep 18 19:56:56.593298 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.592856    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e5f6d5d2-ce02-4de2-a798-be32d7584248-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dfp4z\" (UID: \"e5f6d5d2-ce02-4de2-a798-be32d7584248\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfp4z"
Sep 18 19:56:56.593298 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.592924    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48-run-systemd\") pod \"ovnkube-node-scn7p\" (UID: \"a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-scn7p"
Sep 18 19:56:56.593298 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.592931    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f4120871-0e94-46f5-bd6d-739f9b0383bf-etc-kubernetes\") pod \"multus-gk46f\" (UID: \"f4120871-0e94-46f5-bd6d-739f9b0383bf\") " pod="openshift-multus/multus-gk46f"
Sep 18 19:56:56.593298 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.592980    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f4120871-0e94-46f5-bd6d-739f9b0383bf-host-run-multus-certs\") pod \"multus-gk46f\" (UID: \"f4120871-0e94-46f5-bd6d-739f9b0383bf\") " pod="openshift-multus/multus-gk46f"
Sep 18 19:56:56.593298 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.593033    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48-run-systemd\") pod \"ovnkube-node-scn7p\" (UID: \"a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-scn7p"
Sep 18 19:56:56.593298 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.593055    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/14b84844-b7fa-4a6b-a930-4f40a8230980-iptables-alerter-script\") pod \"iptables-alerter-5bnj6\" (UID: \"14b84844-b7fa-4a6b-a930-4f40a8230980\") " pod="openshift-network-operator/iptables-alerter-5bnj6"
Sep 18 19:56:56.593298 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.593083    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f4120871-0e94-46f5-bd6d-739f9b0383bf-host-run-multus-certs\") pod \"multus-gk46f\" (UID: \"f4120871-0e94-46f5-bd6d-739f9b0383bf\") " pod="openshift-multus/multus-gk46f"
Sep 18 19:56:56.593298 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.593175    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48-env-overrides\") pod \"ovnkube-node-scn7p\" (UID: \"a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-scn7p"
Sep 18 19:56:56.593298 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.593179    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/56b6948b-89ff-4d48-ba8c-c0c3f63544e1-cni-binary-copy\") pod \"multus-additional-cni-plugins-8nv28\" (UID: \"56b6948b-89ff-4d48-ba8c-c0c3f63544e1\") " pod="openshift-multus/multus-additional-cni-plugins-8nv28"
Sep 18 19:56:56.593298 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.593213    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f4120871-0e94-46f5-bd6d-739f9b0383bf-host-var-lib-kubelet\") pod \"multus-gk46f\" (UID: \"f4120871-0e94-46f5-bd6d-739f9b0383bf\") " pod="openshift-multus/multus-gk46f"
Sep 18 19:56:56.593298 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.593241    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/254868d4-6e40-410f-9a3d-0f994b704438-original-pull-secret\") pod \"global-pull-secret-syncer-zq878\" (UID: \"254868d4-6e40-410f-9a3d-0f994b704438\") " pod="kube-system/global-pull-secret-syncer-zq878"
Sep 18 19:56:56.593298 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.593259    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f4120871-0e94-46f5-bd6d-739f9b0383bf-host-run-netns\") pod \"multus-gk46f\" (UID: \"f4120871-0e94-46f5-bd6d-739f9b0383bf\") " pod="openshift-multus/multus-gk46f"
Sep 18 19:56:56.593298 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.593275    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/56b6948b-89ff-4d48-ba8c-c0c3f63544e1-system-cni-dir\") pod \"multus-additional-cni-plugins-8nv28\" (UID: \"56b6948b-89ff-4d48-ba8c-c0c3f63544e1\") " pod="openshift-multus/multus-additional-cni-plugins-8nv28"
Sep 18 19:56:56.593972 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.593317    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/56b6948b-89ff-4d48-ba8c-c0c3f63544e1-system-cni-dir\") pod \"multus-additional-cni-plugins-8nv28\" (UID: \"56b6948b-89ff-4d48-ba8c-c0c3f63544e1\") " pod="openshift-multus/multus-additional-cni-plugins-8nv28"
Sep 18 19:56:56.593972 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.593319    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f4120871-0e94-46f5-bd6d-739f9b0383bf-host-run-netns\") pod \"multus-gk46f\" (UID: \"f4120871-0e94-46f5-bd6d-739f9b0383bf\") " pod="openshift-multus/multus-gk46f"
Sep 18 19:56:56.593972 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:56:56.593345    2549 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:56:56.593972 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.593354    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48-host-cni-netd\") pod \"ovnkube-node-scn7p\" (UID: \"a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-scn7p"
Sep 18 19:56:56.593972 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.593381    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e5f6d5d2-ce02-4de2-a798-be32d7584248-device-dir\") pod \"aws-ebs-csi-driver-node-dfp4z\" (UID: \"e5f6d5d2-ce02-4de2-a798-be32d7584248\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfp4z"
Sep 18 19:56:56.593972 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.593347    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f4120871-0e94-46f5-bd6d-739f9b0383bf-host-var-lib-kubelet\") pod \"multus-gk46f\" (UID: \"f4120871-0e94-46f5-bd6d-739f9b0383bf\") " pod="openshift-multus/multus-gk46f"
Sep 18 19:56:56.593972 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:56:56.593463    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/254868d4-6e40-410f-9a3d-0f994b704438-original-pull-secret podName:254868d4-6e40-410f-9a3d-0f994b704438 nodeName:}" failed. No retries permitted until 2026-09-18 19:56:57.093438658 +0000 UTC m=+2.047179824 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/254868d4-6e40-410f-9a3d-0f994b704438-original-pull-secret") pod "global-pull-secret-syncer-zq878" (UID: "254868d4-6e40-410f-9a3d-0f994b704438") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:56:56.593972 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.593458    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48-host-cni-netd\") pod \"ovnkube-node-scn7p\" (UID: \"a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-scn7p"
Sep 18 19:56:56.593972 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.593463    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e5f6d5d2-ce02-4de2-a798-be32d7584248-device-dir\") pod \"aws-ebs-csi-driver-node-dfp4z\" (UID: \"e5f6d5d2-ce02-4de2-a798-be32d7584248\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfp4z"
Sep 18 19:56:56.593972 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.593427    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbszj\" (UniqueName: \"kubernetes.io/projected/cb6d3f0d-bc38-43aa-8810-3b14a624295d-kube-api-access-pbszj\") pod \"node-ca-m6qj6\" (UID: \"cb6d3f0d-bc38-43aa-8810-3b14a624295d\") " pod="openshift-image-registry/node-ca-m6qj6"
Sep 18 19:56:56.593972 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.593517    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f4120871-0e94-46f5-bd6d-739f9b0383bf-multus-cni-dir\") pod \"multus-gk46f\" (UID: \"f4120871-0e94-46f5-bd6d-739f9b0383bf\") " pod="openshift-multus/multus-gk46f"
Sep 18 19:56:56.593972 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.593543    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f4120871-0e94-46f5-bd6d-739f9b0383bf-multus-socket-dir-parent\") pod \"multus-gk46f\" (UID: \"f4120871-0e94-46f5-bd6d-739f9b0383bf\") " pod="openshift-multus/multus-gk46f"
Sep 18 19:56:56.593972 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.593571    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5x6f\" (UniqueName: \"kubernetes.io/projected/f4120871-0e94-46f5-bd6d-739f9b0383bf-kube-api-access-b5x6f\") pod \"multus-gk46f\" (UID: \"f4120871-0e94-46f5-bd6d-739f9b0383bf\") " pod="openshift-multus/multus-gk46f"
Sep 18 19:56:56.593972 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.593598    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f4120871-0e94-46f5-bd6d-739f9b0383bf-multus-cni-dir\") pod \"multus-gk46f\" (UID: \"f4120871-0e94-46f5-bd6d-739f9b0383bf\") " pod="openshift-multus/multus-gk46f"
Sep 18 19:56:56.593972 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.593614    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/14b84844-b7fa-4a6b-a930-4f40a8230980-iptables-alerter-script\") pod \"iptables-alerter-5bnj6\" (UID: \"14b84844-b7fa-4a6b-a930-4f40a8230980\") " pod="openshift-network-operator/iptables-alerter-5bnj6"
Sep 18 19:56:56.593972 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.593601    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9b27\" (UniqueName: \"kubernetes.io/projected/063d8ee5-35b4-401b-a0ff-d8b416e20175-kube-api-access-n9b27\") pod \"network-metrics-daemon-2hw77\" (UID: \"063d8ee5-35b4-401b-a0ff-d8b416e20175\") " pod="openshift-multus/network-metrics-daemon-2hw77"
Sep 18 19:56:56.593972 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.593652    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f4120871-0e94-46f5-bd6d-739f9b0383bf-multus-socket-dir-parent\") pod \"multus-gk46f\" (UID: \"f4120871-0e94-46f5-bd6d-739f9b0383bf\") " pod="openshift-multus/multus-gk46f"
Sep 18 19:56:56.594828 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.593656    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e5f6d5d2-ce02-4de2-a798-be32d7584248-sys-fs\") pod \"aws-ebs-csi-driver-node-dfp4z\" (UID: \"e5f6d5d2-ce02-4de2-a798-be32d7584248\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfp4z"
Sep 18 19:56:56.594828 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.593700    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cb6d3f0d-bc38-43aa-8810-3b14a624295d-host\") pod \"node-ca-m6qj6\" (UID: \"cb6d3f0d-bc38-43aa-8810-3b14a624295d\") " pod="openshift-image-registry/node-ca-m6qj6"
Sep 18 19:56:56.594828 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.593701    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e5f6d5d2-ce02-4de2-a798-be32d7584248-sys-fs\") pod \"aws-ebs-csi-driver-node-dfp4z\" (UID: \"e5f6d5d2-ce02-4de2-a798-be32d7584248\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfp4z"
Sep 18 19:56:56.594828 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.593721    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/56b6948b-89ff-4d48-ba8c-c0c3f63544e1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8nv28\" (UID: \"56b6948b-89ff-4d48-ba8c-c0c3f63544e1\") " pod="openshift-multus/multus-additional-cni-plugins-8nv28"
Sep 18 19:56:56.594828 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.593742    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cb6d3f0d-bc38-43aa-8810-3b14a624295d-host\") pod \"node-ca-m6qj6\" (UID: \"cb6d3f0d-bc38-43aa-8810-3b14a624295d\") " pod="openshift-image-registry/node-ca-m6qj6"
Sep 18 19:56:56.594828 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.593745    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48-run-ovn\") pod \"ovnkube-node-scn7p\" (UID: \"a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-scn7p"
Sep 18 19:56:56.594828 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.593773    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48-run-ovn\") pod \"ovnkube-node-scn7p\" (UID: \"a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-scn7p"
Sep 18 19:56:56.594828 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.593781    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48-ovn-node-metrics-cert\") pod \"ovnkube-node-scn7p\" (UID: \"a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-scn7p"
Sep 18 19:56:56.594828 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.593809    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e5f6d5d2-ce02-4de2-a798-be32d7584248-registration-dir\") pod \"aws-ebs-csi-driver-node-dfp4z\" (UID: \"e5f6d5d2-ce02-4de2-a798-be32d7584248\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfp4z"
Sep 18 19:56:56.594828 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.593849    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5md6\" (UniqueName: \"kubernetes.io/projected/e5f6d5d2-ce02-4de2-a798-be32d7584248-kube-api-access-g5md6\") pod \"aws-ebs-csi-driver-node-dfp4z\" (UID: \"e5f6d5d2-ce02-4de2-a798-be32d7584248\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfp4z"
Sep 18 19:56:56.594828 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.593894    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e5f6d5d2-ce02-4de2-a798-be32d7584248-etc-selinux\") pod \"aws-ebs-csi-driver-node-dfp4z\" (UID: \"e5f6d5d2-ce02-4de2-a798-be32d7584248\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfp4z"
Sep 18 19:56:56.594828 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.593921    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48-host-run-netns\") pod \"ovnkube-node-scn7p\" (UID: \"a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-scn7p"
Sep 18 19:56:56.594828 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.593893    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e5f6d5d2-ce02-4de2-a798-be32d7584248-registration-dir\") pod \"aws-ebs-csi-driver-node-dfp4z\" (UID: \"e5f6d5d2-ce02-4de2-a798-be32d7584248\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfp4z"
Sep 18 19:56:56.594828 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.593946    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48-ovnkube-config\") pod \"ovnkube-node-scn7p\" (UID: \"a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-scn7p"
Sep 18 19:56:56.594828 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.593987    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cb6d3f0d-bc38-43aa-8810-3b14a624295d-serviceca\") pod \"node-ca-m6qj6\" (UID: \"cb6d3f0d-bc38-43aa-8810-3b14a624295d\") " pod="openshift-image-registry/node-ca-m6qj6"
Sep 18 19:56:56.594828 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.594006    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48-host-run-netns\") pod \"ovnkube-node-scn7p\" (UID: \"a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-scn7p"
Sep 18 19:56:56.594828 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.594010    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f4120871-0e94-46f5-bd6d-739f9b0383bf-hostroot\") pod \"multus-gk46f\" (UID: \"f4120871-0e94-46f5-bd6d-739f9b0383bf\") " pod="openshift-multus/multus-gk46f"
Sep 18 19:56:56.595488 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.594027    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e5f6d5d2-ce02-4de2-a798-be32d7584248-etc-selinux\") pod \"aws-ebs-csi-driver-node-dfp4z\" (UID: \"e5f6d5d2-ce02-4de2-a798-be32d7584248\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfp4z"
Sep 18 19:56:56.595488 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.594050    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f4120871-0e94-46f5-bd6d-739f9b0383bf-hostroot\") pod \"multus-gk46f\" (UID: \"f4120871-0e94-46f5-bd6d-739f9b0383bf\") " pod="openshift-multus/multus-gk46f"
Sep 18 19:56:56.595488 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.594052    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/14b84844-b7fa-4a6b-a930-4f40a8230980-host-slash\") pod \"iptables-alerter-5bnj6\" (UID: \"14b84844-b7fa-4a6b-a930-4f40a8230980\") " pod="openshift-network-operator/iptables-alerter-5bnj6"
Sep 18 19:56:56.595488 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.594085    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/14b84844-b7fa-4a6b-a930-4f40a8230980-host-slash\") pod \"iptables-alerter-5bnj6\" (UID: \"14b84844-b7fa-4a6b-a930-4f40a8230980\") " pod="openshift-network-operator/iptables-alerter-5bnj6"
Sep 18 19:56:56.595488 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.594091    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/525400e6-6678-424f-a1ba-0896aac41264-tmp-dir\") pod \"node-resolver-t26hd\" (UID: \"525400e6-6678-424f-a1ba-0896aac41264\") " pod="openshift-dns/node-resolver-t26hd"
Sep 18 19:56:56.595488 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.594120    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn2pt\" (UniqueName: \"kubernetes.io/projected/525400e6-6678-424f-a1ba-0896aac41264-kube-api-access-hn2pt\") pod \"node-resolver-t26hd\" (UID: \"525400e6-6678-424f-a1ba-0896aac41264\") " pod="openshift-dns/node-resolver-t26hd"
Sep 18 19:56:56.595488 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.594149    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48-var-lib-openvswitch\") pod \"ovnkube-node-scn7p\" (UID: \"a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-scn7p"
Sep 18 19:56:56.595488 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.594177    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-scn7p\" (UID: \"a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-scn7p"
Sep 18 19:56:56.595488 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.594204    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/254868d4-6e40-410f-9a3d-0f994b704438-dbus\") pod \"global-pull-secret-syncer-zq878\" (UID: \"254868d4-6e40-410f-9a3d-0f994b704438\") " pod="kube-system/global-pull-secret-syncer-zq878"
Sep 18 19:56:56.595488 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.594230    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f4120871-0e94-46f5-bd6d-739f9b0383bf-cni-binary-copy\") pod \"multus-gk46f\" (UID: \"f4120871-0e94-46f5-bd6d-739f9b0383bf\") " pod="openshift-multus/multus-gk46f"
Sep 18 19:56:56.595488 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.594257    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f4120871-0e94-46f5-bd6d-739f9b0383bf-host-var-lib-cni-multus\") pod \"multus-gk46f\" (UID: \"f4120871-0e94-46f5-bd6d-739f9b0383bf\") " pod="openshift-multus/multus-gk46f"
Sep 18 19:56:56.595488 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.594282    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f4120871-0e94-46f5-bd6d-739f9b0383bf-multus-daemon-config\") pod \"multus-gk46f\" (UID: \"f4120871-0e94-46f5-bd6d-739f9b0383bf\") " pod="openshift-multus/multus-gk46f"
Sep 18 19:56:56.595488 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.594306    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48-systemd-units\") pod \"ovnkube-node-scn7p\" (UID: \"a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-scn7p"
Sep 18 19:56:56.595488 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.594304    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/56b6948b-89ff-4d48-ba8c-c0c3f63544e1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8nv28\" (UID: \"56b6948b-89ff-4d48-ba8c-c0c3f63544e1\") " pod="openshift-multus/multus-additional-cni-plugins-8nv28"
Sep 18 19:56:56.595488 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.594362    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/525400e6-6678-424f-a1ba-0896aac41264-tmp-dir\") pod \"node-resolver-t26hd\" (UID: \"525400e6-6678-424f-a1ba-0896aac41264\") " pod="openshift-dns/node-resolver-t26hd"
Sep 18 19:56:56.595488 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.594365    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/254868d4-6e40-410f-9a3d-0f994b704438-dbus\") pod \"global-pull-secret-syncer-zq878\" (UID: \"254868d4-6e40-410f-9a3d-0f994b704438\") " pod="kube-system/global-pull-secret-syncer-zq878"
Sep 18 19:56:56.595488 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.594396    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48-host-run-ovn-kubernetes\") pod \"ovnkube-node-scn7p\" (UID: \"a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-scn7p"
Sep 18 19:56:56.596144 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.594452    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48-var-lib-openvswitch\") pod \"ovnkube-node-scn7p\" (UID: \"a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-scn7p"
Sep 18 19:56:56.596144 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.594496    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-scn7p\" (UID: \"a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-scn7p"
Sep 18 19:56:56.596144 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.594553    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48-host-run-ovn-kubernetes\") pod \"ovnkube-node-scn7p\" (UID: \"a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-scn7p"
Sep 18 19:56:56.596144 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.594558    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cb6d3f0d-bc38-43aa-8810-3b14a624295d-serviceca\") pod \"node-ca-m6qj6\" (UID: \"cb6d3f0d-bc38-43aa-8810-3b14a624295d\") " pod="openshift-image-registry/node-ca-m6qj6"
Sep 18 19:56:56.596144 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.594570    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48-ovnkube-config\") pod \"ovnkube-node-scn7p\" (UID: \"a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-scn7p"
Sep 18 19:56:56.596144 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.594557    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/56b6948b-89ff-4d48-ba8c-c0c3f63544e1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8nv28\" (UID: \"56b6948b-89ff-4d48-ba8c-c0c3f63544e1\") " pod="openshift-multus/multus-additional-cni-plugins-8nv28"
Sep 18 19:56:56.596144 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.594614    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f4120871-0e94-46f5-bd6d-739f9b0383bf-host-var-lib-cni-multus\") pod \"multus-gk46f\" (UID: \"f4120871-0e94-46f5-bd6d-739f9b0383bf\") " pod="openshift-multus/multus-gk46f"
Sep 18 19:56:56.596144 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.594596    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48-systemd-units\") pod \"ovnkube-node-scn7p\" (UID: \"a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-scn7p"
Sep 18 19:56:56.596144 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.594653    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48-host-kubelet\") pod \"ovnkube-node-scn7p\" (UID: \"a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-scn7p"
Sep 18 19:56:56.596144 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.594684    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg7xq\" (UniqueName: \"kubernetes.io/projected/56b6948b-89ff-4d48-ba8c-c0c3f63544e1-kube-api-access-cg7xq\") pod \"multus-additional-cni-plugins-8nv28\" (UID: \"56b6948b-89ff-4d48-ba8c-c0c3f63544e1\") " pod="openshift-multus/multus-additional-cni-plugins-8nv28"
Sep 18 19:56:56.596144 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.594727    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48-host-kubelet\") pod \"ovnkube-node-scn7p\" (UID: \"a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-scn7p"
Sep 18 19:56:56.596144 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.594770    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48-host-slash\") pod \"ovnkube-node-scn7p\" (UID: \"a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-scn7p"
Sep 18 19:56:56.596144 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.594880    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48-host-slash\") pod \"ovnkube-node-scn7p\" (UID: \"a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-scn7p"
Sep 18 19:56:56.596144 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.594881    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48-run-openvswitch\") pod \"ovnkube-node-scn7p\" (UID: \"a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-scn7p"
Sep 18 19:56:56.596144 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.594924    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48-run-openvswitch\") pod \"ovnkube-node-scn7p\" (UID: \"a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-scn7p"
Sep 18 19:56:56.596144 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.594976    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48-ovnkube-script-lib\") pod \"ovnkube-node-scn7p\" (UID: \"a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-scn7p"
Sep 18 19:56:56.596144 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.595011    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f4120871-0e94-46f5-bd6d-739f9b0383bf-host-run-k8s-cni-cncf-io\") pod \"multus-gk46f\" (UID: \"f4120871-0e94-46f5-bd6d-739f9b0383bf\") " pod="openshift-multus/multus-gk46f"
Sep 18 19:56:56.596712 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.595037    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f4120871-0e94-46f5-bd6d-739f9b0383bf-host-var-lib-cni-bin\") pod \"multus-gk46f\" (UID: \"f4120871-0e94-46f5-bd6d-739f9b0383bf\") " pod="openshift-multus/multus-gk46f"
Sep 18 19:56:56.596712 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.595063    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f4120871-0e94-46f5-bd6d-739f9b0383bf-multus-conf-dir\") pod \"multus-gk46f\" (UID: \"f4120871-0e94-46f5-bd6d-739f9b0383bf\") " pod="openshift-multus/multus-gk46f"
Sep 18 19:56:56.596712 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.595071    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/56b6948b-89ff-4d48-ba8c-c0c3f63544e1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8nv28\" (UID: \"56b6948b-89ff-4d48-ba8c-c0c3f63544e1\") " pod="openshift-multus/multus-additional-cni-plugins-8nv28"
Sep 18 19:56:56.596712 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.595079    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f4120871-0e94-46f5-bd6d-739f9b0383bf-host-run-k8s-cni-cncf-io\") pod \"multus-gk46f\" (UID: \"f4120871-0e94-46f5-bd6d-739f9b0383bf\") " pod="openshift-multus/multus-gk46f"
Sep 18 19:56:56.596712 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.595036    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f4120871-0e94-46f5-bd6d-739f9b0383bf-multus-daemon-config\") pod \"multus-gk46f\" (UID: \"f4120871-0e94-46f5-bd6d-739f9b0383bf\") " pod="openshift-multus/multus-gk46f"
Sep 18 19:56:56.596712 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.595095    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/063d8ee5-35b4-401b-a0ff-d8b416e20175-metrics-certs\") pod \"network-metrics-daemon-2hw77\" (UID: \"063d8ee5-35b4-401b-a0ff-d8b416e20175\") " pod="openshift-multus/network-metrics-daemon-2hw77"
Sep 18 19:56:56.596712 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.595114    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f4120871-0e94-46f5-bd6d-739f9b0383bf-host-var-lib-cni-bin\") pod \"multus-gk46f\" (UID: \"f4120871-0e94-46f5-bd6d-739f9b0383bf\") " pod="openshift-multus/multus-gk46f"
Sep 18 19:56:56.596712 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.595128    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/525400e6-6678-424f-a1ba-0896aac41264-hosts-file\") pod \"node-resolver-t26hd\" (UID: \"525400e6-6678-424f-a1ba-0896aac41264\") " pod="openshift-dns/node-resolver-t26hd"
Sep 18 19:56:56.596712 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.595175    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f4120871-0e94-46f5-bd6d-739f9b0383bf-cnibin\") pod \"multus-gk46f\" (UID: \"f4120871-0e94-46f5-bd6d-739f9b0383bf\") " pod="openshift-multus/multus-gk46f"
Sep 18 19:56:56.596712 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:56:56.595183    2549 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:56:56.596712 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.595177    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/525400e6-6678-424f-a1ba-0896aac41264-hosts-file\") pod \"node-resolver-t26hd\" (UID: \"525400e6-6678-424f-a1ba-0896aac41264\") " pod="openshift-dns/node-resolver-t26hd"
Sep 18 19:56:56.596712 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.595130    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f4120871-0e94-46f5-bd6d-739f9b0383bf-multus-conf-dir\") pod \"multus-gk46f\" (UID: \"f4120871-0e94-46f5-bd6d-739f9b0383bf\") " pod="openshift-multus/multus-gk46f"
Sep 18 19:56:56.596712 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.595204    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/56b6948b-89ff-4d48-ba8c-c0c3f63544e1-cnibin\") pod \"multus-additional-cni-plugins-8nv28\" (UID: \"56b6948b-89ff-4d48-ba8c-c0c3f63544e1\") " pod="openshift-multus/multus-additional-cni-plugins-8nv28"
Sep 18 19:56:56.596712 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:56:56.595234    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/063d8ee5-35b4-401b-a0ff-d8b416e20175-metrics-certs podName:063d8ee5-35b4-401b-a0ff-d8b416e20175 nodeName:}" failed. No retries permitted until 2026-09-18 19:56:57.095217185 +0000 UTC m=+2.048958353 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/063d8ee5-35b4-401b-a0ff-d8b416e20175-metrics-certs") pod "network-metrics-daemon-2hw77" (UID: "063d8ee5-35b4-401b-a0ff-d8b416e20175") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:56:56.596712 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.595239    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/56b6948b-89ff-4d48-ba8c-c0c3f63544e1-cnibin\") pod \"multus-additional-cni-plugins-8nv28\" (UID: \"56b6948b-89ff-4d48-ba8c-c0c3f63544e1\") " pod="openshift-multus/multus-additional-cni-plugins-8nv28"
Sep 18 19:56:56.596712 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.595246    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f4120871-0e94-46f5-bd6d-739f9b0383bf-cnibin\") pod \"multus-gk46f\" (UID: \"f4120871-0e94-46f5-bd6d-739f9b0383bf\") " pod="openshift-multus/multus-gk46f"
Sep 18 19:56:56.596712 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.595273    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/56b6948b-89ff-4d48-ba8c-c0c3f63544e1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8nv28\" (UID: \"56b6948b-89ff-4d48-ba8c-c0c3f63544e1\") " pod="openshift-multus/multus-additional-cni-plugins-8nv28"
Sep 18 19:56:56.597240 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.595294    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd8qv\" (UniqueName: \"kubernetes.io/projected/ed609df8-4cad-445d-99a8-2df85ce75ba8-kube-api-access-nd8qv\") pod \"network-check-target-rktz8\" (UID: \"ed609df8-4cad-445d-99a8-2df85ce75ba8\") " pod="openshift-network-diagnostics/network-check-target-rktz8"
Sep 18 19:56:56.597240 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.595362    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48-host-cni-bin\") pod \"ovnkube-node-scn7p\" (UID: \"a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-scn7p"
Sep 18 19:56:56.597240 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.595407    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pckm\" (UniqueName: \"kubernetes.io/projected/a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48-kube-api-access-4pckm\") pod \"ovnkube-node-scn7p\" (UID: \"a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-scn7p"
Sep 18 19:56:56.597240 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.595436    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e5f6d5d2-ce02-4de2-a798-be32d7584248-socket-dir\") pod \"aws-ebs-csi-driver-node-dfp4z\" (UID: \"e5f6d5d2-ce02-4de2-a798-be32d7584248\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfp4z"
Sep 18 19:56:56.597240 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.595439    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48-host-cni-bin\") pod \"ovnkube-node-scn7p\" (UID: \"a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-scn7p"
Sep 18 19:56:56.597240 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.595462    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/56b6948b-89ff-4d48-ba8c-c0c3f63544e1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8nv28\" (UID: \"56b6948b-89ff-4d48-ba8c-c0c3f63544e1\") " pod="openshift-multus/multus-additional-cni-plugins-8nv28"
Sep 18 19:56:56.597240 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.595501    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48-ovnkube-script-lib\") pod \"ovnkube-node-scn7p\" (UID: \"a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-scn7p"
Sep 18 19:56:56.597240 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.595565    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e5f6d5d2-ce02-4de2-a798-be32d7584248-socket-dir\") pod \"aws-ebs-csi-driver-node-dfp4z\" (UID: \"e5f6d5d2-ce02-4de2-a798-be32d7584248\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfp4z"
Sep 18 19:56:56.597240 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.595663    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f4120871-0e94-46f5-bd6d-739f9b0383bf-cni-binary-copy\") pod \"multus-gk46f\" (UID: \"f4120871-0e94-46f5-bd6d-739f9b0383bf\") " pod="openshift-multus/multus-gk46f"
Sep 18 19:56:56.597240 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.596355    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48-ovn-node-metrics-cert\") pod \"ovnkube-node-scn7p\" (UID: \"a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-scn7p"
Sep 18 19:56:56.609169 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.609138    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg7xq\" (UniqueName: \"kubernetes.io/projected/56b6948b-89ff-4d48-ba8c-c0c3f63544e1-kube-api-access-cg7xq\") pod \"multus-additional-cni-plugins-8nv28\" (UID: \"56b6948b-89ff-4d48-ba8c-c0c3f63544e1\") " pod="openshift-multus/multus-additional-cni-plugins-8nv28"
Sep 18 19:56:56.609295 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:56:56.609188    2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:56:56.609295 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:56:56.609211    2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:56:56.609295 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:56:56.609225    2549 projected.go:196] Error preparing data for projected volume kube-api-access-nd8qv for pod openshift-network-diagnostics/network-check-target-rktz8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:56:56.609295 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:56:56.609278    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ed609df8-4cad-445d-99a8-2df85ce75ba8-kube-api-access-nd8qv podName:ed609df8-4cad-445d-99a8-2df85ce75ba8 nodeName:}" failed. No retries permitted until 2026-09-18 19:56:57.109263394 +0000 UTC m=+2.063004561 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-nd8qv" (UniqueName: "kubernetes.io/projected/ed609df8-4cad-445d-99a8-2df85ce75ba8-kube-api-access-nd8qv") pod "network-check-target-rktz8" (UID: "ed609df8-4cad-445d-99a8-2df85ce75ba8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:56:56.610694 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.610668    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pckm\" (UniqueName: \"kubernetes.io/projected/a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48-kube-api-access-4pckm\") pod \"ovnkube-node-scn7p\" (UID: \"a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-scn7p"
Sep 18 19:56:56.610818 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.610692    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5x6f\" (UniqueName: \"kubernetes.io/projected/f4120871-0e94-46f5-bd6d-739f9b0383bf-kube-api-access-b5x6f\") pod \"multus-gk46f\" (UID: \"f4120871-0e94-46f5-bd6d-739f9b0383bf\") " pod="openshift-multus/multus-gk46f"
Sep 18 19:56:56.610818 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.610761    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqc97\" (UniqueName: \"kubernetes.io/projected/14b84844-b7fa-4a6b-a930-4f40a8230980-kube-api-access-pqc97\") pod \"iptables-alerter-5bnj6\" (UID: \"14b84844-b7fa-4a6b-a930-4f40a8230980\") " pod="openshift-network-operator/iptables-alerter-5bnj6"
Sep 18 19:56:56.611053 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.611026    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5md6\" (UniqueName: \"kubernetes.io/projected/e5f6d5d2-ce02-4de2-a798-be32d7584248-kube-api-access-g5md6\") pod \"aws-ebs-csi-driver-node-dfp4z\" (UID: \"e5f6d5d2-ce02-4de2-a798-be32d7584248\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfp4z"
Sep 18 19:56:56.611517 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.611496    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbszj\" (UniqueName: \"kubernetes.io/projected/cb6d3f0d-bc38-43aa-8810-3b14a624295d-kube-api-access-pbszj\") pod \"node-ca-m6qj6\" (UID: \"cb6d3f0d-bc38-43aa-8810-3b14a624295d\") " pod="openshift-image-registry/node-ca-m6qj6"
Sep 18 19:56:56.612417 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.612400    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9b27\" (UniqueName: \"kubernetes.io/projected/063d8ee5-35b4-401b-a0ff-d8b416e20175-kube-api-access-n9b27\") pod \"network-metrics-daemon-2hw77\" (UID: \"063d8ee5-35b4-401b-a0ff-d8b416e20175\") " pod="openshift-multus/network-metrics-daemon-2hw77"
Sep 18 19:56:56.612910 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.612896    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn2pt\" (UniqueName: \"kubernetes.io/projected/525400e6-6678-424f-a1ba-0896aac41264-kube-api-access-hn2pt\") pod \"node-resolver-t26hd\" (UID: \"525400e6-6678-424f-a1ba-0896aac41264\") " pod="openshift-dns/node-resolver-t26hd"
Sep 18 19:56:56.708182 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.708092    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-tlqj8"
Sep 18 19:56:56.713928 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.713905    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ks678"
Sep 18 19:56:56.719722 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:56.719693    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e178884_5bfb_46cb_9305_6170284cb243.slice/crio-d10f3510758989ceee47682b1dcdd08b0202403b5ef5e2f84a67ab2e8f9ae387 WatchSource:0}: Error finding container d10f3510758989ceee47682b1dcdd08b0202403b5ef5e2f84a67ab2e8f9ae387: Status 404 returned error can't find the container with id d10f3510758989ceee47682b1dcdd08b0202403b5ef5e2f84a67ab2e8f9ae387
Sep 18 19:56:56.723508 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:56.723482    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfd97804_116b_4355_bf8f_adb95dc143af.slice/crio-53ff7ece02de4150627e4696648279d4dc61352fe3555037be4415706fbf8ea2 WatchSource:0}: Error finding container 53ff7ece02de4150627e4696648279d4dc61352fe3555037be4415706fbf8ea2: Status 404 returned error can't find the container with id 53ff7ece02de4150627e4696648279d4dc61352fe3555037be4415706fbf8ea2
Sep 18 19:56:56.740566 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.740542    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-t26hd"
Sep 18 19:56:56.749570 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:56.749547    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod525400e6_6678_424f_a1ba_0896aac41264.slice/crio-b52eb4c87fab351b6ca1e06a050ce85c26906a7e15605e728a211d117f9c8a1a WatchSource:0}: Error finding container b52eb4c87fab351b6ca1e06a050ce85c26906a7e15605e728a211d117f9c8a1a: Status 404 returned error can't find the container with id b52eb4c87fab351b6ca1e06a050ce85c26906a7e15605e728a211d117f9c8a1a
Sep 18 19:56:56.756615 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.756599    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8nv28"
Sep 18 19:56:56.764905 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:56.764884    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56b6948b_89ff_4d48_ba8c_c0c3f63544e1.slice/crio-428a4c8e158f18ca30e8b49fc9118b3c8b72a6518fd904e9941445bacfa515ee WatchSource:0}: Error finding container 428a4c8e158f18ca30e8b49fc9118b3c8b72a6518fd904e9941445bacfa515ee: Status 404 returned error can't find the container with id 428a4c8e158f18ca30e8b49fc9118b3c8b72a6518fd904e9941445bacfa515ee
Sep 18 19:56:56.775827 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.775808    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-scn7p"
Sep 18 19:56:56.784990 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:56.784967    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7a8f2b1_a3d0_4d9a_9099_fb05824d3e48.slice/crio-e20e088937e4ebdbe903bad74bf6aa564fa668f18c985b3e4b7307e33a743645 WatchSource:0}: Error finding container e20e088937e4ebdbe903bad74bf6aa564fa668f18c985b3e4b7307e33a743645: Status 404 returned error can't find the container with id e20e088937e4ebdbe903bad74bf6aa564fa668f18c985b3e4b7307e33a743645
Sep 18 19:56:56.790167 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.790146    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfp4z"
Sep 18 19:56:56.798009 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:56.797989    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5f6d5d2_ce02_4de2_a798_be32d7584248.slice/crio-9624448983b7cae886a649d38544fda9d3280242236502d960f0d6471b65fb53 WatchSource:0}: Error finding container 9624448983b7cae886a649d38544fda9d3280242236502d960f0d6471b65fb53: Status 404 returned error can't find the container with id 9624448983b7cae886a649d38544fda9d3280242236502d960f0d6471b65fb53
Sep 18 19:56:56.807694 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.807676    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-m6qj6"
Sep 18 19:56:56.816173 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:56.816144    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb6d3f0d_bc38_43aa_8810_3b14a624295d.slice/crio-8fab3fa9313a8b9ca3906bf900938eb561adacc16d09da383f7c95901c474691 WatchSource:0}: Error finding container 8fab3fa9313a8b9ca3906bf900938eb561adacc16d09da383f7c95901c474691: Status 404 returned error can't find the container with id 8fab3fa9313a8b9ca3906bf900938eb561adacc16d09da383f7c95901c474691
Sep 18 19:56:56.823421 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.823403    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gk46f"
Sep 18 19:56:56.828392 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:56.828371    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5bnj6"
Sep 18 19:56:56.831903 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:56.831878    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4120871_0e94_46f5_bd6d_739f9b0383bf.slice/crio-539c6e381226e86ea42a376889cc6ec51e06e6d3d51f06ea793f5d872731e39a WatchSource:0}: Error finding container 539c6e381226e86ea42a376889cc6ec51e06e6d3d51f06ea793f5d872731e39a: Status 404 returned error can't find the container with id 539c6e381226e86ea42a376889cc6ec51e06e6d3d51f06ea793f5d872731e39a
Sep 18 19:56:56.837701 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:56:56.837681    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14b84844_b7fa_4a6b_a930_4f40a8230980.slice/crio-bdaf60ea6f2839d6bf153b01ac8f63e39dc1ce0da2feb4928600f70de784201e WatchSource:0}: Error finding container bdaf60ea6f2839d6bf153b01ac8f63e39dc1ce0da2feb4928600f70de784201e: Status 404 returned error can't find the container with id bdaf60ea6f2839d6bf153b01ac8f63e39dc1ce0da2feb4928600f70de784201e
Sep 18 19:56:57.100224 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:57.100187    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/254868d4-6e40-410f-9a3d-0f994b704438-original-pull-secret\") pod \"global-pull-secret-syncer-zq878\" (UID: \"254868d4-6e40-410f-9a3d-0f994b704438\") " pod="kube-system/global-pull-secret-syncer-zq878"
Sep 18 19:56:57.100449 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:57.100276    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/063d8ee5-35b4-401b-a0ff-d8b416e20175-metrics-certs\") pod \"network-metrics-daemon-2hw77\" (UID: \"063d8ee5-35b4-401b-a0ff-d8b416e20175\") " pod="openshift-multus/network-metrics-daemon-2hw77"
Sep 18 19:56:57.100449 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:56:57.100440    2549 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:56:57.100578 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:56:57.100504    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/063d8ee5-35b4-401b-a0ff-d8b416e20175-metrics-certs podName:063d8ee5-35b4-401b-a0ff-d8b416e20175 nodeName:}" failed. No retries permitted until 2026-09-18 19:56:58.100483442 +0000 UTC m=+3.054224596 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/063d8ee5-35b4-401b-a0ff-d8b416e20175-metrics-certs") pod "network-metrics-daemon-2hw77" (UID: "063d8ee5-35b4-401b-a0ff-d8b416e20175") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:56:57.101007 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:56:57.100916    2549 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:56:57.101007 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:56:57.100974    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/254868d4-6e40-410f-9a3d-0f994b704438-original-pull-secret podName:254868d4-6e40-410f-9a3d-0f994b704438 nodeName:}" failed. No retries permitted until 2026-09-18 19:56:58.100957133 +0000 UTC m=+3.054698289 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/254868d4-6e40-410f-9a3d-0f994b704438-original-pull-secret") pod "global-pull-secret-syncer-zq878" (UID: "254868d4-6e40-410f-9a3d-0f994b704438") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:56:57.200878 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:57.200843    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd8qv\" (UniqueName: \"kubernetes.io/projected/ed609df8-4cad-445d-99a8-2df85ce75ba8-kube-api-access-nd8qv\") pod \"network-check-target-rktz8\" (UID: \"ed609df8-4cad-445d-99a8-2df85ce75ba8\") " pod="openshift-network-diagnostics/network-check-target-rktz8"
Sep 18 19:56:57.201045 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:56:57.201010    2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:56:57.201045 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:56:57.201032    2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:56:57.201045 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:56:57.201045    2549 projected.go:196] Error preparing data for projected volume kube-api-access-nd8qv for pod openshift-network-diagnostics/network-check-target-rktz8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:56:57.201234 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:56:57.201107    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ed609df8-4cad-445d-99a8-2df85ce75ba8-kube-api-access-nd8qv podName:ed609df8-4cad-445d-99a8-2df85ce75ba8 nodeName:}" failed. No retries permitted until 2026-09-18 19:56:58.201088308 +0000 UTC m=+3.154829462 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-nd8qv" (UniqueName: "kubernetes.io/projected/ed609df8-4cad-445d-99a8-2df85ce75ba8-kube-api-access-nd8qv") pod "network-check-target-rktz8" (UID: "ed609df8-4cad-445d-99a8-2df85ce75ba8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:56:57.461191 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:57.461063    2549 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 19:51:57 +0000 UTC" deadline="2028-05-14 11:46:57.071254184 +0000 UTC"
Sep 18 19:56:57.461191 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:57.461136    2549 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14487h49m59.610122322s"
Sep 18 19:56:57.468475 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:57.468272    2549 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:56:57.515992 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:57.515738    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rktz8"
Sep 18 19:56:57.516165 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:56:57.516100    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rktz8" podUID="ed609df8-4cad-445d-99a8-2df85ce75ba8"
Sep 18 19:56:57.517249 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:57.516652    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zq878"
Sep 18 19:56:57.517249 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:56:57.516774    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zq878" podUID="254868d4-6e40-410f-9a3d-0f994b704438"
Sep 18 19:56:57.540066 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:57.539994    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5bnj6" event={"ID":"14b84844-b7fa-4a6b-a930-4f40a8230980","Type":"ContainerStarted","Data":"bdaf60ea6f2839d6bf153b01ac8f63e39dc1ce0da2feb4928600f70de784201e"}
Sep 18 19:56:57.564215 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:57.564051    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfp4z" event={"ID":"e5f6d5d2-ce02-4de2-a798-be32d7584248","Type":"ContainerStarted","Data":"9624448983b7cae886a649d38544fda9d3280242236502d960f0d6471b65fb53"}
Sep 18 19:56:57.573234 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:57.572866    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-scn7p" event={"ID":"a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48","Type":"ContainerStarted","Data":"e20e088937e4ebdbe903bad74bf6aa564fa668f18c985b3e4b7307e33a743645"}
Sep 18 19:56:57.608973 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:57.608933    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8nv28" event={"ID":"56b6948b-89ff-4d48-ba8c-c0c3f63544e1","Type":"ContainerStarted","Data":"428a4c8e158f18ca30e8b49fc9118b3c8b72a6518fd904e9941445bacfa515ee"}
Sep 18 19:56:57.620742 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:57.617975    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ks678" event={"ID":"bfd97804-116b-4355-bf8f-adb95dc143af","Type":"ContainerStarted","Data":"53ff7ece02de4150627e4696648279d4dc61352fe3555037be4415706fbf8ea2"}
Sep 18 19:56:57.629548 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:57.629513    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-tlqj8" event={"ID":"2e178884-5bfb-46cb-9305-6170284cb243","Type":"ContainerStarted","Data":"d10f3510758989ceee47682b1dcdd08b0202403b5ef5e2f84a67ab2e8f9ae387"}
Sep 18 19:56:57.647584 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:57.647551    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gk46f" event={"ID":"f4120871-0e94-46f5-bd6d-739f9b0383bf","Type":"ContainerStarted","Data":"539c6e381226e86ea42a376889cc6ec51e06e6d3d51f06ea793f5d872731e39a"}
Sep 18 19:56:57.660416 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:57.660379    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-m6qj6" event={"ID":"cb6d3f0d-bc38-43aa-8810-3b14a624295d","Type":"ContainerStarted","Data":"8fab3fa9313a8b9ca3906bf900938eb561adacc16d09da383f7c95901c474691"}
Sep 18 19:56:57.674250 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:57.674217    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-t26hd" event={"ID":"525400e6-6678-424f-a1ba-0896aac41264","Type":"ContainerStarted","Data":"b52eb4c87fab351b6ca1e06a050ce85c26906a7e15605e728a211d117f9c8a1a"}
Sep 18 19:56:58.107739 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:58.107684    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/254868d4-6e40-410f-9a3d-0f994b704438-original-pull-secret\") pod \"global-pull-secret-syncer-zq878\" (UID: \"254868d4-6e40-410f-9a3d-0f994b704438\") " pod="kube-system/global-pull-secret-syncer-zq878"
Sep 18 19:56:58.107934 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:58.107791    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/063d8ee5-35b4-401b-a0ff-d8b416e20175-metrics-certs\") pod \"network-metrics-daemon-2hw77\" (UID: \"063d8ee5-35b4-401b-a0ff-d8b416e20175\") " pod="openshift-multus/network-metrics-daemon-2hw77"
Sep 18 19:56:58.108003 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:56:58.107946    2549 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:56:58.108063 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:56:58.108012    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/063d8ee5-35b4-401b-a0ff-d8b416e20175-metrics-certs podName:063d8ee5-35b4-401b-a0ff-d8b416e20175 nodeName:}" failed. No retries permitted until 2026-09-18 19:57:00.107992377 +0000 UTC m=+5.061733547 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/063d8ee5-35b4-401b-a0ff-d8b416e20175-metrics-certs") pod "network-metrics-daemon-2hw77" (UID: "063d8ee5-35b4-401b-a0ff-d8b416e20175") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:56:58.108505 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:56:58.108441    2549 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:56:58.108598 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:56:58.108510    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/254868d4-6e40-410f-9a3d-0f994b704438-original-pull-secret podName:254868d4-6e40-410f-9a3d-0f994b704438 nodeName:}" failed. No retries permitted until 2026-09-18 19:57:00.108493438 +0000 UTC m=+5.062234591 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/254868d4-6e40-410f-9a3d-0f994b704438-original-pull-secret") pod "global-pull-secret-syncer-zq878" (UID: "254868d4-6e40-410f-9a3d-0f994b704438") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:56:58.208236 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:58.208166    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd8qv\" (UniqueName: \"kubernetes.io/projected/ed609df8-4cad-445d-99a8-2df85ce75ba8-kube-api-access-nd8qv\") pod \"network-check-target-rktz8\" (UID: \"ed609df8-4cad-445d-99a8-2df85ce75ba8\") " pod="openshift-network-diagnostics/network-check-target-rktz8"
Sep 18 19:56:58.208424 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:56:58.208321    2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:56:58.208424 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:56:58.208351    2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:56:58.208424 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:56:58.208364    2549 projected.go:196] Error preparing data for projected volume kube-api-access-nd8qv for pod openshift-network-diagnostics/network-check-target-rktz8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:56:58.208622 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:56:58.208441    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ed609df8-4cad-445d-99a8-2df85ce75ba8-kube-api-access-nd8qv podName:ed609df8-4cad-445d-99a8-2df85ce75ba8 nodeName:}" failed. No retries permitted until 2026-09-18 19:57:00.208406043 +0000 UTC m=+5.162147197 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-nd8qv" (UniqueName: "kubernetes.io/projected/ed609df8-4cad-445d-99a8-2df85ce75ba8-kube-api-access-nd8qv") pod "network-check-target-rktz8" (UID: "ed609df8-4cad-445d-99a8-2df85ce75ba8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:56:58.461607 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:58.461446    2549 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 19:51:57 +0000 UTC" deadline="2028-07-04 18:26:46.443642474 +0000 UTC"
Sep 18 19:56:58.461607 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:58.461496    2549 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15718h29m47.982152143s"
Sep 18 19:56:58.516076 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:58.515577    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2hw77"
Sep 18 19:56:58.516076 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:56:58.515713    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2hw77" podUID="063d8ee5-35b4-401b-a0ff-d8b416e20175"
Sep 18 19:56:59.389307 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:59.389263    2549 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:56:59.517123 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:59.516590    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rktz8"
Sep 18 19:56:59.517123 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:56:59.516732    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rktz8" podUID="ed609df8-4cad-445d-99a8-2df85ce75ba8"
Sep 18 19:56:59.517123 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:56:59.516786    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zq878"
Sep 18 19:56:59.517123 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:56:59.516919    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zq878" podUID="254868d4-6e40-410f-9a3d-0f994b704438"
Sep 18 19:57:00.123235 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:00.123194    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/254868d4-6e40-410f-9a3d-0f994b704438-original-pull-secret\") pod \"global-pull-secret-syncer-zq878\" (UID: \"254868d4-6e40-410f-9a3d-0f994b704438\") " pod="kube-system/global-pull-secret-syncer-zq878"
Sep 18 19:57:00.123420 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:00.123283    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/063d8ee5-35b4-401b-a0ff-d8b416e20175-metrics-certs\") pod \"network-metrics-daemon-2hw77\" (UID: \"063d8ee5-35b4-401b-a0ff-d8b416e20175\") " pod="openshift-multus/network-metrics-daemon-2hw77"
Sep 18 19:57:00.123508 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:57:00.123463    2549 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:57:00.123563 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:57:00.123531    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/063d8ee5-35b4-401b-a0ff-d8b416e20175-metrics-certs podName:063d8ee5-35b4-401b-a0ff-d8b416e20175 nodeName:}" failed. No retries permitted until 2026-09-18 19:57:04.123510013 +0000 UTC m=+9.077251209 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/063d8ee5-35b4-401b-a0ff-d8b416e20175-metrics-certs") pod "network-metrics-daemon-2hw77" (UID: "063d8ee5-35b4-401b-a0ff-d8b416e20175") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:57:00.123962 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:57:00.123943    2549 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:57:00.124042 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:57:00.123998    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/254868d4-6e40-410f-9a3d-0f994b704438-original-pull-secret podName:254868d4-6e40-410f-9a3d-0f994b704438 nodeName:}" failed. No retries permitted until 2026-09-18 19:57:04.123982735 +0000 UTC m=+9.077723899 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/254868d4-6e40-410f-9a3d-0f994b704438-original-pull-secret") pod "global-pull-secret-syncer-zq878" (UID: "254868d4-6e40-410f-9a3d-0f994b704438") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:57:00.224070 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:00.223987    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd8qv\" (UniqueName: \"kubernetes.io/projected/ed609df8-4cad-445d-99a8-2df85ce75ba8-kube-api-access-nd8qv\") pod \"network-check-target-rktz8\" (UID: \"ed609df8-4cad-445d-99a8-2df85ce75ba8\") " pod="openshift-network-diagnostics/network-check-target-rktz8"
Sep 18 19:57:00.224246 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:57:00.224191    2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:57:00.224246 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:57:00.224212    2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:57:00.224246 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:57:00.224225    2549 projected.go:196] Error preparing data for projected volume kube-api-access-nd8qv for pod openshift-network-diagnostics/network-check-target-rktz8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:57:00.224412 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:57:00.224286    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ed609df8-4cad-445d-99a8-2df85ce75ba8-kube-api-access-nd8qv podName:ed609df8-4cad-445d-99a8-2df85ce75ba8 nodeName:}" failed. No retries permitted until 2026-09-18 19:57:04.224264457 +0000 UTC m=+9.178005623 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-nd8qv" (UniqueName: "kubernetes.io/projected/ed609df8-4cad-445d-99a8-2df85ce75ba8-kube-api-access-nd8qv") pod "network-check-target-rktz8" (UID: "ed609df8-4cad-445d-99a8-2df85ce75ba8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:57:00.515764 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:00.515507    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2hw77"
Sep 18 19:57:00.515764 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:57:00.515663    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2hw77" podUID="063d8ee5-35b4-401b-a0ff-d8b416e20175"
Sep 18 19:57:01.515551 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:01.515472    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zq878"
Sep 18 19:57:01.516248 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:57:01.515623    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zq878" podUID="254868d4-6e40-410f-9a3d-0f994b704438"
Sep 18 19:57:01.516248 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:01.515467    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rktz8"
Sep 18 19:57:01.516248 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:57:01.516107    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rktz8" podUID="ed609df8-4cad-445d-99a8-2df85ce75ba8"
Sep 18 19:57:02.515539 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:02.515491    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2hw77"
Sep 18 19:57:02.515713 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:57:02.515683    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2hw77" podUID="063d8ee5-35b4-401b-a0ff-d8b416e20175"
Sep 18 19:57:03.515081 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:03.514958    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rktz8"
Sep 18 19:57:03.515495 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:57:03.515109    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rktz8" podUID="ed609df8-4cad-445d-99a8-2df85ce75ba8"
Sep 18 19:57:03.515495 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:03.515167    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zq878"
Sep 18 19:57:03.515495 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:57:03.515294    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zq878" podUID="254868d4-6e40-410f-9a3d-0f994b704438"
Sep 18 19:57:04.155673 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:04.155631    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/063d8ee5-35b4-401b-a0ff-d8b416e20175-metrics-certs\") pod \"network-metrics-daemon-2hw77\" (UID: \"063d8ee5-35b4-401b-a0ff-d8b416e20175\") " pod="openshift-multus/network-metrics-daemon-2hw77"
Sep 18 19:57:04.156185 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:04.155717    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/254868d4-6e40-410f-9a3d-0f994b704438-original-pull-secret\") pod \"global-pull-secret-syncer-zq878\" (UID: \"254868d4-6e40-410f-9a3d-0f994b704438\") " pod="kube-system/global-pull-secret-syncer-zq878"
Sep 18 19:57:04.156185 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:57:04.155805    2549 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:57:04.156185 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:57:04.155833    2549 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:57:04.156185 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:57:04.155892    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/063d8ee5-35b4-401b-a0ff-d8b416e20175-metrics-certs podName:063d8ee5-35b4-401b-a0ff-d8b416e20175 nodeName:}" failed. No retries permitted until 2026-09-18 19:57:12.155870036 +0000 UTC m=+17.109611206 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/063d8ee5-35b4-401b-a0ff-d8b416e20175-metrics-certs") pod "network-metrics-daemon-2hw77" (UID: "063d8ee5-35b4-401b-a0ff-d8b416e20175") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:57:04.156185 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:57:04.155913    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/254868d4-6e40-410f-9a3d-0f994b704438-original-pull-secret podName:254868d4-6e40-410f-9a3d-0f994b704438 nodeName:}" failed. No retries permitted until 2026-09-18 19:57:12.155902985 +0000 UTC m=+17.109644136 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/254868d4-6e40-410f-9a3d-0f994b704438-original-pull-secret") pod "global-pull-secret-syncer-zq878" (UID: "254868d4-6e40-410f-9a3d-0f994b704438") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:57:04.256892 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:04.256856    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd8qv\" (UniqueName: \"kubernetes.io/projected/ed609df8-4cad-445d-99a8-2df85ce75ba8-kube-api-access-nd8qv\") pod \"network-check-target-rktz8\" (UID: \"ed609df8-4cad-445d-99a8-2df85ce75ba8\") " pod="openshift-network-diagnostics/network-check-target-rktz8"
Sep 18 19:57:04.257063 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:57:04.257017    2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:57:04.257063 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:57:04.257037    2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:57:04.257063 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:57:04.257050    2549 projected.go:196] Error preparing data for projected volume kube-api-access-nd8qv for pod openshift-network-diagnostics/network-check-target-rktz8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:57:04.257246 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:57:04.257112    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ed609df8-4cad-445d-99a8-2df85ce75ba8-kube-api-access-nd8qv podName:ed609df8-4cad-445d-99a8-2df85ce75ba8 nodeName:}" failed. No retries permitted until 2026-09-18 19:57:12.2570917 +0000 UTC m=+17.210832873 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-nd8qv" (UniqueName: "kubernetes.io/projected/ed609df8-4cad-445d-99a8-2df85ce75ba8-kube-api-access-nd8qv") pod "network-check-target-rktz8" (UID: "ed609df8-4cad-445d-99a8-2df85ce75ba8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:57:04.515427 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:04.515293    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2hw77"
Sep 18 19:57:04.515605 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:57:04.515464    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2hw77" podUID="063d8ee5-35b4-401b-a0ff-d8b416e20175"
Sep 18 19:57:05.516816 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:05.516206    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rktz8"
Sep 18 19:57:05.516816 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:57:05.516356    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rktz8" podUID="ed609df8-4cad-445d-99a8-2df85ce75ba8"
Sep 18 19:57:05.516816 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:05.516201    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zq878"
Sep 18 19:57:05.516816 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:57:05.516515    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zq878" podUID="254868d4-6e40-410f-9a3d-0f994b704438"
Sep 18 19:57:06.515176 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:06.515140    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2hw77"
Sep 18 19:57:06.515356 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:57:06.515266    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2hw77" podUID="063d8ee5-35b4-401b-a0ff-d8b416e20175"
Sep 18 19:57:07.515284 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:07.515245    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rktz8"
Sep 18 19:57:07.515777 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:57:07.515365    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rktz8" podUID="ed609df8-4cad-445d-99a8-2df85ce75ba8"
Sep 18 19:57:07.515777 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:07.515410    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zq878"
Sep 18 19:57:07.515777 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:57:07.515491    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zq878" podUID="254868d4-6e40-410f-9a3d-0f994b704438"
Sep 18 19:57:08.515218 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:08.515187    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2hw77"
Sep 18 19:57:08.515362 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:57:08.515290    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2hw77" podUID="063d8ee5-35b4-401b-a0ff-d8b416e20175"
Sep 18 19:57:09.515444 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:09.515409    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rktz8"
Sep 18 19:57:09.515890 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:57:09.515529    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rktz8" podUID="ed609df8-4cad-445d-99a8-2df85ce75ba8"
Sep 18 19:57:09.515890 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:09.515552    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zq878"
Sep 18 19:57:09.515890 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:57:09.515697    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zq878" podUID="254868d4-6e40-410f-9a3d-0f994b704438"
Sep 18 19:57:10.514644 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:10.514613    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2hw77"
Sep 18 19:57:10.514822 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:57:10.514732    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2hw77" podUID="063d8ee5-35b4-401b-a0ff-d8b416e20175"
Sep 18 19:57:11.515522 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:11.515480    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zq878"
Sep 18 19:57:11.516079 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:11.515480    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rktz8"
Sep 18 19:57:11.516079 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:57:11.515611    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zq878" podUID="254868d4-6e40-410f-9a3d-0f994b704438"
Sep 18 19:57:11.516079 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:57:11.515700    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rktz8" podUID="ed609df8-4cad-445d-99a8-2df85ce75ba8"
Sep 18 19:57:12.216918 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:12.216886    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/063d8ee5-35b4-401b-a0ff-d8b416e20175-metrics-certs\") pod \"network-metrics-daemon-2hw77\" (UID: \"063d8ee5-35b4-401b-a0ff-d8b416e20175\") " pod="openshift-multus/network-metrics-daemon-2hw77"
Sep 18 19:57:12.217167 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:12.216935    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/254868d4-6e40-410f-9a3d-0f994b704438-original-pull-secret\") pod \"global-pull-secret-syncer-zq878\" (UID: \"254868d4-6e40-410f-9a3d-0f994b704438\") " pod="kube-system/global-pull-secret-syncer-zq878"
Sep 18 19:57:12.217167 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:57:12.217034    2549 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:57:12.217167 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:57:12.217043    2549 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:57:12.217167 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:57:12.217090    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/254868d4-6e40-410f-9a3d-0f994b704438-original-pull-secret podName:254868d4-6e40-410f-9a3d-0f994b704438 nodeName:}" failed. No retries permitted until 2026-09-18 19:57:28.217074894 +0000 UTC m=+33.170816048 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/254868d4-6e40-410f-9a3d-0f994b704438-original-pull-secret") pod "global-pull-secret-syncer-zq878" (UID: "254868d4-6e40-410f-9a3d-0f994b704438") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:57:12.217167 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:57:12.217106    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/063d8ee5-35b4-401b-a0ff-d8b416e20175-metrics-certs podName:063d8ee5-35b4-401b-a0ff-d8b416e20175 nodeName:}" failed. No retries permitted until 2026-09-18 19:57:28.217098109 +0000 UTC m=+33.170839256 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/063d8ee5-35b4-401b-a0ff-d8b416e20175-metrics-certs") pod "network-metrics-daemon-2hw77" (UID: "063d8ee5-35b4-401b-a0ff-d8b416e20175") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:57:12.317422 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:12.317370    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd8qv\" (UniqueName: \"kubernetes.io/projected/ed609df8-4cad-445d-99a8-2df85ce75ba8-kube-api-access-nd8qv\") pod \"network-check-target-rktz8\" (UID: \"ed609df8-4cad-445d-99a8-2df85ce75ba8\") " pod="openshift-network-diagnostics/network-check-target-rktz8"
Sep 18 19:57:12.317608 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:57:12.317552    2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:57:12.317608 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:57:12.317574    2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:57:12.317608 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:57:12.317589    2549 projected.go:196] Error preparing data for projected volume kube-api-access-nd8qv for pod openshift-network-diagnostics/network-check-target-rktz8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:57:12.317759 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:57:12.317650    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ed609df8-4cad-445d-99a8-2df85ce75ba8-kube-api-access-nd8qv podName:ed609df8-4cad-445d-99a8-2df85ce75ba8 nodeName:}" failed. No retries permitted until 2026-09-18 19:57:28.317635998 +0000 UTC m=+33.271377144 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-nd8qv" (UniqueName: "kubernetes.io/projected/ed609df8-4cad-445d-99a8-2df85ce75ba8-kube-api-access-nd8qv") pod "network-check-target-rktz8" (UID: "ed609df8-4cad-445d-99a8-2df85ce75ba8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:57:12.515546 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:12.515457    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2hw77"
Sep 18 19:57:12.515954 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:57:12.515600    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2hw77" podUID="063d8ee5-35b4-401b-a0ff-d8b416e20175"
Sep 18 19:57:13.515491 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:13.515450    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rktz8"
Sep 18 19:57:13.515681 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:13.515450    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zq878"
Sep 18 19:57:13.515681 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:57:13.515581    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rktz8" podUID="ed609df8-4cad-445d-99a8-2df85ce75ba8"
Sep 18 19:57:13.515681 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:57:13.515633    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zq878" podUID="254868d4-6e40-410f-9a3d-0f994b704438"
Sep 18 19:57:14.515447 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:14.515417    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2hw77"
Sep 18 19:57:14.515611 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:57:14.515536    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2hw77" podUID="063d8ee5-35b4-401b-a0ff-d8b416e20175"
Sep 18 19:57:15.515704 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:15.515639    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zq878"
Sep 18 19:57:15.516152 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:57:15.515760    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zq878" podUID="254868d4-6e40-410f-9a3d-0f994b704438"
Sep 18 19:57:15.516152 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:15.515798    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rktz8"
Sep 18 19:57:15.516152 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:57:15.515869    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rktz8" podUID="ed609df8-4cad-445d-99a8-2df85ce75ba8"
Sep 18 19:57:16.514780 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:16.514600    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2hw77"
Sep 18 19:57:16.514889 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:57:16.514858    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2hw77" podUID="063d8ee5-35b4-401b-a0ff-d8b416e20175"
Sep 18 19:57:16.722927 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:16.722904    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-scn7p_a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48/ovn-acl-logging/0.log"
Sep 18 19:57:16.723578 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:16.723312    2549 generic.go:356] "Generic (PLEG): container finished" podID="a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48" containerID="c431d7de95774eaee6fc7909569de6bfc80e47b17fa712577c3c696002948700" exitCode=1
Sep 18 19:57:16.723578 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:16.723397    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-scn7p" event={"ID":"a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48","Type":"ContainerStarted","Data":"9ffcb4330de44fd2bd1e054c06aace8d1eebec6f013820c301c51ee11e318799"}
Sep 18 19:57:16.723578 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:16.723481    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-scn7p" event={"ID":"a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48","Type":"ContainerStarted","Data":"57bbbee5a7d44f6124f112179ed3f872a288e5d527fd6ddfdf354a71b6aafdcc"}
Sep 18 19:57:16.723578 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:16.723534    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-scn7p" event={"ID":"a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48","Type":"ContainerStarted","Data":"b16c9560f3a332d8e620218c99509aabee36f5ee4b9291363b6fc76856c023b1"}
Sep 18 19:57:16.723714 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:16.723579    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-scn7p" event={"ID":"a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48","Type":"ContainerDied","Data":"c431d7de95774eaee6fc7909569de6bfc80e47b17fa712577c3c696002948700"}
Sep 18 19:57:16.723714 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:16.723632    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-scn7p" event={"ID":"a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48","Type":"ContainerStarted","Data":"1d6abcb9332b2b804d13c7f58942059b5489f6a5051fa502059a98e7091fe9c9"}
Sep 18 19:57:16.725482 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:16.725454    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ks678" event={"ID":"bfd97804-116b-4355-bf8f-adb95dc143af","Type":"ContainerStarted","Data":"bc0010056412646bd9d02dd3df6243ba8a8dac9f5f7845a2a49ba46f212cd2a4"}
Sep 18 19:57:16.727365 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:16.727345    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-178.ec2.internal" event={"ID":"2748dceba5a1bbaba05c6bb022d69842","Type":"ContainerStarted","Data":"eb24be7da4b0febca2a66ad41cab3e84716caf720155fd1a3ce0f5f9b2c41993"}
Sep 18 19:57:16.729001 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:16.728975    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gk46f" event={"ID":"f4120871-0e94-46f5-bd6d-739f9b0383bf","Type":"ContainerStarted","Data":"5b47e84766d436389c4b5e1d9f56d146caa7c2ccc603c5e90f12c0d1964c4053"}
Sep 18 19:57:16.772798 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:16.772754    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-gk46f" podStartSLOduration=1.360978091 podStartE2EDuration="20.772739282s" totalImagesPullingTime="19.411761191s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:56:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:57:16.772427556 +0000 UTC m=+21.726168724" watchObservedRunningTime="2026-09-18 19:57:16.772739282 +0000 UTC m=+21.726480449"
Sep 18 19:57:16.772948 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:16.772930    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-ks678" podStartSLOduration=1.374369585 podStartE2EDuration="20.772926672s" totalImagesPullingTime="19.398557087s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:56:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:57:16.743504337 +0000 UTC m=+21.697245504" watchObservedRunningTime="2026-09-18 19:57:16.772926672 +0000 UTC m=+21.726667839"
Sep 18 19:57:16.805032 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:16.804732    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-178.ec2.internal" podStartSLOduration=19.804713042 podStartE2EDuration="19.804713042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:56:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:57:16.803920776 +0000 UTC m=+21.757661944" watchObservedRunningTime="2026-09-18 19:57:16.804713042 +0000 UTC m=+21.758454211"
Sep 18 19:57:17.518319 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:17.518243    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zq878"
Sep 18 19:57:17.518493 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:17.518251    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rktz8"
Sep 18 19:57:17.518493 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:57:17.518364    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zq878" podUID="254868d4-6e40-410f-9a3d-0f994b704438"
Sep 18 19:57:17.518493 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:57:17.518478    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rktz8" podUID="ed609df8-4cad-445d-99a8-2df85ce75ba8"
Sep 18 19:57:17.733027 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:17.732992    2549 generic.go:356] "Generic (PLEG): container finished" podID="e511ba71d033b03f5d71c14df866e732" containerID="8ef480156328d16f2db8f1b1233dcbd17231c13ba5c99c2ad9c605968df6115f" exitCode=0
Sep 18 19:57:17.733631 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:17.733070    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-178.ec2.internal" event={"ID":"e511ba71d033b03f5d71c14df866e732","Type":"ContainerDied","Data":"8ef480156328d16f2db8f1b1233dcbd17231c13ba5c99c2ad9c605968df6115f"}
Sep 18 19:57:17.734614 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:17.734591    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-m6qj6" event={"ID":"cb6d3f0d-bc38-43aa-8810-3b14a624295d","Type":"ContainerStarted","Data":"a07d0bd6343919776115240808b0dc427879c8ce5c48b6b5a619fd826fa02274"}
Sep 18 19:57:17.736028 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:17.736003    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-t26hd" event={"ID":"525400e6-6678-424f-a1ba-0896aac41264","Type":"ContainerStarted","Data":"004a847082879f35804a746fd0849a6bb58fd5b9498e3d24c164b5be814e1158"}
Sep 18 19:57:17.737397 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:17.737365    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5bnj6" event={"ID":"14b84844-b7fa-4a6b-a930-4f40a8230980","Type":"ContainerStarted","Data":"af14a092ae77ccd5d907213e95f2d6c4a125453b3afd3999ce2e17cc4e78590a"}
Sep 18 19:57:17.738784 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:17.738765    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfp4z" event={"ID":"e5f6d5d2-ce02-4de2-a798-be32d7584248","Type":"ContainerStarted","Data":"395d214b5f90aeac9f195bfee639f94474bce2bb806637ae8d0b815adccf6255"}
Sep 18 19:57:17.741308 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:17.741292    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-scn7p_a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48/ovn-acl-logging/0.log"
Sep 18 19:57:17.741695 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:17.741666    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-scn7p" event={"ID":"a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48","Type":"ContainerStarted","Data":"80fea3e30ecb65e1c9f1546f466764f14753116e9eee03a2edcbf49200a777d6"}
Sep 18 19:57:17.743035 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:17.743016    2549 generic.go:356] "Generic (PLEG): container finished" podID="56b6948b-89ff-4d48-ba8c-c0c3f63544e1" containerID="2a5d7ead418d9da50f1a8e0a02ffca264e44ebad09c9ef3865c26ba270d4ba6a" exitCode=0
Sep 18 19:57:17.743115 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:17.743070    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8nv28" event={"ID":"56b6948b-89ff-4d48-ba8c-c0c3f63544e1","Type":"ContainerDied","Data":"2a5d7ead418d9da50f1a8e0a02ffca264e44ebad09c9ef3865c26ba270d4ba6a"}
Sep 18 19:57:17.744441 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:17.744425    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-tlqj8" event={"ID":"2e178884-5bfb-46cb-9305-6170284cb243","Type":"ContainerStarted","Data":"de478f2bba90cb73b83a67c1f4136f866b47b8ed6a1b2ad06a1a2c25dec7df90"}
Sep 18 19:57:17.796079 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:17.795993    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-m6qj6" podStartSLOduration=2.504973341 podStartE2EDuration="21.795976657s" totalImagesPullingTime="19.291003316s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:56:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:57:17.795510935 +0000 UTC m=+22.749252102" watchObservedRunningTime="2026-09-18 19:57:17.795976657 +0000 UTC m=+22.749717825"
Sep 18 19:57:17.816856 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:17.816805    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-5bnj6" podStartSLOduration=2.491306205 podStartE2EDuration="21.816789327s" totalImagesPullingTime="19.325483122s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:56:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:57:17.816537606 +0000 UTC m=+22.770278775" watchObservedRunningTime="2026-09-18 19:57:17.816789327 +0000 UTC m=+22.770530496"
Sep 18 19:57:17.834805 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:17.834759    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-t26hd" podStartSLOduration=2.416353635 podStartE2EDuration="21.834741669s" totalImagesPullingTime="19.418388034s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:56:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:57:17.834340634 +0000 UTC m=+22.788081802" watchObservedRunningTime="2026-09-18 19:57:17.834741669 +0000 UTC m=+22.788482838"
Sep 18 19:57:17.852053 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:17.852014    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-tlqj8" podStartSLOduration=2.407069202 podStartE2EDuration="21.852003057s" totalImagesPullingTime="19.444933855s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:56:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:57:17.851319809 +0000 UTC m=+22.805060977" watchObservedRunningTime="2026-09-18 19:57:17.852003057 +0000 UTC m=+22.805744224"
Sep 18 19:57:18.068610 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:18.068517    2549 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 18 19:57:18.453235 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:18.453081    2549 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-18T19:57:18.068534951Z","UUID":"c6882294-50c4-4368-96e0-80045e30ae4d","Handler":null,"Name":"","Endpoint":""}
Sep 18 19:57:18.454811 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:18.454786    2549 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 18 19:57:18.454912 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:18.454824    2549 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 18 19:57:18.515414 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:18.515364    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2hw77"
Sep 18 19:57:18.515570 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:57:18.515529    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2hw77" podUID="063d8ee5-35b4-401b-a0ff-d8b416e20175"
Sep 18 19:57:18.749836 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:18.749701    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-178.ec2.internal" event={"ID":"e511ba71d033b03f5d71c14df866e732","Type":"ContainerStarted","Data":"6aeee77d7b6681299ff73f1b0c13f6caa751a82aee2b69f0059a3ed8a6a50489"}
Sep 18 19:57:18.752034 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:18.752008    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfp4z" event={"ID":"e5f6d5d2-ce02-4de2-a798-be32d7584248","Type":"ContainerStarted","Data":"95e6bd485294dccc6fc2d6ddfc814b6e73d902a913acfa0015e90ce7f64e6246"}
Sep 18 19:57:18.765629 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:18.765580    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-178.ec2.internal" podStartSLOduration=21.76556641 podStartE2EDuration="21.76556641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:56:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:57:18.76518991 +0000 UTC m=+23.718931080" watchObservedRunningTime="2026-09-18 19:57:18.76556641 +0000 UTC m=+23.719307578"
Sep 18 19:57:18.820466 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:18.820425    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-tlqj8"
Sep 18 19:57:18.821470 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:18.821448    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-tlqj8"
Sep 18 19:57:19.515168 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:19.514946    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zq878"
Sep 18 19:57:19.515320 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:19.515004    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rktz8"
Sep 18 19:57:19.515320 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:57:19.515280    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zq878" podUID="254868d4-6e40-410f-9a3d-0f994b704438"
Sep 18 19:57:19.515426 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:57:19.515363    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rktz8" podUID="ed609df8-4cad-445d-99a8-2df85ce75ba8"
Sep 18 19:57:19.756743 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:19.756633    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfp4z" event={"ID":"e5f6d5d2-ce02-4de2-a798-be32d7584248","Type":"ContainerStarted","Data":"def3856955f4b78f05289865c929d218e256aeeb24125924c4bf2055070c5fa9"}
Sep 18 19:57:19.759717 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:19.759699    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-scn7p_a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48/ovn-acl-logging/0.log"
Sep 18 19:57:19.760159 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:19.760138    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-scn7p" event={"ID":"a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48","Type":"ContainerStarted","Data":"ce77ee809ad91dd3efe907126f9bbd33dcca48a913c6221328857dace5763d78"}
Sep 18 19:57:19.779196 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:19.779149    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfp4z" podStartSLOduration=1.966435913 podStartE2EDuration="23.779134777s" totalImagesPullingTime="21.812698864s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:56:56 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:57:19.77885118 +0000 UTC m=+24.732592351" watchObservedRunningTime="2026-09-18 19:57:19.779134777 +0000 UTC m=+24.732875945"
Sep 18 19:57:20.515293 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:20.515255    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2hw77"
Sep 18 19:57:20.515505 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:57:20.515373    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2hw77" podUID="063d8ee5-35b4-401b-a0ff-d8b416e20175"
Sep 18 19:57:20.762902 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:20.762869    2549 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 18 19:57:21.515599 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:21.515558    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rktz8"
Sep 18 19:57:21.515791 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:21.515613    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zq878"
Sep 18 19:57:21.515791 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:57:21.515693    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rktz8" podUID="ed609df8-4cad-445d-99a8-2df85ce75ba8"
Sep 18 19:57:21.515876 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:57:21.515821    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zq878" podUID="254868d4-6e40-410f-9a3d-0f994b704438"
Sep 18 19:57:22.290945 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:22.290748    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-tlqj8"
Sep 18 19:57:22.291369 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:22.291033    2549 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 18 19:57:22.291452 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:22.291419    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-tlqj8"
Sep 18 19:57:22.514984 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:22.514954    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2hw77"
Sep 18 19:57:22.515144 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:57:22.515072    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2hw77" podUID="063d8ee5-35b4-401b-a0ff-d8b416e20175"
Sep 18 19:57:22.770525 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:22.770446    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-scn7p_a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48/ovn-acl-logging/0.log"
Sep 18 19:57:22.770818 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:22.770795    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-scn7p" event={"ID":"a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48","Type":"ContainerStarted","Data":"5f1e20a40b25773859ff5d41dff26496e6ab0beec022c7a50b5cf2d1483961b7"}
Sep 18 19:57:22.771048 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:22.771026    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-scn7p"
Sep 18 19:57:22.771281 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:22.771260    2549 scope.go:117] "RemoveContainer" containerID="c431d7de95774eaee6fc7909569de6bfc80e47b17fa712577c3c696002948700"
Sep 18 19:57:22.772766 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:22.772747    2549 generic.go:356] "Generic (PLEG): container finished" podID="56b6948b-89ff-4d48-ba8c-c0c3f63544e1" containerID="a1b538bd4996f6a2567458a82c9019c4412331cc17f0f38beeb106c8d5f7663e" exitCode=0
Sep 18 19:57:22.772869 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:22.772829    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8nv28" event={"ID":"56b6948b-89ff-4d48-ba8c-c0c3f63544e1","Type":"ContainerDied","Data":"a1b538bd4996f6a2567458a82c9019c4412331cc17f0f38beeb106c8d5f7663e"}
Sep 18 19:57:22.787767 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:22.787745    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-scn7p"
Sep 18 19:57:23.515348 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:23.515312    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rktz8"
Sep 18 19:57:23.515711 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:57:23.515452    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rktz8" podUID="ed609df8-4cad-445d-99a8-2df85ce75ba8"
Sep 18 19:57:23.515711 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:23.515519    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zq878"
Sep 18 19:57:23.515711 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:57:23.515609    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zq878" podUID="254868d4-6e40-410f-9a3d-0f994b704438"
Sep 18 19:57:23.778775 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:23.778703    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-scn7p_a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48/ovn-acl-logging/0.log"
Sep 18 19:57:23.779155 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:23.779127    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-scn7p" event={"ID":"a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48","Type":"ContainerStarted","Data":"bcff3cb3175fa2897719bf644d940f460f1ef80719846f274cc926a9960774c9"}
Sep 18 19:57:23.779509 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:23.779491    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-scn7p"
Sep 18 19:57:23.779669 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:23.779636    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-scn7p"
Sep 18 19:57:23.781491 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:23.781470    2549 generic.go:356] "Generic (PLEG): container finished" podID="56b6948b-89ff-4d48-ba8c-c0c3f63544e1" containerID="0b42fc0fbda26cfd4510a2eda1341455966a35532a1c1a4f7ebf81fb1a84c403" exitCode=0
Sep 18 19:57:23.781617 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:23.781518    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8nv28" event={"ID":"56b6948b-89ff-4d48-ba8c-c0c3f63544e1","Type":"ContainerDied","Data":"0b42fc0fbda26cfd4510a2eda1341455966a35532a1c1a4f7ebf81fb1a84c403"}
Sep 18 19:57:23.794719 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:23.794694    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-scn7p"
Sep 18 19:57:23.819984 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:23.819939    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-scn7p" podStartSLOduration=8.43988046 podStartE2EDuration="27.819923557s" totalImagesPullingTime="19.380043097s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:56:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:57:23.819530838 +0000 UTC m=+28.773272017" watchObservedRunningTime="2026-09-18 19:57:23.819923557 +0000 UTC m=+28.773664724"
Sep 18 19:57:24.151375 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:24.151301    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zq878"]
Sep 18 19:57:24.151623 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:24.151521    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zq878"
Sep 18 19:57:24.151713 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:57:24.151638    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zq878" podUID="254868d4-6e40-410f-9a3d-0f994b704438"
Sep 18 19:57:24.152048 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:24.151822    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2hw77"]
Sep 18 19:57:24.152048 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:24.151933    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2hw77"
Sep 18 19:57:24.152308 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:57:24.152055    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2hw77" podUID="063d8ee5-35b4-401b-a0ff-d8b416e20175"
Sep 18 19:57:24.152606 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:24.152538    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rktz8"]
Sep 18 19:57:24.152699 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:24.152633    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rktz8"
Sep 18 19:57:24.152755 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:57:24.152724    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rktz8" podUID="ed609df8-4cad-445d-99a8-2df85ce75ba8"
Sep 18 19:57:25.516161 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:25.516134    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rktz8"
Sep 18 19:57:25.516817 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:57:25.516227    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rktz8" podUID="ed609df8-4cad-445d-99a8-2df85ce75ba8"
Sep 18 19:57:25.516817 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:25.516311    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2hw77"
Sep 18 19:57:25.516817 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:25.516325    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zq878"
Sep 18 19:57:25.516817 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:57:25.516426    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2hw77" podUID="063d8ee5-35b4-401b-a0ff-d8b416e20175"
Sep 18 19:57:25.516817 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:57:25.516499    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zq878" podUID="254868d4-6e40-410f-9a3d-0f994b704438"
Sep 18 19:57:25.788317 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:25.788220    2549 generic.go:356] "Generic (PLEG): container finished" podID="56b6948b-89ff-4d48-ba8c-c0c3f63544e1" containerID="ed641504c6899fb4da658923bf70fd1f7502307a646767b939b057c383d1b1a1" exitCode=0
Sep 18 19:57:25.788317 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:25.788270    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8nv28" event={"ID":"56b6948b-89ff-4d48-ba8c-c0c3f63544e1","Type":"ContainerDied","Data":"ed641504c6899fb4da658923bf70fd1f7502307a646767b939b057c383d1b1a1"}
Sep 18 19:57:27.514871 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:27.514835    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zq878"
Sep 18 19:57:27.515289 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:27.514835    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rktz8"
Sep 18 19:57:27.515289 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:57:27.514935    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zq878" podUID="254868d4-6e40-410f-9a3d-0f994b704438"
Sep 18 19:57:27.515289 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:57:27.515011    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rktz8" podUID="ed609df8-4cad-445d-99a8-2df85ce75ba8"
Sep 18 19:57:27.515289 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:27.515066    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2hw77"
Sep 18 19:57:27.515289 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:57:27.515124    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2hw77" podUID="063d8ee5-35b4-401b-a0ff-d8b416e20175"
Sep 18 19:57:28.232633 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:28.232591    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/254868d4-6e40-410f-9a3d-0f994b704438-original-pull-secret\") pod \"global-pull-secret-syncer-zq878\" (UID: \"254868d4-6e40-410f-9a3d-0f994b704438\") " pod="kube-system/global-pull-secret-syncer-zq878"
Sep 18 19:57:28.232813 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:28.232675    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/063d8ee5-35b4-401b-a0ff-d8b416e20175-metrics-certs\") pod \"network-metrics-daemon-2hw77\" (UID: \"063d8ee5-35b4-401b-a0ff-d8b416e20175\") " pod="openshift-multus/network-metrics-daemon-2hw77"
Sep 18 19:57:28.232813 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:57:28.232795    2549 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:57:28.232937 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:57:28.232849    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/063d8ee5-35b4-401b-a0ff-d8b416e20175-metrics-certs podName:063d8ee5-35b4-401b-a0ff-d8b416e20175 nodeName:}" failed. No retries permitted until 2026-09-18 19:58:00.232830188 +0000 UTC m=+65.186571335 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/063d8ee5-35b4-401b-a0ff-d8b416e20175-metrics-certs") pod "network-metrics-daemon-2hw77" (UID: "063d8ee5-35b4-401b-a0ff-d8b416e20175") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:57:28.233130 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:57:28.233105    2549 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:57:28.233267 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:57:28.233175    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/254868d4-6e40-410f-9a3d-0f994b704438-original-pull-secret podName:254868d4-6e40-410f-9a3d-0f994b704438 nodeName:}" failed. No retries permitted until 2026-09-18 19:58:00.233157905 +0000 UTC m=+65.186899061 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/254868d4-6e40-410f-9a3d-0f994b704438-original-pull-secret") pod "global-pull-secret-syncer-zq878" (UID: "254868d4-6e40-410f-9a3d-0f994b704438") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:57:28.333096 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:28.333056    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd8qv\" (UniqueName: \"kubernetes.io/projected/ed609df8-4cad-445d-99a8-2df85ce75ba8-kube-api-access-nd8qv\") pod \"network-check-target-rktz8\" (UID: \"ed609df8-4cad-445d-99a8-2df85ce75ba8\") " pod="openshift-network-diagnostics/network-check-target-rktz8"
Sep 18 19:57:28.333275 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:57:28.333235    2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:57:28.333275 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:57:28.333265    2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:57:28.333400 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:57:28.333278    2549 projected.go:196] Error preparing data for projected volume kube-api-access-nd8qv for pod openshift-network-diagnostics/network-check-target-rktz8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:57:28.333400 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:57:28.333339    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ed609df8-4cad-445d-99a8-2df85ce75ba8-kube-api-access-nd8qv podName:ed609df8-4cad-445d-99a8-2df85ce75ba8 nodeName:}" failed. No retries permitted until 2026-09-18 19:58:00.333323326 +0000 UTC m=+65.287064475 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-nd8qv" (UniqueName: "kubernetes.io/projected/ed609df8-4cad-445d-99a8-2df85ce75ba8-kube-api-access-nd8qv") pod "network-check-target-rktz8" (UID: "ed609df8-4cad-445d-99a8-2df85ce75ba8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:57:28.809519 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:28.809487    2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-178.ec2.internal" event="NodeReady"
Sep 18 19:57:28.809941 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:28.809654    2549 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 18 19:57:28.865851 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:28.865816    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-mbfbc"]
Sep 18 19:57:28.874852 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:28.874824    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-xrlbt"]
Sep 18 19:57:28.874997 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:28.874945    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mbfbc"
Sep 18 19:57:28.877792 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:28.877767    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 18 19:57:28.877915 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:28.877778    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-r2wjd\""
Sep 18 19:57:28.878271 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:28.878250    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 18 19:57:28.884238 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:28.884019    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mbfbc"]
Sep 18 19:57:28.884343 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:28.884251    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xrlbt"
Sep 18 19:57:28.885674 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:28.885646    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xrlbt"]
Sep 18 19:57:28.888838 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:28.888805    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 18 19:57:28.889204 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:28.889174    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-xx2wd\""
Sep 18 19:57:28.889313 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:28.889203    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 18 19:57:28.889632 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:28.889455    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 18 19:57:28.938016 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:28.937977    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0b1d2455-d7fc-4bde-a435-7dfaedd8390e-metrics-tls\") pod \"dns-default-mbfbc\" (UID: \"0b1d2455-d7fc-4bde-a435-7dfaedd8390e\") " pod="openshift-dns/dns-default-mbfbc"
Sep 18 19:57:28.938187 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:28.938022    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0b1d2455-d7fc-4bde-a435-7dfaedd8390e-config-volume\") pod \"dns-default-mbfbc\" (UID: \"0b1d2455-d7fc-4bde-a435-7dfaedd8390e\") " pod="openshift-dns/dns-default-mbfbc"
Sep 18 19:57:28.938187 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:28.938068    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bb9c6\" (UniqueName: \"kubernetes.io/projected/0b1d2455-d7fc-4bde-a435-7dfaedd8390e-kube-api-access-bb9c6\") pod \"dns-default-mbfbc\" (UID: \"0b1d2455-d7fc-4bde-a435-7dfaedd8390e\") " pod="openshift-dns/dns-default-mbfbc"
Sep 18 19:57:28.938187 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:28.938107    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nx2g4\" (UniqueName: \"kubernetes.io/projected/605e3d2c-3e1c-48a4-b02a-77bedbaa6a4c-kube-api-access-nx2g4\") pod \"ingress-canary-xrlbt\" (UID: \"605e3d2c-3e1c-48a4-b02a-77bedbaa6a4c\") " pod="openshift-ingress-canary/ingress-canary-xrlbt"
Sep 18 19:57:28.938187 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:28.938144    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/605e3d2c-3e1c-48a4-b02a-77bedbaa6a4c-cert\") pod \"ingress-canary-xrlbt\" (UID: \"605e3d2c-3e1c-48a4-b02a-77bedbaa6a4c\") " pod="openshift-ingress-canary/ingress-canary-xrlbt"
Sep 18 19:57:28.938187 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:28.938174    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b1d2455-d7fc-4bde-a435-7dfaedd8390e-tmp-dir\") pod \"dns-default-mbfbc\" (UID: \"0b1d2455-d7fc-4bde-a435-7dfaedd8390e\") " pod="openshift-dns/dns-default-mbfbc"
Sep 18 19:57:29.038808 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:29.038714    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0b1d2455-d7fc-4bde-a435-7dfaedd8390e-metrics-tls\") pod \"dns-default-mbfbc\" (UID: \"0b1d2455-d7fc-4bde-a435-7dfaedd8390e\") " pod="openshift-dns/dns-default-mbfbc"
Sep 18 19:57:29.038808 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:29.038765    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0b1d2455-d7fc-4bde-a435-7dfaedd8390e-config-volume\") pod \"dns-default-mbfbc\" (UID: \"0b1d2455-d7fc-4bde-a435-7dfaedd8390e\") " pod="openshift-dns/dns-default-mbfbc"
Sep 18 19:57:29.039028 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:29.038835    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb9c6\" (UniqueName: \"kubernetes.io/projected/0b1d2455-d7fc-4bde-a435-7dfaedd8390e-kube-api-access-bb9c6\") pod \"dns-default-mbfbc\" (UID: \"0b1d2455-d7fc-4bde-a435-7dfaedd8390e\") " pod="openshift-dns/dns-default-mbfbc"
Sep 18 19:57:29.039028 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:29.038856    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx2g4\" (UniqueName: \"kubernetes.io/projected/605e3d2c-3e1c-48a4-b02a-77bedbaa6a4c-kube-api-access-nx2g4\") pod \"ingress-canary-xrlbt\" (UID: \"605e3d2c-3e1c-48a4-b02a-77bedbaa6a4c\") " pod="openshift-ingress-canary/ingress-canary-xrlbt"
Sep 18 19:57:29.039028 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:57:29.038898    2549 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:57:29.039028 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:57:29.038975    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b1d2455-d7fc-4bde-a435-7dfaedd8390e-metrics-tls podName:0b1d2455-d7fc-4bde-a435-7dfaedd8390e nodeName:}" failed. No retries permitted until 2026-09-18 19:57:29.53895313 +0000 UTC m=+34.492694289 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0b1d2455-d7fc-4bde-a435-7dfaedd8390e-metrics-tls") pod "dns-default-mbfbc" (UID: "0b1d2455-d7fc-4bde-a435-7dfaedd8390e") : secret "dns-default-metrics-tls" not found
Sep 18 19:57:29.039028 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:29.039000    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/605e3d2c-3e1c-48a4-b02a-77bedbaa6a4c-cert\") pod \"ingress-canary-xrlbt\" (UID: \"605e3d2c-3e1c-48a4-b02a-77bedbaa6a4c\") " pod="openshift-ingress-canary/ingress-canary-xrlbt"
Sep 18 19:57:29.039278 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:29.039051    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b1d2455-d7fc-4bde-a435-7dfaedd8390e-tmp-dir\") pod \"dns-default-mbfbc\" (UID: \"0b1d2455-d7fc-4bde-a435-7dfaedd8390e\") " pod="openshift-dns/dns-default-mbfbc"
Sep 18 19:57:29.039430 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:29.039410    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b1d2455-d7fc-4bde-a435-7dfaedd8390e-tmp-dir\") pod \"dns-default-mbfbc\" (UID: \"0b1d2455-d7fc-4bde-a435-7dfaedd8390e\") " pod="openshift-dns/dns-default-mbfbc"
Sep 18 19:57:29.039634 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:29.039610    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0b1d2455-d7fc-4bde-a435-7dfaedd8390e-config-volume\") pod \"dns-default-mbfbc\" (UID: \"0b1d2455-d7fc-4bde-a435-7dfaedd8390e\") " pod="openshift-dns/dns-default-mbfbc"
Sep 18 19:57:29.043984 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:29.043960    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/605e3d2c-3e1c-48a4-b02a-77bedbaa6a4c-cert\") pod \"ingress-canary-xrlbt\" (UID: \"605e3d2c-3e1c-48a4-b02a-77bedbaa6a4c\") " pod="openshift-ingress-canary/ingress-canary-xrlbt"
Sep 18 19:57:29.052754 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:29.052720    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb9c6\" (UniqueName: \"kubernetes.io/projected/0b1d2455-d7fc-4bde-a435-7dfaedd8390e-kube-api-access-bb9c6\") pod \"dns-default-mbfbc\" (UID: \"0b1d2455-d7fc-4bde-a435-7dfaedd8390e\") " pod="openshift-dns/dns-default-mbfbc"
Sep 18 19:57:29.053024 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:29.053007    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nx2g4\" (UniqueName: \"kubernetes.io/projected/605e3d2c-3e1c-48a4-b02a-77bedbaa6a4c-kube-api-access-nx2g4\") pod \"ingress-canary-xrlbt\" (UID: \"605e3d2c-3e1c-48a4-b02a-77bedbaa6a4c\") " pod="openshift-ingress-canary/ingress-canary-xrlbt"
Sep 18 19:57:29.198337 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:29.198300    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xrlbt"
Sep 18 19:57:29.515207 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:29.515170    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2hw77"
Sep 18 19:57:29.515516 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:29.515170    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rktz8"
Sep 18 19:57:29.515516 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:29.515170    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zq878"
Sep 18 19:57:29.518663 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:29.518642    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 19:57:29.520068 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:29.520046    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gb8gb\""
Sep 18 19:57:29.520189 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:29.520096    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 19:57:29.520252 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:29.520211    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6rwzk\""
Sep 18 19:57:29.520307 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:29.520292    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 19:57:29.520483 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:29.520466    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 19:57:29.542907 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:29.542886    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0b1d2455-d7fc-4bde-a435-7dfaedd8390e-metrics-tls\") pod \"dns-default-mbfbc\" (UID: \"0b1d2455-d7fc-4bde-a435-7dfaedd8390e\") " pod="openshift-dns/dns-default-mbfbc"
Sep 18 19:57:29.543055 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:57:29.543039    2549 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:57:29.543119 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:57:29.543108    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b1d2455-d7fc-4bde-a435-7dfaedd8390e-metrics-tls podName:0b1d2455-d7fc-4bde-a435-7dfaedd8390e nodeName:}" failed. No retries permitted until 2026-09-18 19:57:30.543089109 +0000 UTC m=+35.496830274 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0b1d2455-d7fc-4bde-a435-7dfaedd8390e-metrics-tls") pod "dns-default-mbfbc" (UID: "0b1d2455-d7fc-4bde-a435-7dfaedd8390e") : secret "dns-default-metrics-tls" not found
Sep 18 19:57:30.550897 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:30.550821    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0b1d2455-d7fc-4bde-a435-7dfaedd8390e-metrics-tls\") pod \"dns-default-mbfbc\" (UID: \"0b1d2455-d7fc-4bde-a435-7dfaedd8390e\") " pod="openshift-dns/dns-default-mbfbc"
Sep 18 19:57:30.551222 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:57:30.550978    2549 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:57:30.551222 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:57:30.551045    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b1d2455-d7fc-4bde-a435-7dfaedd8390e-metrics-tls podName:0b1d2455-d7fc-4bde-a435-7dfaedd8390e nodeName:}" failed. No retries permitted until 2026-09-18 19:57:32.55102969 +0000 UTC m=+37.504770836 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0b1d2455-d7fc-4bde-a435-7dfaedd8390e-metrics-tls") pod "dns-default-mbfbc" (UID: "0b1d2455-d7fc-4bde-a435-7dfaedd8390e") : secret "dns-default-metrics-tls" not found
Sep 18 19:57:30.592617 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:30.592402    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xrlbt"]
Sep 18 19:57:30.595718 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:57:30.595682    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod605e3d2c_3e1c_48a4_b02a_77bedbaa6a4c.slice/crio-e871d24b31992def93b46ce4c1857f56d0048f07d7bdfebfe87779fea2e59477 WatchSource:0}: Error finding container e871d24b31992def93b46ce4c1857f56d0048f07d7bdfebfe87779fea2e59477: Status 404 returned error can't find the container with id e871d24b31992def93b46ce4c1857f56d0048f07d7bdfebfe87779fea2e59477
Sep 18 19:57:30.804906 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:30.804879    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8nv28" event={"ID":"56b6948b-89ff-4d48-ba8c-c0c3f63544e1","Type":"ContainerStarted","Data":"5879acb8db68ee85cf20dd5b691e28cbbfde2d4ef926874b2c6addbd1a3a5dab"}
Sep 18 19:57:30.806028 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:30.806005    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xrlbt" event={"ID":"605e3d2c-3e1c-48a4-b02a-77bedbaa6a4c","Type":"ContainerStarted","Data":"e871d24b31992def93b46ce4c1857f56d0048f07d7bdfebfe87779fea2e59477"}
Sep 18 19:57:31.811192 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:31.811155    2549 generic.go:356] "Generic (PLEG): container finished" podID="56b6948b-89ff-4d48-ba8c-c0c3f63544e1" containerID="5879acb8db68ee85cf20dd5b691e28cbbfde2d4ef926874b2c6addbd1a3a5dab" exitCode=0
Sep 18 19:57:31.811684 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:31.811212    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8nv28" event={"ID":"56b6948b-89ff-4d48-ba8c-c0c3f63544e1","Type":"ContainerDied","Data":"5879acb8db68ee85cf20dd5b691e28cbbfde2d4ef926874b2c6addbd1a3a5dab"}
Sep 18 19:57:32.570273 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:32.570232    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0b1d2455-d7fc-4bde-a435-7dfaedd8390e-metrics-tls\") pod \"dns-default-mbfbc\" (UID: \"0b1d2455-d7fc-4bde-a435-7dfaedd8390e\") " pod="openshift-dns/dns-default-mbfbc"
Sep 18 19:57:32.570485 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:57:32.570404    2549 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:57:32.570555 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:57:32.570487    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b1d2455-d7fc-4bde-a435-7dfaedd8390e-metrics-tls podName:0b1d2455-d7fc-4bde-a435-7dfaedd8390e nodeName:}" failed. No retries permitted until 2026-09-18 19:57:36.570465195 +0000 UTC m=+41.524206346 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0b1d2455-d7fc-4bde-a435-7dfaedd8390e-metrics-tls") pod "dns-default-mbfbc" (UID: "0b1d2455-d7fc-4bde-a435-7dfaedd8390e") : secret "dns-default-metrics-tls" not found
Sep 18 19:57:32.818187 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:32.818152    2549 generic.go:356] "Generic (PLEG): container finished" podID="56b6948b-89ff-4d48-ba8c-c0c3f63544e1" containerID="bfefbe62f8e7bd8f639e68a3cdd5fc5017f7d652a7b4e589a2b8a39949e77b85" exitCode=0
Sep 18 19:57:32.818187 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:32.818186    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8nv28" event={"ID":"56b6948b-89ff-4d48-ba8c-c0c3f63544e1","Type":"ContainerDied","Data":"bfefbe62f8e7bd8f639e68a3cdd5fc5017f7d652a7b4e589a2b8a39949e77b85"}
Sep 18 19:57:33.823519 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:33.823475    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8nv28" event={"ID":"56b6948b-89ff-4d48-ba8c-c0c3f63544e1","Type":"ContainerStarted","Data":"5a491cc13b26021237a4745a14fe3d0f663a59d8ec29b43212539458b7e548cd"}
Sep 18 19:57:33.824907 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:33.824886    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xrlbt" event={"ID":"605e3d2c-3e1c-48a4-b02a-77bedbaa6a4c","Type":"ContainerStarted","Data":"aac9de3f12f7f6108b20b800ac05652e9f4ddfe7d0204f971194f8f1c1182cef"}
Sep 18 19:57:33.865962 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:33.865909    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-8nv28" podStartSLOduration=7.394211172 podStartE2EDuration="37.865895625s" totalImagesPullingTime="30.471684453s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:56:56 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:57:33.865888021 +0000 UTC m=+38.819629189" watchObservedRunningTime="2026-09-18 19:57:33.865895625 +0000 UTC m=+38.819636790"
Sep 18 19:57:36.599021 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:36.598981    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0b1d2455-d7fc-4bde-a435-7dfaedd8390e-metrics-tls\") pod \"dns-default-mbfbc\" (UID: \"0b1d2455-d7fc-4bde-a435-7dfaedd8390e\") " pod="openshift-dns/dns-default-mbfbc"
Sep 18 19:57:36.599456 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:57:36.599148    2549 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:57:36.599456 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:57:36.599223    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b1d2455-d7fc-4bde-a435-7dfaedd8390e-metrics-tls podName:0b1d2455-d7fc-4bde-a435-7dfaedd8390e nodeName:}" failed. No retries permitted until 2026-09-18 19:57:44.599208715 +0000 UTC m=+49.552949861 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0b1d2455-d7fc-4bde-a435-7dfaedd8390e-metrics-tls") pod "dns-default-mbfbc" (UID: "0b1d2455-d7fc-4bde-a435-7dfaedd8390e") : secret "dns-default-metrics-tls" not found
Sep 18 19:57:44.648566 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:44.648513    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0b1d2455-d7fc-4bde-a435-7dfaedd8390e-metrics-tls\") pod \"dns-default-mbfbc\" (UID: \"0b1d2455-d7fc-4bde-a435-7dfaedd8390e\") " pod="openshift-dns/dns-default-mbfbc"
Sep 18 19:57:44.648951 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:57:44.648665    2549 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:57:44.648951 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:57:44.648730    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b1d2455-d7fc-4bde-a435-7dfaedd8390e-metrics-tls podName:0b1d2455-d7fc-4bde-a435-7dfaedd8390e nodeName:}" failed. No retries permitted until 2026-09-18 19:58:00.648715423 +0000 UTC m=+65.602456573 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0b1d2455-d7fc-4bde-a435-7dfaedd8390e-metrics-tls") pod "dns-default-mbfbc" (UID: "0b1d2455-d7fc-4bde-a435-7dfaedd8390e") : secret "dns-default-metrics-tls" not found
Sep 18 19:57:55.803577 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:55.803549    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-scn7p"
Sep 18 19:57:55.837537 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:57:55.837463    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-xrlbt" podStartSLOduration=25.148166567 podStartE2EDuration="27.837443732s" totalImagesPullingTime="2.689277165s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:57:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:57:33.899103715 +0000 UTC m=+38.852844883" watchObservedRunningTime="2026-09-18 19:57:55.837443732 +0000 UTC m=+60.791184901"
Sep 18 19:58:00.253971 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:00.253917    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/063d8ee5-35b4-401b-a0ff-d8b416e20175-metrics-certs\") pod \"network-metrics-daemon-2hw77\" (UID: \"063d8ee5-35b4-401b-a0ff-d8b416e20175\") " pod="openshift-multus/network-metrics-daemon-2hw77"
Sep 18 19:58:00.254360 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:00.254005    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/254868d4-6e40-410f-9a3d-0f994b704438-original-pull-secret\") pod \"global-pull-secret-syncer-zq878\" (UID: \"254868d4-6e40-410f-9a3d-0f994b704438\") " pod="kube-system/global-pull-secret-syncer-zq878"
Sep 18 19:58:00.257047 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:00.257027    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 19:58:00.257113 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:00.257092    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 19:58:00.264594 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:58:00.264568    2549 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 18 19:58:00.264711 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:58:00.264640    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/063d8ee5-35b4-401b-a0ff-d8b416e20175-metrics-certs podName:063d8ee5-35b4-401b-a0ff-d8b416e20175 nodeName:}" failed. No retries permitted until 2026-09-18 19:59:04.264623871 +0000 UTC m=+129.218365021 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/063d8ee5-35b4-401b-a0ff-d8b416e20175-metrics-certs") pod "network-metrics-daemon-2hw77" (UID: "063d8ee5-35b4-401b-a0ff-d8b416e20175") : secret "metrics-daemon-secret" not found
Sep 18 19:58:00.267628 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:00.267597    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/254868d4-6e40-410f-9a3d-0f994b704438-original-pull-secret\") pod \"global-pull-secret-syncer-zq878\" (UID: \"254868d4-6e40-410f-9a3d-0f994b704438\") " pod="kube-system/global-pull-secret-syncer-zq878"
Sep 18 19:58:00.354964 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:00.354919    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd8qv\" (UniqueName: \"kubernetes.io/projected/ed609df8-4cad-445d-99a8-2df85ce75ba8-kube-api-access-nd8qv\") pod \"network-check-target-rktz8\" (UID: \"ed609df8-4cad-445d-99a8-2df85ce75ba8\") " pod="openshift-network-diagnostics/network-check-target-rktz8"
Sep 18 19:58:00.358421 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:00.358402    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 19:58:00.368671 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:00.368654    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 19:58:00.379387 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:00.379367    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd8qv\" (UniqueName: \"kubernetes.io/projected/ed609df8-4cad-445d-99a8-2df85ce75ba8-kube-api-access-nd8qv\") pod \"network-check-target-rktz8\" (UID: \"ed609df8-4cad-445d-99a8-2df85ce75ba8\") " pod="openshift-network-diagnostics/network-check-target-rktz8"
Sep 18 19:58:00.436494 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:00.436464    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gb8gb\""
Sep 18 19:58:00.438553 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:00.438539    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zq878"
Sep 18 19:58:00.444297 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:00.444274    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rktz8"
Sep 18 19:58:00.656844 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:00.656804    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0b1d2455-d7fc-4bde-a435-7dfaedd8390e-metrics-tls\") pod \"dns-default-mbfbc\" (UID: \"0b1d2455-d7fc-4bde-a435-7dfaedd8390e\") " pod="openshift-dns/dns-default-mbfbc"
Sep 18 19:58:00.657013 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:58:00.656952    2549 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:58:00.657054 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:58:00.657025    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b1d2455-d7fc-4bde-a435-7dfaedd8390e-metrics-tls podName:0b1d2455-d7fc-4bde-a435-7dfaedd8390e nodeName:}" failed. No retries permitted until 2026-09-18 19:58:32.657008563 +0000 UTC m=+97.610749721 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0b1d2455-d7fc-4bde-a435-7dfaedd8390e-metrics-tls") pod "dns-default-mbfbc" (UID: "0b1d2455-d7fc-4bde-a435-7dfaedd8390e") : secret "dns-default-metrics-tls" not found
Sep 18 19:58:00.712906 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:00.712877    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zq878"]
Sep 18 19:58:00.716065 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:58:00.716037    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod254868d4_6e40_410f_9a3d_0f994b704438.slice/crio-463d9a516f0a515753ec31dd3f8f52abf372e5ea1004320f09c864f847b0bc9d WatchSource:0}: Error finding container 463d9a516f0a515753ec31dd3f8f52abf372e5ea1004320f09c864f847b0bc9d: Status 404 returned error can't find the container with id 463d9a516f0a515753ec31dd3f8f52abf372e5ea1004320f09c864f847b0bc9d
Sep 18 19:58:00.716065 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:00.716056    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rktz8"]
Sep 18 19:58:00.718998 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:58:00.718974    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded609df8_4cad_445d_99a8_2df85ce75ba8.slice/crio-782bea961552e3805b269f7e54a6867935b5d31609cc8087056cd2f3cc712015 WatchSource:0}: Error finding container 782bea961552e3805b269f7e54a6867935b5d31609cc8087056cd2f3cc712015: Status 404 returned error can't find the container with id 782bea961552e3805b269f7e54a6867935b5d31609cc8087056cd2f3cc712015
Sep 18 19:58:00.888044 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:00.888012    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rktz8" event={"ID":"ed609df8-4cad-445d-99a8-2df85ce75ba8","Type":"ContainerStarted","Data":"782bea961552e3805b269f7e54a6867935b5d31609cc8087056cd2f3cc712015"}
Sep 18 19:58:00.889073 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:00.889051    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zq878" event={"ID":"254868d4-6e40-410f-9a3d-0f994b704438","Type":"ContainerStarted","Data":"463d9a516f0a515753ec31dd3f8f52abf372e5ea1004320f09c864f847b0bc9d"}
Sep 18 19:58:05.902871 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:05.902830    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zq878" event={"ID":"254868d4-6e40-410f-9a3d-0f994b704438","Type":"ContainerStarted","Data":"db47d8ed04f6124515554c388a49ff1cf304cbb8b825061eea8aa9e16c66272d"}
Sep 18 19:58:05.904257 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:05.904234    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rktz8" event={"ID":"ed609df8-4cad-445d-99a8-2df85ce75ba8","Type":"ContainerStarted","Data":"ef7bc490d0a5a046eee0c104904350aec7579a0d8f9e78e3333ba7e0af6c1ff8"}
Sep 18 19:58:05.904360 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:05.904332    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-rktz8"
Sep 18 19:58:05.921906 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:05.921866    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-zq878" podStartSLOduration=65.363824074 podStartE2EDuration="1m9.921851557s" totalImagesPullingTime="4.558027483s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:56:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:58:05.921309456 +0000 UTC m=+70.875050635" watchObservedRunningTime="2026-09-18 19:58:05.921851557 +0000 UTC m=+70.875592719"
Sep 18 19:58:05.940307 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:05.940247    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-rktz8" podStartSLOduration=65.387510199 podStartE2EDuration="1m9.940232739s" totalImagesPullingTime="4.55272254s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:56:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:58:05.93916149 +0000 UTC m=+70.892902652" watchObservedRunningTime="2026-09-18 19:58:05.940232739 +0000 UTC m=+70.893973904"
Sep 18 19:58:20.487257 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:20.487145    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-whdp6"]
Sep 18 19:58:20.489336 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:20.489321    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-whdp6"
Sep 18 19:58:20.489729 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:20.489704    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2lhk\" (UniqueName: \"kubernetes.io/projected/ddaf88b0-c4ce-42ac-a06a-e95db6854eab-kube-api-access-d2lhk\") pod \"volume-data-source-validator-5c4db8bdf9-whdp6\" (UID: \"ddaf88b0-c4ce-42ac-a06a-e95db6854eab\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-whdp6"
Sep 18 19:58:20.492806 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:20.492786    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-rgv6w\""
Sep 18 19:58:20.493998 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:20.493978    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 18 19:58:20.494062 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:20.494023    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 18 19:58:20.499847 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:20.499825    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-whdp6"]
Sep 18 19:58:20.590508 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:20.590468    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2lhk\" (UniqueName: \"kubernetes.io/projected/ddaf88b0-c4ce-42ac-a06a-e95db6854eab-kube-api-access-d2lhk\") pod \"volume-data-source-validator-5c4db8bdf9-whdp6\" (UID: \"ddaf88b0-c4ce-42ac-a06a-e95db6854eab\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-whdp6"
Sep 18 19:58:20.591994 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:20.591962    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-z7zsx"]
Sep 18 19:58:20.593746 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:20.593728    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-sfdnq"]
Sep 18 19:58:20.593872 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:20.593852    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-z7zsx"
Sep 18 19:58:20.595542 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:20.595519    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-7ff8bb4888-5sfkj"]
Sep 18 19:58:20.595649 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:20.595626    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-sfdnq"
Sep 18 19:58:20.596881 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:20.596860    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7ff8bb4888-5sfkj"
Sep 18 19:58:20.597889 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:20.597865    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 18 19:58:20.598001 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:20.597920    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 18 19:58:20.598115 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:20.598098    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-q9szp\""
Sep 18 19:58:20.599849 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:20.599828    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 18 19:58:20.600055 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:20.599943    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-bgwhj\""
Sep 18 19:58:20.600116 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:20.600066    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 18 19:58:20.600168 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:20.600126    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 18 19:58:20.600496 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:20.600473    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 18 19:58:20.600605 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:20.600498    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 18 19:58:20.600605 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:20.600516    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 18 19:58:20.600605 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:20.600588    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 18 19:58:20.600742 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:20.600613    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 18 19:58:20.603105 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:20.603088    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 18 19:58:20.611473 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:20.611455    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2lhk\" (UniqueName: \"kubernetes.io/projected/ddaf88b0-c4ce-42ac-a06a-e95db6854eab-kube-api-access-d2lhk\") pod \"volume-data-source-validator-5c4db8bdf9-whdp6\" (UID: \"ddaf88b0-c4ce-42ac-a06a-e95db6854eab\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-whdp6"
Sep 18 19:58:20.611566 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:20.611476    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-z7zsx"]
Sep 18 19:58:20.613761 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:20.613743    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-sfdnq"]
Sep 18 19:58:20.616487 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:20.616466    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 18 19:58:20.620698 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:20.620681    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 18 19:58:20.622532 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:20.622517    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-n9sxw\""
Sep 18 19:58:20.630301 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:20.630283    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7ff8bb4888-5sfkj"]
Sep 18 19:58:20.653598 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:20.653579    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 18 19:58:20.691613 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:20.691586    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/174ef0f9-1b9f-48b9-b6c1-1fb01a1f7b1b-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-sfdnq\" (UID: \"174ef0f9-1b9f-48b9-b6c1-1fb01a1f7b1b\") " pod="openshift-insights/insights-operator-6f9c644dd9-sfdnq"
Sep 18 19:58:20.691719 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:20.691614    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl45h\" (UniqueName: \"kubernetes.io/projected/174ef0f9-1b9f-48b9-b6c1-1fb01a1f7b1b-kube-api-access-zl45h\") pod \"insights-operator-6f9c644dd9-sfdnq\" (UID: \"174ef0f9-1b9f-48b9-b6c1-1fb01a1f7b1b\") " pod="openshift-insights/insights-operator-6f9c644dd9-sfdnq"
Sep 18 19:58:20.691719 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:20.691645    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c7b94cd-de57-4efe-a96c-1d349424b2fb-service-ca-bundle\") pod \"router-default-7ff8bb4888-5sfkj\" (UID: \"5c7b94cd-de57-4efe-a96c-1d349424b2fb\") " pod="openshift-ingress/router-default-7ff8bb4888-5sfkj"
Sep 18 19:58:20.691719 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:20.691662    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/174ef0f9-1b9f-48b9-b6c1-1fb01a1f7b1b-tmp\") pod \"insights-operator-6f9c644dd9-sfdnq\" (UID: \"174ef0f9-1b9f-48b9-b6c1-1fb01a1f7b1b\") " pod="openshift-insights/insights-operator-6f9c644dd9-sfdnq"
Sep 18 19:58:20.691719 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:20.691678    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/174ef0f9-1b9f-48b9-b6c1-1fb01a1f7b1b-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-sfdnq\" (UID: \"174ef0f9-1b9f-48b9-b6c1-1fb01a1f7b1b\") " pod="openshift-insights/insights-operator-6f9c644dd9-sfdnq"
Sep 18 19:58:20.691719 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:20.691706    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wdfh\" (UniqueName: \"kubernetes.io/projected/706ee6ce-5cef-4fe6-847d-8745f7884423-kube-api-access-4wdfh\") pod \"cluster-samples-operator-685d7d68b5-z7zsx\" (UID: \"706ee6ce-5cef-4fe6-847d-8745f7884423\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-z7zsx"
Sep 18 19:58:20.691905 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:20.691751    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/174ef0f9-1b9f-48b9-b6c1-1fb01a1f7b1b-snapshots\") pod \"insights-operator-6f9c644dd9-sfdnq\" (UID: \"174ef0f9-1b9f-48b9-b6c1-1fb01a1f7b1b\") " pod="openshift-insights/insights-operator-6f9c644dd9-sfdnq"
Sep 18 19:58:20.691905 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:20.691771    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/5c7b94cd-de57-4efe-a96c-1d349424b2fb-default-certificate\") pod \"router-default-7ff8bb4888-5sfkj\" (UID: \"5c7b94cd-de57-4efe-a96c-1d349424b2fb\") " pod="openshift-ingress/router-default-7ff8bb4888-5sfkj"
Sep 18 19:58:20.691905 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:20.691791    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/5c7b94cd-de57-4efe-a96c-1d349424b2fb-stats-auth\") pod \"router-default-7ff8bb4888-5sfkj\" (UID: \"5c7b94cd-de57-4efe-a96c-1d349424b2fb\") " pod="openshift-ingress/router-default-7ff8bb4888-5sfkj"
Sep 18 19:58:20.691905 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:20.691810    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c7b94cd-de57-4efe-a96c-1d349424b2fb-metrics-certs\") pod \"router-default-7ff8bb4888-5sfkj\" (UID: \"5c7b94cd-de57-4efe-a96c-1d349424b2fb\") " pod="openshift-ingress/router-default-7ff8bb4888-5sfkj"
Sep 18 19:58:20.691905 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:20.691884    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/174ef0f9-1b9f-48b9-b6c1-1fb01a1f7b1b-serving-cert\") pod \"insights-operator-6f9c644dd9-sfdnq\" (UID: \"174ef0f9-1b9f-48b9-b6c1-1fb01a1f7b1b\") " pod="openshift-insights/insights-operator-6f9c644dd9-sfdnq"
Sep 18 19:58:20.691905 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:20.691898    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9ss6\" (UniqueName: \"kubernetes.io/projected/5c7b94cd-de57-4efe-a96c-1d349424b2fb-kube-api-access-d9ss6\") pod \"router-default-7ff8bb4888-5sfkj\" (UID: \"5c7b94cd-de57-4efe-a96c-1d349424b2fb\") " pod="openshift-ingress/router-default-7ff8bb4888-5sfkj"
Sep 18 19:58:20.692105 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:20.691931    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/706ee6ce-5cef-4fe6-847d-8745f7884423-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-z7zsx\" (UID: \"706ee6ce-5cef-4fe6-847d-8745f7884423\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-z7zsx"
Sep 18 19:58:20.710121 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:20.710065    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-49kw5"]
Sep 18 19:58:20.711978 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:20.711963    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-49kw5"
Sep 18 19:58:20.720317 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:20.720300    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 18 19:58:20.720632 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:20.720617    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 18 19:58:20.720863 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:20.720848    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 18 19:58:20.721150 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:20.721133    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-zgknc\""
Sep 18 19:58:20.721204 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:20.721189    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 18 19:58:20.738113 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:20.738063    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-49kw5"]
Sep 18 19:58:20.792746 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:20.792719    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/174ef0f9-1b9f-48b9-b6c1-1fb01a1f7b1b-snapshots\") pod \"insights-operator-6f9c644dd9-sfdnq\" (UID: \"174ef0f9-1b9f-48b9-b6c1-1fb01a1f7b1b\") " pod="openshift-insights/insights-operator-6f9c644dd9-sfdnq"
Sep 18 19:58:20.792746 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:20.792747    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/5c7b94cd-de57-4efe-a96c-1d349424b2fb-default-certificate\") pod \"router-default-7ff8bb4888-5sfkj\" (UID: \"5c7b94cd-de57-4efe-a96c-1d349424b2fb\") " pod="openshift-ingress/router-default-7ff8bb4888-5sfkj"
Sep 18 19:58:20.792910 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:20.792768    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/5c7b94cd-de57-4efe-a96c-1d349424b2fb-stats-auth\") pod \"router-default-7ff8bb4888-5sfkj\" (UID: \"5c7b94cd-de57-4efe-a96c-1d349424b2fb\") " pod="openshift-ingress/router-default-7ff8bb4888-5sfkj"
Sep 18 19:58:20.792910 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:20.792786    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c7b94cd-de57-4efe-a96c-1d349424b2fb-metrics-certs\") pod \"router-default-7ff8bb4888-5sfkj\" (UID: \"5c7b94cd-de57-4efe-a96c-1d349424b2fb\") " pod="openshift-ingress/router-default-7ff8bb4888-5sfkj"
Sep 18 19:58:20.792910 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:20.792824    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/174ef0f9-1b9f-48b9-b6c1-1fb01a1f7b1b-serving-cert\") pod \"insights-operator-6f9c644dd9-sfdnq\" (UID: \"174ef0f9-1b9f-48b9-b6c1-1fb01a1f7b1b\") " pod="openshift-insights/insights-operator-6f9c644dd9-sfdnq"
Sep 18 19:58:20.792910 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:20.792846    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9ss6\" (UniqueName: \"kubernetes.io/projected/5c7b94cd-de57-4efe-a96c-1d349424b2fb-kube-api-access-d9ss6\") pod \"router-default-7ff8bb4888-5sfkj\" (UID: \"5c7b94cd-de57-4efe-a96c-1d349424b2fb\") " pod="openshift-ingress/router-default-7ff8bb4888-5sfkj"
Sep 18 19:58:20.792910 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:20.792872    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/96908402-98ec-4672-a484-d136b3546b7e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-49kw5\" (UID: \"96908402-98ec-4672-a484-d136b3546b7e\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-49kw5"
Sep 18 19:58:20.792910 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:58:20.792895    2549 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 18 19:58:20.793194 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:20.792898    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpt4m\" (UniqueName: \"kubernetes.io/projected/96908402-98ec-4672-a484-d136b3546b7e-kube-api-access-qpt4m\") pod \"cluster-monitoring-operator-847b8bd8d7-49kw5\" (UID: \"96908402-98ec-4672-a484-d136b3546b7e\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-49kw5"
Sep 18 19:58:20.793194 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:58:20.792955    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c7b94cd-de57-4efe-a96c-1d349424b2fb-metrics-certs podName:5c7b94cd-de57-4efe-a96c-1d349424b2fb nodeName:}" failed. No retries permitted until 2026-09-18 19:58:21.292936898 +0000 UTC m=+86.246678053 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5c7b94cd-de57-4efe-a96c-1d349424b2fb-metrics-certs") pod "router-default-7ff8bb4888-5sfkj" (UID: "5c7b94cd-de57-4efe-a96c-1d349424b2fb") : secret "router-metrics-certs-default" not found
Sep 18 19:58:20.793194 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:20.792995    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/706ee6ce-5cef-4fe6-847d-8745f7884423-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-z7zsx\" (UID: \"706ee6ce-5cef-4fe6-847d-8745f7884423\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-z7zsx"
Sep 18 19:58:20.793194 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:20.793020    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/96908402-98ec-4672-a484-d136b3546b7e-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-49kw5\" (UID: \"96908402-98ec-4672-a484-d136b3546b7e\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-49kw5"
Sep 18 19:58:20.793194 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:20.793065    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/174ef0f9-1b9f-48b9-b6c1-1fb01a1f7b1b-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-sfdnq\" (UID: \"174ef0f9-1b9f-48b9-b6c1-1fb01a1f7b1b\") " pod="openshift-insights/insights-operator-6f9c644dd9-sfdnq"
Sep 18 19:58:20.793194 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:20.793092    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl45h\" (UniqueName: \"kubernetes.io/projected/174ef0f9-1b9f-48b9-b6c1-1fb01a1f7b1b-kube-api-access-zl45h\") pod \"insights-operator-6f9c644dd9-sfdnq\" (UID: \"174ef0f9-1b9f-48b9-b6c1-1fb01a1f7b1b\") " pod="openshift-insights/insights-operator-6f9c644dd9-sfdnq"
Sep 18 19:58:20.793194 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:20.793147    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c7b94cd-de57-4efe-a96c-1d349424b2fb-service-ca-bundle\") pod \"router-default-7ff8bb4888-5sfkj\" (UID: \"5c7b94cd-de57-4efe-a96c-1d349424b2fb\") " pod="openshift-ingress/router-default-7ff8bb4888-5sfkj"
Sep 18 19:58:20.793194 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:58:20.793136    2549 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 18 19:58:20.793194 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:20.793181    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/174ef0f9-1b9f-48b9-b6c1-1fb01a1f7b1b-tmp\") pod \"insights-operator-6f9c644dd9-sfdnq\" (UID: \"174ef0f9-1b9f-48b9-b6c1-1fb01a1f7b1b\") " pod="openshift-insights/insights-operator-6f9c644dd9-sfdnq"
Sep 18 19:58:20.793676 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:58:20.793212    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/706ee6ce-5cef-4fe6-847d-8745f7884423-samples-operator-tls podName:706ee6ce-5cef-4fe6-847d-8745f7884423 nodeName:}" failed. No retries permitted until 2026-09-18 19:58:21.293196894 +0000 UTC m=+86.246938044 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/706ee6ce-5cef-4fe6-847d-8745f7884423-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-z7zsx" (UID: "706ee6ce-5cef-4fe6-847d-8745f7884423") : secret "samples-operator-tls" not found
Sep 18 19:58:20.793676 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:20.793245    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/174ef0f9-1b9f-48b9-b6c1-1fb01a1f7b1b-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-sfdnq\" (UID: \"174ef0f9-1b9f-48b9-b6c1-1fb01a1f7b1b\") " pod="openshift-insights/insights-operator-6f9c644dd9-sfdnq"
Sep 18 19:58:20.793676 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:20.793271    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wdfh\" (UniqueName: \"kubernetes.io/projected/706ee6ce-5cef-4fe6-847d-8745f7884423-kube-api-access-4wdfh\") pod \"cluster-samples-operator-685d7d68b5-z7zsx\" (UID: \"706ee6ce-5cef-4fe6-847d-8745f7884423\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-z7zsx"
Sep 18 19:58:20.793676 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:58:20.793435    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5c7b94cd-de57-4efe-a96c-1d349424b2fb-service-ca-bundle podName:5c7b94cd-de57-4efe-a96c-1d349424b2fb nodeName:}" failed. No retries permitted until 2026-09-18 19:58:21.293417685 +0000 UTC m=+86.247158848 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5c7b94cd-de57-4efe-a96c-1d349424b2fb-service-ca-bundle") pod "router-default-7ff8bb4888-5sfkj" (UID: "5c7b94cd-de57-4efe-a96c-1d349424b2fb") : configmap references non-existent config key: service-ca.crt
Sep 18 19:58:20.793676 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:20.793478    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/174ef0f9-1b9f-48b9-b6c1-1fb01a1f7b1b-snapshots\") pod \"insights-operator-6f9c644dd9-sfdnq\" (UID: \"174ef0f9-1b9f-48b9-b6c1-1fb01a1f7b1b\") " pod="openshift-insights/insights-operator-6f9c644dd9-sfdnq"
Sep 18 19:58:20.793676 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:20.793485    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/174ef0f9-1b9f-48b9-b6c1-1fb01a1f7b1b-tmp\") pod \"insights-operator-6f9c644dd9-sfdnq\" (UID: \"174ef0f9-1b9f-48b9-b6c1-1fb01a1f7b1b\") " pod="openshift-insights/insights-operator-6f9c644dd9-sfdnq"
Sep 18 19:58:20.793918 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:20.793729    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/174ef0f9-1b9f-48b9-b6c1-1fb01a1f7b1b-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-sfdnq\" (UID: \"174ef0f9-1b9f-48b9-b6c1-1fb01a1f7b1b\") " pod="openshift-insights/insights-operator-6f9c644dd9-sfdnq"
Sep 18 19:58:20.794252 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:20.794234    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/174ef0f9-1b9f-48b9-b6c1-1fb01a1f7b1b-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-sfdnq\" (UID: \"174ef0f9-1b9f-48b9-b6c1-1fb01a1f7b1b\") " pod="openshift-insights/insights-operator-6f9c644dd9-sfdnq"
Sep 18 19:58:20.795181 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:20.795163    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/174ef0f9-1b9f-48b9-b6c1-1fb01a1f7b1b-serving-cert\") pod \"insights-operator-6f9c644dd9-sfdnq\" (UID: \"174ef0f9-1b9f-48b9-b6c1-1fb01a1f7b1b\") " pod="openshift-insights/insights-operator-6f9c644dd9-sfdnq"
Sep 18 19:58:20.795656 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:20.795640    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/5c7b94cd-de57-4efe-a96c-1d349424b2fb-stats-auth\") pod \"router-default-7ff8bb4888-5sfkj\" (UID: \"5c7b94cd-de57-4efe-a96c-1d349424b2fb\") " pod="openshift-ingress/router-default-7ff8bb4888-5sfkj"
Sep 18 19:58:20.795715 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:20.795658    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/5c7b94cd-de57-4efe-a96c-1d349424b2fb-default-certificate\") pod \"router-default-7ff8bb4888-5sfkj\" (UID: \"5c7b94cd-de57-4efe-a96c-1d349424b2fb\") " pod="openshift-ingress/router-default-7ff8bb4888-5sfkj"
Sep 18 19:58:20.797447 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:20.797430    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-whdp6"
Sep 18 19:58:20.802728 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:20.802700    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wdfh\" (UniqueName: \"kubernetes.io/projected/706ee6ce-5cef-4fe6-847d-8745f7884423-kube-api-access-4wdfh\") pod \"cluster-samples-operator-685d7d68b5-z7zsx\" (UID: \"706ee6ce-5cef-4fe6-847d-8745f7884423\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-z7zsx"
Sep 18 19:58:20.803225 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:20.803203    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9ss6\" (UniqueName: \"kubernetes.io/projected/5c7b94cd-de57-4efe-a96c-1d349424b2fb-kube-api-access-d9ss6\") pod \"router-default-7ff8bb4888-5sfkj\" (UID: \"5c7b94cd-de57-4efe-a96c-1d349424b2fb\") " pod="openshift-ingress/router-default-7ff8bb4888-5sfkj"
Sep 18 19:58:20.803302 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:20.803280    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl45h\" (UniqueName: \"kubernetes.io/projected/174ef0f9-1b9f-48b9-b6c1-1fb01a1f7b1b-kube-api-access-zl45h\") pod \"insights-operator-6f9c644dd9-sfdnq\" (UID: \"174ef0f9-1b9f-48b9-b6c1-1fb01a1f7b1b\") " pod="openshift-insights/insights-operator-6f9c644dd9-sfdnq"
Sep 18 19:58:20.894598 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:20.894552    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/96908402-98ec-4672-a484-d136b3546b7e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-49kw5\" (UID: \"96908402-98ec-4672-a484-d136b3546b7e\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-49kw5"
Sep 18 19:58:20.894740 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:20.894622    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpt4m\" (UniqueName: \"kubernetes.io/projected/96908402-98ec-4672-a484-d136b3546b7e-kube-api-access-qpt4m\") pod \"cluster-monitoring-operator-847b8bd8d7-49kw5\" (UID: \"96908402-98ec-4672-a484-d136b3546b7e\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-49kw5"
Sep 18 19:58:20.894740 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:20.894691    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/96908402-98ec-4672-a484-d136b3546b7e-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-49kw5\" (UID: \"96908402-98ec-4672-a484-d136b3546b7e\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-49kw5"
Sep 18 19:58:20.895596 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:20.895572    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/96908402-98ec-4672-a484-d136b3546b7e-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-49kw5\" (UID: \"96908402-98ec-4672-a484-d136b3546b7e\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-49kw5"
Sep 18 19:58:20.895725 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:58:20.895707    2549 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 19:58:20.895789 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:58:20.895779    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96908402-98ec-4672-a484-d136b3546b7e-cluster-monitoring-operator-tls podName:96908402-98ec-4672-a484-d136b3546b7e nodeName:}" failed. No retries permitted until 2026-09-18 19:58:21.395761528 +0000 UTC m=+86.349502689 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/96908402-98ec-4672-a484-d136b3546b7e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-49kw5" (UID: "96908402-98ec-4672-a484-d136b3546b7e") : secret "cluster-monitoring-operator-tls" not found
Sep 18 19:58:20.905451 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:20.905426    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpt4m\" (UniqueName: \"kubernetes.io/projected/96908402-98ec-4672-a484-d136b3546b7e-kube-api-access-qpt4m\") pod \"cluster-monitoring-operator-847b8bd8d7-49kw5\" (UID: \"96908402-98ec-4672-a484-d136b3546b7e\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-49kw5"
Sep 18 19:58:20.909235 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:20.909219    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-sfdnq"
Sep 18 19:58:21.064893 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:21.064826    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-whdp6"]
Sep 18 19:58:21.067886 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:58:21.067859    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddaf88b0_c4ce_42ac_a06a_e95db6854eab.slice/crio-c03a1091e452cd9fbb4286d4cc776a91dde7ff80d15cb433677915b2692462f5 WatchSource:0}: Error finding container c03a1091e452cd9fbb4286d4cc776a91dde7ff80d15cb433677915b2692462f5: Status 404 returned error can't find the container with id c03a1091e452cd9fbb4286d4cc776a91dde7ff80d15cb433677915b2692462f5
Sep 18 19:58:21.174578 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:21.174489    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-sfdnq"]
Sep 18 19:58:21.178132 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:58:21.178105    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod174ef0f9_1b9f_48b9_b6c1_1fb01a1f7b1b.slice/crio-07dc4c5aa2f434f8e34d4fecb469217126dee3a91decf8ddc3522a3d0b757a8d WatchSource:0}: Error finding container 07dc4c5aa2f434f8e34d4fecb469217126dee3a91decf8ddc3522a3d0b757a8d: Status 404 returned error can't find the container with id 07dc4c5aa2f434f8e34d4fecb469217126dee3a91decf8ddc3522a3d0b757a8d
Sep 18 19:58:21.298889 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:21.298858    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c7b94cd-de57-4efe-a96c-1d349424b2fb-metrics-certs\") pod \"router-default-7ff8bb4888-5sfkj\" (UID: \"5c7b94cd-de57-4efe-a96c-1d349424b2fb\") " pod="openshift-ingress/router-default-7ff8bb4888-5sfkj"
Sep 18 19:58:21.299035 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:21.298922    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/706ee6ce-5cef-4fe6-847d-8745f7884423-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-z7zsx\" (UID: \"706ee6ce-5cef-4fe6-847d-8745f7884423\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-z7zsx"
Sep 18 19:58:21.299035 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:58:21.299013    2549 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 18 19:58:21.299102 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:21.299062    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c7b94cd-de57-4efe-a96c-1d349424b2fb-service-ca-bundle\") pod \"router-default-7ff8bb4888-5sfkj\" (UID: \"5c7b94cd-de57-4efe-a96c-1d349424b2fb\") " pod="openshift-ingress/router-default-7ff8bb4888-5sfkj"
Sep 18 19:58:21.299102 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:58:21.299085    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/706ee6ce-5cef-4fe6-847d-8745f7884423-samples-operator-tls podName:706ee6ce-5cef-4fe6-847d-8745f7884423 nodeName:}" failed. No retries permitted until 2026-09-18 19:58:22.29906783 +0000 UTC m=+87.252808983 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/706ee6ce-5cef-4fe6-847d-8745f7884423-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-z7zsx" (UID: "706ee6ce-5cef-4fe6-847d-8745f7884423") : secret "samples-operator-tls" not found
Sep 18 19:58:21.299169 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:58:21.299013    2549 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 18 19:58:21.299169 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:58:21.299132    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5c7b94cd-de57-4efe-a96c-1d349424b2fb-service-ca-bundle podName:5c7b94cd-de57-4efe-a96c-1d349424b2fb nodeName:}" failed. No retries permitted until 2026-09-18 19:58:22.299122159 +0000 UTC m=+87.252863312 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5c7b94cd-de57-4efe-a96c-1d349424b2fb-service-ca-bundle") pod "router-default-7ff8bb4888-5sfkj" (UID: "5c7b94cd-de57-4efe-a96c-1d349424b2fb") : configmap references non-existent config key: service-ca.crt
Sep 18 19:58:21.299233 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:58:21.299187    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c7b94cd-de57-4efe-a96c-1d349424b2fb-metrics-certs podName:5c7b94cd-de57-4efe-a96c-1d349424b2fb nodeName:}" failed. No retries permitted until 2026-09-18 19:58:22.299166551 +0000 UTC m=+87.252907703 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5c7b94cd-de57-4efe-a96c-1d349424b2fb-metrics-certs") pod "router-default-7ff8bb4888-5sfkj" (UID: "5c7b94cd-de57-4efe-a96c-1d349424b2fb") : secret "router-metrics-certs-default" not found
Sep 18 19:58:21.399507 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:21.399477    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/96908402-98ec-4672-a484-d136b3546b7e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-49kw5\" (UID: \"96908402-98ec-4672-a484-d136b3546b7e\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-49kw5"
Sep 18 19:58:21.399648 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:58:21.399619    2549 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 19:58:21.399688 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:58:21.399677    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96908402-98ec-4672-a484-d136b3546b7e-cluster-monitoring-operator-tls podName:96908402-98ec-4672-a484-d136b3546b7e nodeName:}" failed. No retries permitted until 2026-09-18 19:58:22.399660443 +0000 UTC m=+87.353401600 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/96908402-98ec-4672-a484-d136b3546b7e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-49kw5" (UID: "96908402-98ec-4672-a484-d136b3546b7e") : secret "cluster-monitoring-operator-tls" not found
Sep 18 19:58:21.946637 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:21.946592    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-whdp6" event={"ID":"ddaf88b0-c4ce-42ac-a06a-e95db6854eab","Type":"ContainerStarted","Data":"c03a1091e452cd9fbb4286d4cc776a91dde7ff80d15cb433677915b2692462f5"}
Sep 18 19:58:21.948136 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:21.948104    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-sfdnq" event={"ID":"174ef0f9-1b9f-48b9-b6c1-1fb01a1f7b1b","Type":"ContainerStarted","Data":"07dc4c5aa2f434f8e34d4fecb469217126dee3a91decf8ddc3522a3d0b757a8d"}
Sep 18 19:58:22.308435 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:22.308399    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c7b94cd-de57-4efe-a96c-1d349424b2fb-metrics-certs\") pod \"router-default-7ff8bb4888-5sfkj\" (UID: \"5c7b94cd-de57-4efe-a96c-1d349424b2fb\") " pod="openshift-ingress/router-default-7ff8bb4888-5sfkj"
Sep 18 19:58:22.308592 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:22.308461    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/706ee6ce-5cef-4fe6-847d-8745f7884423-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-z7zsx\" (UID: \"706ee6ce-5cef-4fe6-847d-8745f7884423\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-z7zsx"
Sep 18 19:58:22.308592 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:58:22.308544    2549 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 18 19:58:22.308592 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:58:22.308555    2549 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 18 19:58:22.308592 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:58:22.308590    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/706ee6ce-5cef-4fe6-847d-8745f7884423-samples-operator-tls podName:706ee6ce-5cef-4fe6-847d-8745f7884423 nodeName:}" failed. No retries permitted until 2026-09-18 19:58:24.30857725 +0000 UTC m=+89.262318409 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/706ee6ce-5cef-4fe6-847d-8745f7884423-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-z7zsx" (UID: "706ee6ce-5cef-4fe6-847d-8745f7884423") : secret "samples-operator-tls" not found
Sep 18 19:58:22.308818 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:58:22.308620    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c7b94cd-de57-4efe-a96c-1d349424b2fb-metrics-certs podName:5c7b94cd-de57-4efe-a96c-1d349424b2fb nodeName:}" failed. No retries permitted until 2026-09-18 19:58:24.308598835 +0000 UTC m=+89.262339999 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5c7b94cd-de57-4efe-a96c-1d349424b2fb-metrics-certs") pod "router-default-7ff8bb4888-5sfkj" (UID: "5c7b94cd-de57-4efe-a96c-1d349424b2fb") : secret "router-metrics-certs-default" not found
Sep 18 19:58:22.308818 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:22.308696    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c7b94cd-de57-4efe-a96c-1d349424b2fb-service-ca-bundle\") pod \"router-default-7ff8bb4888-5sfkj\" (UID: \"5c7b94cd-de57-4efe-a96c-1d349424b2fb\") " pod="openshift-ingress/router-default-7ff8bb4888-5sfkj"
Sep 18 19:58:22.308918 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:58:22.308819    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5c7b94cd-de57-4efe-a96c-1d349424b2fb-service-ca-bundle podName:5c7b94cd-de57-4efe-a96c-1d349424b2fb nodeName:}" failed. No retries permitted until 2026-09-18 19:58:24.308806053 +0000 UTC m=+89.262547214 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5c7b94cd-de57-4efe-a96c-1d349424b2fb-service-ca-bundle") pod "router-default-7ff8bb4888-5sfkj" (UID: "5c7b94cd-de57-4efe-a96c-1d349424b2fb") : configmap references non-existent config key: service-ca.crt
Sep 18 19:58:22.409171 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:22.409140    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/96908402-98ec-4672-a484-d136b3546b7e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-49kw5\" (UID: \"96908402-98ec-4672-a484-d136b3546b7e\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-49kw5"
Sep 18 19:58:22.409323 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:58:22.409274    2549 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 19:58:22.409369 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:58:22.409336    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96908402-98ec-4672-a484-d136b3546b7e-cluster-monitoring-operator-tls podName:96908402-98ec-4672-a484-d136b3546b7e nodeName:}" failed. No retries permitted until 2026-09-18 19:58:24.409321273 +0000 UTC m=+89.363062424 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/96908402-98ec-4672-a484-d136b3546b7e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-49kw5" (UID: "96908402-98ec-4672-a484-d136b3546b7e") : secret "cluster-monitoring-operator-tls" not found
Sep 18 19:58:22.952251 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:22.952219    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-whdp6" event={"ID":"ddaf88b0-c4ce-42ac-a06a-e95db6854eab","Type":"ContainerStarted","Data":"33c8c82e73b6ca9dc7c25fef16d5a74004f55af30f5fba234a02be75493d868f"}
Sep 18 19:58:22.973804 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:22.973761    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-whdp6" podStartSLOduration=1.904795648 podStartE2EDuration="2.973746093s" totalImagesPullingTime="1.068950445s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:58:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:58:22.973080449 +0000 UTC m=+87.926821618" watchObservedRunningTime="2026-09-18 19:58:22.973746093 +0000 UTC m=+87.927487298"
Sep 18 19:58:23.940846 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:23.940814    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-6fm6j"]
Sep 18 19:58:23.942833 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:23.942817    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6fm6j"
Sep 18 19:58:23.948494 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:23.948467    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 18 19:58:23.949264 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:23.949241    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-dnrhk\""
Sep 18 19:58:23.950532 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:23.950513    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 18 19:58:23.957204 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:23.957177    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-sfdnq" event={"ID":"174ef0f9-1b9f-48b9-b6c1-1fb01a1f7b1b","Type":"ContainerStarted","Data":"485d9c5b2ddaea18ad1b2a8d38d09d1fb9f41c7d77307efb7e83f5563124f7a9"}
Sep 18 19:58:23.957584 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:23.957473    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6fm6j"]
Sep 18 19:58:24.021831 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:24.021807    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0a2a67f2-15ec-472e-95a5-79667b3a8eef-data-volume\") pod \"insights-runtime-extractor-6fm6j\" (UID: \"0a2a67f2-15ec-472e-95a5-79667b3a8eef\") " pod="openshift-insights/insights-runtime-extractor-6fm6j"
Sep 18 19:58:24.021954 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:24.021919    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0a2a67f2-15ec-472e-95a5-79667b3a8eef-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6fm6j\" (UID: \"0a2a67f2-15ec-472e-95a5-79667b3a8eef\") " pod="openshift-insights/insights-runtime-extractor-6fm6j"
Sep 18 19:58:24.022020 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:24.021977    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a2a67f2-15ec-472e-95a5-79667b3a8eef-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6fm6j\" (UID: \"0a2a67f2-15ec-472e-95a5-79667b3a8eef\") " pod="openshift-insights/insights-runtime-extractor-6fm6j"
Sep 18 19:58:24.022020 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:24.022010    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ss7hq\" (UniqueName: \"kubernetes.io/projected/0a2a67f2-15ec-472e-95a5-79667b3a8eef-kube-api-access-ss7hq\") pod \"insights-runtime-extractor-6fm6j\" (UID: \"0a2a67f2-15ec-472e-95a5-79667b3a8eef\") " pod="openshift-insights/insights-runtime-extractor-6fm6j"
Sep 18 19:58:24.022127 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:24.022057    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0a2a67f2-15ec-472e-95a5-79667b3a8eef-crio-socket\") pod \"insights-runtime-extractor-6fm6j\" (UID: \"0a2a67f2-15ec-472e-95a5-79667b3a8eef\") " pod="openshift-insights/insights-runtime-extractor-6fm6j"
Sep 18 19:58:24.026661 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:24.026613    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-6f9c644dd9-sfdnq" podStartSLOduration=2.221079661 podStartE2EDuration="4.026597911s" totalImagesPullingTime="1.80551825s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:58:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:58:24.026515088 +0000 UTC m=+88.980256257" watchObservedRunningTime="2026-09-18 19:58:24.026597911 +0000 UTC m=+88.980339415"
Sep 18 19:58:24.123152 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:24.123121    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0a2a67f2-15ec-472e-95a5-79667b3a8eef-crio-socket\") pod \"insights-runtime-extractor-6fm6j\" (UID: \"0a2a67f2-15ec-472e-95a5-79667b3a8eef\") " pod="openshift-insights/insights-runtime-extractor-6fm6j"
Sep 18 19:58:24.123322 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:24.123201    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0a2a67f2-15ec-472e-95a5-79667b3a8eef-data-volume\") pod \"insights-runtime-extractor-6fm6j\" (UID: \"0a2a67f2-15ec-472e-95a5-79667b3a8eef\") " pod="openshift-insights/insights-runtime-extractor-6fm6j"
Sep 18 19:58:24.123322 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:24.123259    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0a2a67f2-15ec-472e-95a5-79667b3a8eef-crio-socket\") pod \"insights-runtime-extractor-6fm6j\" (UID: \"0a2a67f2-15ec-472e-95a5-79667b3a8eef\") " pod="openshift-insights/insights-runtime-extractor-6fm6j"
Sep 18 19:58:24.123322 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:24.123303    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0a2a67f2-15ec-472e-95a5-79667b3a8eef-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6fm6j\" (UID: \"0a2a67f2-15ec-472e-95a5-79667b3a8eef\") " pod="openshift-insights/insights-runtime-extractor-6fm6j"
Sep 18 19:58:24.123508 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:24.123335    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a2a67f2-15ec-472e-95a5-79667b3a8eef-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6fm6j\" (UID: \"0a2a67f2-15ec-472e-95a5-79667b3a8eef\") " pod="openshift-insights/insights-runtime-extractor-6fm6j"
Sep 18 19:58:24.123508 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:24.123354    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss7hq\" (UniqueName: \"kubernetes.io/projected/0a2a67f2-15ec-472e-95a5-79667b3a8eef-kube-api-access-ss7hq\") pod \"insights-runtime-extractor-6fm6j\" (UID: \"0a2a67f2-15ec-472e-95a5-79667b3a8eef\") " pod="openshift-insights/insights-runtime-extractor-6fm6j"
Sep 18 19:58:24.123508 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:58:24.123441    2549 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 19:58:24.123508 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:58:24.123490    2549 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:58:24.123676 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:58:24.123497    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0a2a67f2-15ec-472e-95a5-79667b3a8eef-kube-rbac-proxy-cm podName:0a2a67f2-15ec-472e-95a5-79667b3a8eef nodeName:}" failed. No retries permitted until 2026-09-18 19:58:24.623481417 +0000 UTC m=+89.577222563 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0a2a67f2-15ec-472e-95a5-79667b3a8eef-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6fm6j" (UID: "0a2a67f2-15ec-472e-95a5-79667b3a8eef") : configmap "kube-rbac-proxy" not found
Sep 18 19:58:24.123676 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:58:24.123554    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a2a67f2-15ec-472e-95a5-79667b3a8eef-insights-runtime-extractor-tls podName:0a2a67f2-15ec-472e-95a5-79667b3a8eef nodeName:}" failed. No retries permitted until 2026-09-18 19:58:24.623537957 +0000 UTC m=+89.577279104 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0a2a67f2-15ec-472e-95a5-79667b3a8eef-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6fm6j" (UID: "0a2a67f2-15ec-472e-95a5-79667b3a8eef") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:58:24.123676 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:24.123580    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0a2a67f2-15ec-472e-95a5-79667b3a8eef-data-volume\") pod \"insights-runtime-extractor-6fm6j\" (UID: \"0a2a67f2-15ec-472e-95a5-79667b3a8eef\") " pod="openshift-insights/insights-runtime-extractor-6fm6j"
Sep 18 19:58:24.135257 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:24.135229    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ss7hq\" (UniqueName: \"kubernetes.io/projected/0a2a67f2-15ec-472e-95a5-79667b3a8eef-kube-api-access-ss7hq\") pod \"insights-runtime-extractor-6fm6j\" (UID: \"0a2a67f2-15ec-472e-95a5-79667b3a8eef\") " pod="openshift-insights/insights-runtime-extractor-6fm6j"
Sep 18 19:58:24.325210 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:24.325181    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c7b94cd-de57-4efe-a96c-1d349424b2fb-metrics-certs\") pod \"router-default-7ff8bb4888-5sfkj\" (UID: \"5c7b94cd-de57-4efe-a96c-1d349424b2fb\") " pod="openshift-ingress/router-default-7ff8bb4888-5sfkj"
Sep 18 19:58:24.325368 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:24.325254    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/706ee6ce-5cef-4fe6-847d-8745f7884423-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-z7zsx\" (UID: \"706ee6ce-5cef-4fe6-847d-8745f7884423\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-z7zsx"
Sep 18 19:58:24.325368 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:58:24.325326    2549 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 18 19:58:24.325368 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:24.325338    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c7b94cd-de57-4efe-a96c-1d349424b2fb-service-ca-bundle\") pod \"router-default-7ff8bb4888-5sfkj\" (UID: \"5c7b94cd-de57-4efe-a96c-1d349424b2fb\") " pod="openshift-ingress/router-default-7ff8bb4888-5sfkj"
Sep 18 19:58:24.325543 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:58:24.325404    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c7b94cd-de57-4efe-a96c-1d349424b2fb-metrics-certs podName:5c7b94cd-de57-4efe-a96c-1d349424b2fb nodeName:}" failed. No retries permitted until 2026-09-18 19:58:28.325374776 +0000 UTC m=+93.279115922 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5c7b94cd-de57-4efe-a96c-1d349424b2fb-metrics-certs") pod "router-default-7ff8bb4888-5sfkj" (UID: "5c7b94cd-de57-4efe-a96c-1d349424b2fb") : secret "router-metrics-certs-default" not found
Sep 18 19:58:24.325543 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:58:24.325402    2549 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 18 19:58:24.325543 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:58:24.325431    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/706ee6ce-5cef-4fe6-847d-8745f7884423-samples-operator-tls podName:706ee6ce-5cef-4fe6-847d-8745f7884423 nodeName:}" failed. No retries permitted until 2026-09-18 19:58:28.325425502 +0000 UTC m=+93.279166649 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/706ee6ce-5cef-4fe6-847d-8745f7884423-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-z7zsx" (UID: "706ee6ce-5cef-4fe6-847d-8745f7884423") : secret "samples-operator-tls" not found
Sep 18 19:58:24.325543 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:58:24.325500    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5c7b94cd-de57-4efe-a96c-1d349424b2fb-service-ca-bundle podName:5c7b94cd-de57-4efe-a96c-1d349424b2fb nodeName:}" failed. No retries permitted until 2026-09-18 19:58:28.325482989 +0000 UTC m=+93.279224150 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5c7b94cd-de57-4efe-a96c-1d349424b2fb-service-ca-bundle") pod "router-default-7ff8bb4888-5sfkj" (UID: "5c7b94cd-de57-4efe-a96c-1d349424b2fb") : configmap references non-existent config key: service-ca.crt
Sep 18 19:58:24.425833 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:24.425803    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/96908402-98ec-4672-a484-d136b3546b7e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-49kw5\" (UID: \"96908402-98ec-4672-a484-d136b3546b7e\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-49kw5"
Sep 18 19:58:24.425993 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:58:24.425953    2549 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 19:58:24.426059 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:58:24.426027    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96908402-98ec-4672-a484-d136b3546b7e-cluster-monitoring-operator-tls podName:96908402-98ec-4672-a484-d136b3546b7e nodeName:}" failed. No retries permitted until 2026-09-18 19:58:28.426009633 +0000 UTC m=+93.379750794 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/96908402-98ec-4672-a484-d136b3546b7e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-49kw5" (UID: "96908402-98ec-4672-a484-d136b3546b7e") : secret "cluster-monitoring-operator-tls" not found
Sep 18 19:58:24.627469 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:24.627378    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0a2a67f2-15ec-472e-95a5-79667b3a8eef-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6fm6j\" (UID: \"0a2a67f2-15ec-472e-95a5-79667b3a8eef\") " pod="openshift-insights/insights-runtime-extractor-6fm6j"
Sep 18 19:58:24.627469 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:24.627433    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a2a67f2-15ec-472e-95a5-79667b3a8eef-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6fm6j\" (UID: \"0a2a67f2-15ec-472e-95a5-79667b3a8eef\") " pod="openshift-insights/insights-runtime-extractor-6fm6j"
Sep 18 19:58:24.627679 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:58:24.627505    2549 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 19:58:24.627679 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:58:24.627516    2549 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:58:24.627679 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:58:24.627574    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0a2a67f2-15ec-472e-95a5-79667b3a8eef-kube-rbac-proxy-cm podName:0a2a67f2-15ec-472e-95a5-79667b3a8eef nodeName:}" failed. No retries permitted until 2026-09-18 19:58:25.627555404 +0000 UTC m=+90.581296565 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0a2a67f2-15ec-472e-95a5-79667b3a8eef-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6fm6j" (UID: "0a2a67f2-15ec-472e-95a5-79667b3a8eef") : configmap "kube-rbac-proxy" not found
Sep 18 19:58:24.627679 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:58:24.627611    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a2a67f2-15ec-472e-95a5-79667b3a8eef-insights-runtime-extractor-tls podName:0a2a67f2-15ec-472e-95a5-79667b3a8eef nodeName:}" failed. No retries permitted until 2026-09-18 19:58:25.627598576 +0000 UTC m=+90.581339722 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0a2a67f2-15ec-472e-95a5-79667b3a8eef-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6fm6j" (UID: "0a2a67f2-15ec-472e-95a5-79667b3a8eef") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:58:25.635800 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:25.635770    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a2a67f2-15ec-472e-95a5-79667b3a8eef-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6fm6j\" (UID: \"0a2a67f2-15ec-472e-95a5-79667b3a8eef\") " pod="openshift-insights/insights-runtime-extractor-6fm6j"
Sep 18 19:58:25.636184 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:25.635856    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0a2a67f2-15ec-472e-95a5-79667b3a8eef-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6fm6j\" (UID: \"0a2a67f2-15ec-472e-95a5-79667b3a8eef\") " pod="openshift-insights/insights-runtime-extractor-6fm6j"
Sep 18 19:58:25.636184 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:58:25.635909    2549 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 19:58:25.636184 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:58:25.635909    2549 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:58:25.636184 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:58:25.635954    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0a2a67f2-15ec-472e-95a5-79667b3a8eef-kube-rbac-proxy-cm podName:0a2a67f2-15ec-472e-95a5-79667b3a8eef nodeName:}" failed. No retries permitted until 2026-09-18 19:58:27.635941414 +0000 UTC m=+92.589682561 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0a2a67f2-15ec-472e-95a5-79667b3a8eef-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6fm6j" (UID: "0a2a67f2-15ec-472e-95a5-79667b3a8eef") : configmap "kube-rbac-proxy" not found
Sep 18 19:58:25.636184 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:58:25.635966    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a2a67f2-15ec-472e-95a5-79667b3a8eef-insights-runtime-extractor-tls podName:0a2a67f2-15ec-472e-95a5-79667b3a8eef nodeName:}" failed. No retries permitted until 2026-09-18 19:58:27.635960522 +0000 UTC m=+92.589701669 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0a2a67f2-15ec-472e-95a5-79667b3a8eef-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6fm6j" (UID: "0a2a67f2-15ec-472e-95a5-79667b3a8eef") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:58:26.021885 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:26.021814    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-t26hd_525400e6-6678-424f-a1ba-0896aac41264/dns-node-resolver/0.log"
Sep 18 19:58:26.822444 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:26.822418    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-m6qj6_cb6d3f0d-bc38-43aa-8810-3b14a624295d/node-ca/0.log"
Sep 18 19:58:27.467462 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:27.467430    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-wshq5"]
Sep 18 19:58:27.469374 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:27.469353    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-wshq5"
Sep 18 19:58:27.473638 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:27.473609    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 18 19:58:27.476318 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:27.473765    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 18 19:58:27.476318 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:27.473899    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-pq2pl\""
Sep 18 19:58:27.479463 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:27.479440    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-wshq5"]
Sep 18 19:58:27.552706 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:27.552678    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22r49\" (UniqueName: \"kubernetes.io/projected/abb6ed43-070c-4bc5-af80-ee32996c9a2e-kube-api-access-22r49\") pod \"migrator-76794cbd6c-wshq5\" (UID: \"abb6ed43-070c-4bc5-af80-ee32996c9a2e\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-wshq5"
Sep 18 19:58:27.652948 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:27.652919    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a2a67f2-15ec-472e-95a5-79667b3a8eef-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6fm6j\" (UID: \"0a2a67f2-15ec-472e-95a5-79667b3a8eef\") " pod="openshift-insights/insights-runtime-extractor-6fm6j"
Sep 18 19:58:27.653061 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:27.652963    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0a2a67f2-15ec-472e-95a5-79667b3a8eef-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6fm6j\" (UID: \"0a2a67f2-15ec-472e-95a5-79667b3a8eef\") " pod="openshift-insights/insights-runtime-extractor-6fm6j"
Sep 18 19:58:27.653061 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:27.653046    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22r49\" (UniqueName: \"kubernetes.io/projected/abb6ed43-070c-4bc5-af80-ee32996c9a2e-kube-api-access-22r49\") pod \"migrator-76794cbd6c-wshq5\" (UID: \"abb6ed43-070c-4bc5-af80-ee32996c9a2e\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-wshq5"
Sep 18 19:58:27.653127 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:58:27.653071    2549 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:58:27.653127 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:58:27.653108    2549 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 19:58:27.653187 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:58:27.653127    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a2a67f2-15ec-472e-95a5-79667b3a8eef-insights-runtime-extractor-tls podName:0a2a67f2-15ec-472e-95a5-79667b3a8eef nodeName:}" failed. No retries permitted until 2026-09-18 19:58:31.653109864 +0000 UTC m=+96.606851023 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0a2a67f2-15ec-472e-95a5-79667b3a8eef-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6fm6j" (UID: "0a2a67f2-15ec-472e-95a5-79667b3a8eef") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:58:27.653187 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:58:27.653165    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0a2a67f2-15ec-472e-95a5-79667b3a8eef-kube-rbac-proxy-cm podName:0a2a67f2-15ec-472e-95a5-79667b3a8eef nodeName:}" failed. No retries permitted until 2026-09-18 19:58:31.653151998 +0000 UTC m=+96.606893144 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0a2a67f2-15ec-472e-95a5-79667b3a8eef-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6fm6j" (UID: "0a2a67f2-15ec-472e-95a5-79667b3a8eef") : configmap "kube-rbac-proxy" not found
Sep 18 19:58:27.665360 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:27.665332    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22r49\" (UniqueName: \"kubernetes.io/projected/abb6ed43-070c-4bc5-af80-ee32996c9a2e-kube-api-access-22r49\") pod \"migrator-76794cbd6c-wshq5\" (UID: \"abb6ed43-070c-4bc5-af80-ee32996c9a2e\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-wshq5"
Sep 18 19:58:27.779669 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:27.779595    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-wshq5"
Sep 18 19:58:27.822507 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:27.822460    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xrlbt_605e3d2c-3e1c-48a4-b02a-77bedbaa6a4c/serve-healthcheck-canary/0.log"
Sep 18 19:58:28.046139 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:28.046059    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-wshq5"]
Sep 18 19:58:28.049582 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:58:28.049556    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabb6ed43_070c_4bc5_af80_ee32996c9a2e.slice/crio-4c08748e0f2b1ece2d5d978576fa14d2530bdf8d2cfc36ce11a48d20018a76fa WatchSource:0}: Error finding container 4c08748e0f2b1ece2d5d978576fa14d2530bdf8d2cfc36ce11a48d20018a76fa: Status 404 returned error can't find the container with id 4c08748e0f2b1ece2d5d978576fa14d2530bdf8d2cfc36ce11a48d20018a76fa
Sep 18 19:58:28.357714 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:28.357681    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/706ee6ce-5cef-4fe6-847d-8745f7884423-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-z7zsx\" (UID: \"706ee6ce-5cef-4fe6-847d-8745f7884423\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-z7zsx"
Sep 18 19:58:28.357881 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:28.357728    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c7b94cd-de57-4efe-a96c-1d349424b2fb-service-ca-bundle\") pod \"router-default-7ff8bb4888-5sfkj\" (UID: \"5c7b94cd-de57-4efe-a96c-1d349424b2fb\") " pod="openshift-ingress/router-default-7ff8bb4888-5sfkj"
Sep 18 19:58:28.357881 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:28.357797    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c7b94cd-de57-4efe-a96c-1d349424b2fb-metrics-certs\") pod \"router-default-7ff8bb4888-5sfkj\" (UID: \"5c7b94cd-de57-4efe-a96c-1d349424b2fb\") " pod="openshift-ingress/router-default-7ff8bb4888-5sfkj"
Sep 18 19:58:28.357881 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:58:28.357831    2549 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 18 19:58:28.357984 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:58:28.357897    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/706ee6ce-5cef-4fe6-847d-8745f7884423-samples-operator-tls podName:706ee6ce-5cef-4fe6-847d-8745f7884423 nodeName:}" failed. No retries permitted until 2026-09-18 19:58:36.35788206 +0000 UTC m=+101.311623212 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/706ee6ce-5cef-4fe6-847d-8745f7884423-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-z7zsx" (UID: "706ee6ce-5cef-4fe6-847d-8745f7884423") : secret "samples-operator-tls" not found
Sep 18 19:58:28.357984 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:58:28.357901    2549 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 18 19:58:28.357984 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:58:28.357913    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5c7b94cd-de57-4efe-a96c-1d349424b2fb-service-ca-bundle podName:5c7b94cd-de57-4efe-a96c-1d349424b2fb nodeName:}" failed. No retries permitted until 2026-09-18 19:58:36.357904965 +0000 UTC m=+101.311646114 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5c7b94cd-de57-4efe-a96c-1d349424b2fb-service-ca-bundle") pod "router-default-7ff8bb4888-5sfkj" (UID: "5c7b94cd-de57-4efe-a96c-1d349424b2fb") : configmap references non-existent config key: service-ca.crt
Sep 18 19:58:28.357984 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:58:28.357935    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c7b94cd-de57-4efe-a96c-1d349424b2fb-metrics-certs podName:5c7b94cd-de57-4efe-a96c-1d349424b2fb nodeName:}" failed. No retries permitted until 2026-09-18 19:58:36.357925075 +0000 UTC m=+101.311666224 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5c7b94cd-de57-4efe-a96c-1d349424b2fb-metrics-certs") pod "router-default-7ff8bb4888-5sfkj" (UID: "5c7b94cd-de57-4efe-a96c-1d349424b2fb") : secret "router-metrics-certs-default" not found
Sep 18 19:58:28.459000 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:28.458971    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/96908402-98ec-4672-a484-d136b3546b7e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-49kw5\" (UID: \"96908402-98ec-4672-a484-d136b3546b7e\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-49kw5"
Sep 18 19:58:28.459138 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:58:28.459109    2549 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 19:58:28.459177 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:58:28.459163    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96908402-98ec-4672-a484-d136b3546b7e-cluster-monitoring-operator-tls podName:96908402-98ec-4672-a484-d136b3546b7e nodeName:}" failed. No retries permitted until 2026-09-18 19:58:36.459150013 +0000 UTC m=+101.412891172 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/96908402-98ec-4672-a484-d136b3546b7e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-49kw5" (UID: "96908402-98ec-4672-a484-d136b3546b7e") : secret "cluster-monitoring-operator-tls" not found
Sep 18 19:58:28.971824 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:28.971788    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-wshq5" event={"ID":"abb6ed43-070c-4bc5-af80-ee32996c9a2e","Type":"ContainerStarted","Data":"4c08748e0f2b1ece2d5d978576fa14d2530bdf8d2cfc36ce11a48d20018a76fa"}
Sep 18 19:58:29.977159 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:29.977124    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-wshq5" event={"ID":"abb6ed43-070c-4bc5-af80-ee32996c9a2e","Type":"ContainerStarted","Data":"050a03b87f9b6a90d23a8e9fd6c11147cb4d1ea9defe4313b872c33c9bcc25a2"}
Sep 18 19:58:29.977521 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:29.977177    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-wshq5" event={"ID":"abb6ed43-070c-4bc5-af80-ee32996c9a2e","Type":"ContainerStarted","Data":"837da2c2666cae9d15a8b372d9e778e65f1720536958dba1cfa9cb452588c86b"}
Sep 18 19:58:29.996586 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:29.996532    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-wshq5" podStartSLOduration=1.785072735 podStartE2EDuration="2.996517188s" totalImagesPullingTime="1.211444453s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:58:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:58:29.996031943 +0000 UTC m=+94.949773110" watchObservedRunningTime="2026-09-18 19:58:29.996517188 +0000 UTC m=+94.950258346"
Sep 18 19:58:31.684174 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:31.684139    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a2a67f2-15ec-472e-95a5-79667b3a8eef-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6fm6j\" (UID: \"0a2a67f2-15ec-472e-95a5-79667b3a8eef\") " pod="openshift-insights/insights-runtime-extractor-6fm6j"
Sep 18 19:58:31.684622 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:31.684190    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0a2a67f2-15ec-472e-95a5-79667b3a8eef-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6fm6j\" (UID: \"0a2a67f2-15ec-472e-95a5-79667b3a8eef\") " pod="openshift-insights/insights-runtime-extractor-6fm6j"
Sep 18 19:58:31.684622 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:58:31.684290    2549 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:58:31.684622 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:58:31.684323    2549 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 19:58:31.684622 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:58:31.684360    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a2a67f2-15ec-472e-95a5-79667b3a8eef-insights-runtime-extractor-tls podName:0a2a67f2-15ec-472e-95a5-79667b3a8eef nodeName:}" failed. No retries permitted until 2026-09-18 19:58:39.684343345 +0000 UTC m=+104.638084508 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0a2a67f2-15ec-472e-95a5-79667b3a8eef-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6fm6j" (UID: "0a2a67f2-15ec-472e-95a5-79667b3a8eef") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:58:31.684622 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:58:31.684378    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0a2a67f2-15ec-472e-95a5-79667b3a8eef-kube-rbac-proxy-cm podName:0a2a67f2-15ec-472e-95a5-79667b3a8eef nodeName:}" failed. No retries permitted until 2026-09-18 19:58:39.684366282 +0000 UTC m=+104.638107428 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0a2a67f2-15ec-472e-95a5-79667b3a8eef-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6fm6j" (UID: "0a2a67f2-15ec-472e-95a5-79667b3a8eef") : configmap "kube-rbac-proxy" not found
Sep 18 19:58:32.692306 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:32.692269    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0b1d2455-d7fc-4bde-a435-7dfaedd8390e-metrics-tls\") pod \"dns-default-mbfbc\" (UID: \"0b1d2455-d7fc-4bde-a435-7dfaedd8390e\") " pod="openshift-dns/dns-default-mbfbc"
Sep 18 19:58:32.692689 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:58:32.692370    2549 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:58:32.692689 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:58:32.692441    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b1d2455-d7fc-4bde-a435-7dfaedd8390e-metrics-tls podName:0b1d2455-d7fc-4bde-a435-7dfaedd8390e nodeName:}" failed. No retries permitted until 2026-09-18 19:59:36.692429457 +0000 UTC m=+161.646170616 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0b1d2455-d7fc-4bde-a435-7dfaedd8390e-metrics-tls") pod "dns-default-mbfbc" (UID: "0b1d2455-d7fc-4bde-a435-7dfaedd8390e") : secret "dns-default-metrics-tls" not found
Sep 18 19:58:36.422650 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:36.422614    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c7b94cd-de57-4efe-a96c-1d349424b2fb-metrics-certs\") pod \"router-default-7ff8bb4888-5sfkj\" (UID: \"5c7b94cd-de57-4efe-a96c-1d349424b2fb\") " pod="openshift-ingress/router-default-7ff8bb4888-5sfkj"
Sep 18 19:58:36.423115 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:36.422661    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/706ee6ce-5cef-4fe6-847d-8745f7884423-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-z7zsx\" (UID: \"706ee6ce-5cef-4fe6-847d-8745f7884423\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-z7zsx"
Sep 18 19:58:36.423115 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:36.422792    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c7b94cd-de57-4efe-a96c-1d349424b2fb-service-ca-bundle\") pod \"router-default-7ff8bb4888-5sfkj\" (UID: \"5c7b94cd-de57-4efe-a96c-1d349424b2fb\") " pod="openshift-ingress/router-default-7ff8bb4888-5sfkj"
Sep 18 19:58:36.423347 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:36.423326    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c7b94cd-de57-4efe-a96c-1d349424b2fb-service-ca-bundle\") pod \"router-default-7ff8bb4888-5sfkj\" (UID: \"5c7b94cd-de57-4efe-a96c-1d349424b2fb\") " pod="openshift-ingress/router-default-7ff8bb4888-5sfkj"
Sep 18 19:58:36.425120 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:36.425093    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c7b94cd-de57-4efe-a96c-1d349424b2fb-metrics-certs\") pod \"router-default-7ff8bb4888-5sfkj\" (UID: \"5c7b94cd-de57-4efe-a96c-1d349424b2fb\") " pod="openshift-ingress/router-default-7ff8bb4888-5sfkj"
Sep 18 19:58:36.425120 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:36.425115    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/706ee6ce-5cef-4fe6-847d-8745f7884423-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-z7zsx\" (UID: \"706ee6ce-5cef-4fe6-847d-8745f7884423\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-z7zsx"
Sep 18 19:58:36.503455 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:36.503431    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-z7zsx"
Sep 18 19:58:36.519506 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:36.519485    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7ff8bb4888-5sfkj"
Sep 18 19:58:36.523895 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:36.523867    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/96908402-98ec-4672-a484-d136b3546b7e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-49kw5\" (UID: \"96908402-98ec-4672-a484-d136b3546b7e\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-49kw5"
Sep 18 19:58:36.524065 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:58:36.524043    2549 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 19:58:36.524154 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:58:36.524138    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96908402-98ec-4672-a484-d136b3546b7e-cluster-monitoring-operator-tls podName:96908402-98ec-4672-a484-d136b3546b7e nodeName:}" failed. No retries permitted until 2026-09-18 19:58:52.524115333 +0000 UTC m=+117.477856492 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/96908402-98ec-4672-a484-d136b3546b7e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-49kw5" (UID: "96908402-98ec-4672-a484-d136b3546b7e") : secret "cluster-monitoring-operator-tls" not found
Sep 18 19:58:36.799839 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:36.799763    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-z7zsx"]
Sep 18 19:58:36.818177 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:36.818153    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7ff8bb4888-5sfkj"]
Sep 18 19:58:36.821214 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:58:36.821190    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c7b94cd_de57_4efe_a96c_1d349424b2fb.slice/crio-194285ab8c339aaf1ce528bb416a9d93d0c24d1a12317856e04bf5d93be5dd65 WatchSource:0}: Error finding container 194285ab8c339aaf1ce528bb416a9d93d0c24d1a12317856e04bf5d93be5dd65: Status 404 returned error can't find the container with id 194285ab8c339aaf1ce528bb416a9d93d0c24d1a12317856e04bf5d93be5dd65
Sep 18 19:58:36.908072 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:36.908048    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-rktz8"
Sep 18 19:58:36.997649 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:36.997617    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7ff8bb4888-5sfkj" event={"ID":"5c7b94cd-de57-4efe-a96c-1d349424b2fb","Type":"ContainerStarted","Data":"0bcfbcb5311036c43287e80a5a162dfe070a4f833fb6f18f4554b297234fe3c2"}
Sep 18 19:58:36.997832 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:36.997671    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7ff8bb4888-5sfkj" event={"ID":"5c7b94cd-de57-4efe-a96c-1d349424b2fb","Type":"ContainerStarted","Data":"194285ab8c339aaf1ce528bb416a9d93d0c24d1a12317856e04bf5d93be5dd65"}
Sep 18 19:58:36.998776 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:36.998743    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-z7zsx" event={"ID":"706ee6ce-5cef-4fe6-847d-8745f7884423","Type":"ContainerStarted","Data":"052e4387985fbd766ef43f63c15f1fc1c9243906a7d95939e9cd824dcf746e1f"}
Sep 18 19:58:37.031906 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:37.031866    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-7ff8bb4888-5sfkj" podStartSLOduration=17.031852818 podStartE2EDuration="17.031852818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:58:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:58:37.031852848 +0000 UTC m=+101.985594017" watchObservedRunningTime="2026-09-18 19:58:37.031852818 +0000 UTC m=+101.985593986"
Sep 18 19:58:37.519668 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:37.519634    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-7ff8bb4888-5sfkj"
Sep 18 19:58:37.522736 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:37.522711    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-7ff8bb4888-5sfkj"
Sep 18 19:58:38.001939 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:38.001909    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-7ff8bb4888-5sfkj"
Sep 18 19:58:38.003133 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:38.003112    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-7ff8bb4888-5sfkj"
Sep 18 19:58:39.749872 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:39.749839    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a2a67f2-15ec-472e-95a5-79667b3a8eef-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6fm6j\" (UID: \"0a2a67f2-15ec-472e-95a5-79667b3a8eef\") " pod="openshift-insights/insights-runtime-extractor-6fm6j"
Sep 18 19:58:39.750327 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:39.749879    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0a2a67f2-15ec-472e-95a5-79667b3a8eef-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6fm6j\" (UID: \"0a2a67f2-15ec-472e-95a5-79667b3a8eef\") " pod="openshift-insights/insights-runtime-extractor-6fm6j"
Sep 18 19:58:39.750327 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:58:39.750013    2549 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:58:39.750327 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:58:39.750092    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a2a67f2-15ec-472e-95a5-79667b3a8eef-insights-runtime-extractor-tls podName:0a2a67f2-15ec-472e-95a5-79667b3a8eef nodeName:}" failed. No retries permitted until 2026-09-18 19:58:55.750072522 +0000 UTC m=+120.703813686 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0a2a67f2-15ec-472e-95a5-79667b3a8eef-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6fm6j" (UID: "0a2a67f2-15ec-472e-95a5-79667b3a8eef") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:58:39.750327 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:39.750308    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0a2a67f2-15ec-472e-95a5-79667b3a8eef-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6fm6j\" (UID: \"0a2a67f2-15ec-472e-95a5-79667b3a8eef\") " pod="openshift-insights/insights-runtime-extractor-6fm6j"
Sep 18 19:58:40.009867 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:40.009780    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-z7zsx" event={"ID":"706ee6ce-5cef-4fe6-847d-8745f7884423","Type":"ContainerStarted","Data":"dd83c85c0a221af20e43760be9361a3ee1a0e24dace3836d461df523cd16f8d0"}
Sep 18 19:58:40.009867 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:40.009836    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-z7zsx" event={"ID":"706ee6ce-5cef-4fe6-847d-8745f7884423","Type":"ContainerStarted","Data":"10d6a6f8e0bf513820885106b21637bac7faa9d19bb4ed06d944f59b7bbc9ebd"}
Sep 18 19:58:40.053055 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:40.053003    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-z7zsx" podStartSLOduration=17.938603536 podStartE2EDuration="20.052987441s" totalImagesPullingTime="2.114383905s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:58:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:58:40.052372414 +0000 UTC m=+105.006113582" watchObservedRunningTime="2026-09-18 19:58:40.052987441 +0000 UTC m=+105.006728608"
Sep 18 19:58:52.264917 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:52.264836    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6594f4c75-fklnx"]
Sep 18 19:58:52.266890 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:52.266871    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-fklnx"
Sep 18 19:58:52.272228 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:52.272208    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 18 19:58:52.272354 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:52.272271    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-wz9xg\""
Sep 18 19:58:52.273611 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:52.273591    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 18 19:58:52.289561 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:52.289532    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-fklnx"]
Sep 18 19:58:52.338037 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:52.338002    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2n79\" (UniqueName: \"kubernetes.io/projected/3c4c58e4-a1ac-4a7a-8230-75567a27ce18-kube-api-access-w2n79\") pod \"downloads-6594f4c75-fklnx\" (UID: \"3c4c58e4-a1ac-4a7a-8230-75567a27ce18\") " pod="openshift-console/downloads-6594f4c75-fklnx"
Sep 18 19:58:52.438802 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:52.438771    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2n79\" (UniqueName: \"kubernetes.io/projected/3c4c58e4-a1ac-4a7a-8230-75567a27ce18-kube-api-access-w2n79\") pod \"downloads-6594f4c75-fklnx\" (UID: \"3c4c58e4-a1ac-4a7a-8230-75567a27ce18\") " pod="openshift-console/downloads-6594f4c75-fklnx"
Sep 18 19:58:52.463631 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:52.463598    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2n79\" (UniqueName: \"kubernetes.io/projected/3c4c58e4-a1ac-4a7a-8230-75567a27ce18-kube-api-access-w2n79\") pod \"downloads-6594f4c75-fklnx\" (UID: \"3c4c58e4-a1ac-4a7a-8230-75567a27ce18\") " pod="openshift-console/downloads-6594f4c75-fklnx"
Sep 18 19:58:52.539579 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:52.539500    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/96908402-98ec-4672-a484-d136b3546b7e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-49kw5\" (UID: \"96908402-98ec-4672-a484-d136b3546b7e\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-49kw5"
Sep 18 19:58:52.541795 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:52.541775    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/96908402-98ec-4672-a484-d136b3546b7e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-49kw5\" (UID: \"96908402-98ec-4672-a484-d136b3546b7e\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-49kw5"
Sep 18 19:58:52.575630 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:52.575607    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-fklnx"
Sep 18 19:58:52.820256 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:52.820236    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-49kw5"
Sep 18 19:58:52.883115 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:52.883093    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-fklnx"]
Sep 18 19:58:52.885088 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:58:52.885060    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c4c58e4_a1ac_4a7a_8230_75567a27ce18.slice/crio-4e8db7271748e042dd911cb836c32ffd632f39936284b74d343d86b373efcaf5 WatchSource:0}: Error finding container 4e8db7271748e042dd911cb836c32ffd632f39936284b74d343d86b373efcaf5: Status 404 returned error can't find the container with id 4e8db7271748e042dd911cb836c32ffd632f39936284b74d343d86b373efcaf5
Sep 18 19:58:53.053340 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:53.053310    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-fklnx" event={"ID":"3c4c58e4-a1ac-4a7a-8230-75567a27ce18","Type":"ContainerStarted","Data":"4e8db7271748e042dd911cb836c32ffd632f39936284b74d343d86b373efcaf5"}
Sep 18 19:58:53.098073 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:53.098051    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-49kw5"]
Sep 18 19:58:53.099669 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:58:53.099642    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96908402_98ec_4672_a484_d136b3546b7e.slice/crio-41bb572e9ef87f38bbe89dbd4dd7dcb04d7fc88b662d39e694292be030d9e1e6 WatchSource:0}: Error finding container 41bb572e9ef87f38bbe89dbd4dd7dcb04d7fc88b662d39e694292be030d9e1e6: Status 404 returned error can't find the container with id 41bb572e9ef87f38bbe89dbd4dd7dcb04d7fc88b662d39e694292be030d9e1e6
Sep 18 19:58:54.058780 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:54.058709    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-49kw5" event={"ID":"96908402-98ec-4672-a484-d136b3546b7e","Type":"ContainerStarted","Data":"41bb572e9ef87f38bbe89dbd4dd7dcb04d7fc88b662d39e694292be030d9e1e6"}
Sep 18 19:58:55.672411 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:55.672364    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-9csv2"]
Sep 18 19:58:55.676282 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:55.676254    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-9csv2"
Sep 18 19:58:55.679611 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:55.679588    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 18 19:58:55.679859 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:55.679590    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-25xn8\""
Sep 18 19:58:55.688653 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:55.688582    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-9csv2"]
Sep 18 19:58:55.767202 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:55.767171    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a2a67f2-15ec-472e-95a5-79667b3a8eef-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6fm6j\" (UID: \"0a2a67f2-15ec-472e-95a5-79667b3a8eef\") " pod="openshift-insights/insights-runtime-extractor-6fm6j"
Sep 18 19:58:55.767364 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:55.767241    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/35eb5c6b-e034-4ec9-baa7-2bbd78601a41-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-9csv2\" (UID: \"35eb5c6b-e034-4ec9-baa7-2bbd78601a41\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-9csv2"
Sep 18 19:58:55.769723 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:55.769700    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a2a67f2-15ec-472e-95a5-79667b3a8eef-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6fm6j\" (UID: \"0a2a67f2-15ec-472e-95a5-79667b3a8eef\") " pod="openshift-insights/insights-runtime-extractor-6fm6j"
Sep 18 19:58:55.868253 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:55.868219    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/35eb5c6b-e034-4ec9-baa7-2bbd78601a41-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-9csv2\" (UID: \"35eb5c6b-e034-4ec9-baa7-2bbd78601a41\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-9csv2"
Sep 18 19:58:55.870951 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:55.870922    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/35eb5c6b-e034-4ec9-baa7-2bbd78601a41-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-9csv2\" (UID: \"35eb5c6b-e034-4ec9-baa7-2bbd78601a41\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-9csv2"
Sep 18 19:58:55.987282 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:55.987202    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-9csv2"
Sep 18 19:58:56.054861 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:56.054809    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-dnrhk\""
Sep 18 19:58:56.062480 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:56.062457    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6fm6j"
Sep 18 19:58:56.068963 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:56.068928    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-49kw5" event={"ID":"96908402-98ec-4672-a484-d136b3546b7e","Type":"ContainerStarted","Data":"eeba01f994eb880fd65ea9aa8d7cf746941ffa4fb9edc447cdec39ab522a2e87"}
Sep 18 19:58:56.090291 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:56.090056    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-49kw5" podStartSLOduration=34.152864219 podStartE2EDuration="36.090034937s" totalImagesPullingTime="1.937170718s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:58:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:58:56.087552485 +0000 UTC m=+121.041293653" watchObservedRunningTime="2026-09-18 19:58:56.090034937 +0000 UTC m=+121.043776109"
Sep 18 19:58:56.275006 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:56.274885    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-9csv2"]
Sep 18 19:58:56.277616 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:58:56.277588    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35eb5c6b_e034_4ec9_baa7_2bbd78601a41.slice/crio-2ffc9cd9f6354a8277199c04b049c458f65923fdba612c05aa97ece061bc3841 WatchSource:0}: Error finding container 2ffc9cd9f6354a8277199c04b049c458f65923fdba612c05aa97ece061bc3841: Status 404 returned error can't find the container with id 2ffc9cd9f6354a8277199c04b049c458f65923fdba612c05aa97ece061bc3841
Sep 18 19:58:56.353426 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:56.353370    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6fm6j"]
Sep 18 19:58:56.358206 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:58:56.358170    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a2a67f2_15ec_472e_95a5_79667b3a8eef.slice/crio-afe99c3737db012e09d74042f5821dc84ef6d454dd2030c73ef2bbe5ed40bfa3 WatchSource:0}: Error finding container afe99c3737db012e09d74042f5821dc84ef6d454dd2030c73ef2bbe5ed40bfa3: Status 404 returned error can't find the container with id afe99c3737db012e09d74042f5821dc84ef6d454dd2030c73ef2bbe5ed40bfa3
Sep 18 19:58:57.074367 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:57.074330    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-758b887fb4-mmjkk"]
Sep 18 19:58:57.077455 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:57.077213    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-9csv2" event={"ID":"35eb5c6b-e034-4ec9-baa7-2bbd78601a41","Type":"ContainerStarted","Data":"2ffc9cd9f6354a8277199c04b049c458f65923fdba612c05aa97ece061bc3841"}
Sep 18 19:58:57.077455 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:57.077320    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-758b887fb4-mmjkk"
Sep 18 19:58:57.080036 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:57.080003    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6fm6j" event={"ID":"0a2a67f2-15ec-472e-95a5-79667b3a8eef","Type":"ContainerStarted","Data":"7a7e4b113e858bbcd542dbd19d15f7171204547bee25c7d50ce9cfe7caf5c19e"}
Sep 18 19:58:57.080140 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:57.080064    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6fm6j" event={"ID":"0a2a67f2-15ec-472e-95a5-79667b3a8eef","Type":"ContainerStarted","Data":"259162c8df71eb91fc15e39f64a68abc08568b26cb168d2c0795b618deed2c0f"}
Sep 18 19:58:57.080140 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:57.080091    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6fm6j" event={"ID":"0a2a67f2-15ec-472e-95a5-79667b3a8eef","Type":"ContainerStarted","Data":"afe99c3737db012e09d74042f5821dc84ef6d454dd2030c73ef2bbe5ed40bfa3"}
Sep 18 19:58:57.081438 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:57.081359    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 18 19:58:57.082766 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:57.082486    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 18 19:58:57.082766 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:57.082520    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 18 19:58:57.082928 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:57.082885    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 18 19:58:57.083128 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:57.083113    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-4v52r\""
Sep 18 19:58:57.083297 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:57.083279    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 18 19:58:57.092904 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:57.092853    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-758b887fb4-mmjkk"]
Sep 18 19:58:57.178556 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:57.178530    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/91c6036d-63f3-407e-bea7-18510ae4e95a-console-serving-cert\") pod \"console-758b887fb4-mmjkk\" (UID: \"91c6036d-63f3-407e-bea7-18510ae4e95a\") " pod="openshift-console/console-758b887fb4-mmjkk"
Sep 18 19:58:57.178556 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:57.178570    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/91c6036d-63f3-407e-bea7-18510ae4e95a-console-config\") pod \"console-758b887fb4-mmjkk\" (UID: \"91c6036d-63f3-407e-bea7-18510ae4e95a\") " pod="openshift-console/console-758b887fb4-mmjkk"
Sep 18 19:58:57.178878 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:57.178644    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/91c6036d-63f3-407e-bea7-18510ae4e95a-console-oauth-config\") pod \"console-758b887fb4-mmjkk\" (UID: \"91c6036d-63f3-407e-bea7-18510ae4e95a\") " pod="openshift-console/console-758b887fb4-mmjkk"
Sep 18 19:58:57.178878 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:57.178720    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/91c6036d-63f3-407e-bea7-18510ae4e95a-service-ca\") pod \"console-758b887fb4-mmjkk\" (UID: \"91c6036d-63f3-407e-bea7-18510ae4e95a\") " pod="openshift-console/console-758b887fb4-mmjkk"
Sep 18 19:58:57.178878 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:57.178748    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2ffq\" (UniqueName: \"kubernetes.io/projected/91c6036d-63f3-407e-bea7-18510ae4e95a-kube-api-access-c2ffq\") pod \"console-758b887fb4-mmjkk\" (UID: \"91c6036d-63f3-407e-bea7-18510ae4e95a\") " pod="openshift-console/console-758b887fb4-mmjkk"
Sep 18 19:58:57.178878 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:57.178789    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/91c6036d-63f3-407e-bea7-18510ae4e95a-oauth-serving-cert\") pod \"console-758b887fb4-mmjkk\" (UID: \"91c6036d-63f3-407e-bea7-18510ae4e95a\") " pod="openshift-console/console-758b887fb4-mmjkk"
Sep 18 19:58:57.279907 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:57.279831    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/91c6036d-63f3-407e-bea7-18510ae4e95a-console-config\") pod \"console-758b887fb4-mmjkk\" (UID: \"91c6036d-63f3-407e-bea7-18510ae4e95a\") " pod="openshift-console/console-758b887fb4-mmjkk"
Sep 18 19:58:57.279907 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:57.279870    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/91c6036d-63f3-407e-bea7-18510ae4e95a-console-oauth-config\") pod \"console-758b887fb4-mmjkk\" (UID: \"91c6036d-63f3-407e-bea7-18510ae4e95a\") " pod="openshift-console/console-758b887fb4-mmjkk"
Sep 18 19:58:57.280129 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:57.280111    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/91c6036d-63f3-407e-bea7-18510ae4e95a-service-ca\") pod \"console-758b887fb4-mmjkk\" (UID: \"91c6036d-63f3-407e-bea7-18510ae4e95a\") " pod="openshift-console/console-758b887fb4-mmjkk"
Sep 18 19:58:57.280204 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:57.280163    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2ffq\" (UniqueName: \"kubernetes.io/projected/91c6036d-63f3-407e-bea7-18510ae4e95a-kube-api-access-c2ffq\") pod \"console-758b887fb4-mmjkk\" (UID: \"91c6036d-63f3-407e-bea7-18510ae4e95a\") " pod="openshift-console/console-758b887fb4-mmjkk"
Sep 18 19:58:57.280257 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:57.280218    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/91c6036d-63f3-407e-bea7-18510ae4e95a-oauth-serving-cert\") pod \"console-758b887fb4-mmjkk\" (UID: \"91c6036d-63f3-407e-bea7-18510ae4e95a\") " pod="openshift-console/console-758b887fb4-mmjkk"
Sep 18 19:58:57.280314 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:57.280282    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/91c6036d-63f3-407e-bea7-18510ae4e95a-console-serving-cert\") pod \"console-758b887fb4-mmjkk\" (UID: \"91c6036d-63f3-407e-bea7-18510ae4e95a\") " pod="openshift-console/console-758b887fb4-mmjkk"
Sep 18 19:58:57.280681 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:57.280644    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/91c6036d-63f3-407e-bea7-18510ae4e95a-console-config\") pod \"console-758b887fb4-mmjkk\" (UID: \"91c6036d-63f3-407e-bea7-18510ae4e95a\") " pod="openshift-console/console-758b887fb4-mmjkk"
Sep 18 19:58:57.280893 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:57.280848    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/91c6036d-63f3-407e-bea7-18510ae4e95a-service-ca\") pod \"console-758b887fb4-mmjkk\" (UID: \"91c6036d-63f3-407e-bea7-18510ae4e95a\") " pod="openshift-console/console-758b887fb4-mmjkk"
Sep 18 19:58:57.281246 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:57.281158    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/91c6036d-63f3-407e-bea7-18510ae4e95a-oauth-serving-cert\") pod \"console-758b887fb4-mmjkk\" (UID: \"91c6036d-63f3-407e-bea7-18510ae4e95a\") " pod="openshift-console/console-758b887fb4-mmjkk"
Sep 18 19:58:57.283076 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:57.283052    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/91c6036d-63f3-407e-bea7-18510ae4e95a-console-oauth-config\") pod \"console-758b887fb4-mmjkk\" (UID: \"91c6036d-63f3-407e-bea7-18510ae4e95a\") " pod="openshift-console/console-758b887fb4-mmjkk"
Sep 18 19:58:57.283177 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:57.283165    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/91c6036d-63f3-407e-bea7-18510ae4e95a-console-serving-cert\") pod \"console-758b887fb4-mmjkk\" (UID: \"91c6036d-63f3-407e-bea7-18510ae4e95a\") " pod="openshift-console/console-758b887fb4-mmjkk"
Sep 18 19:58:57.289733 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:57.289694    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2ffq\" (UniqueName: \"kubernetes.io/projected/91c6036d-63f3-407e-bea7-18510ae4e95a-kube-api-access-c2ffq\") pod \"console-758b887fb4-mmjkk\" (UID: \"91c6036d-63f3-407e-bea7-18510ae4e95a\") " pod="openshift-console/console-758b887fb4-mmjkk"
Sep 18 19:58:57.391781 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:57.391678    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-758b887fb4-mmjkk"
Sep 18 19:58:57.709448 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:57.709408    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-758b887fb4-mmjkk"]
Sep 18 19:58:57.713965 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:58:57.713931    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91c6036d_63f3_407e_bea7_18510ae4e95a.slice/crio-dcd8eb9a233df15d18af82a5a67d7a993f771593a0ab7889e3c78de46d9bd5bc WatchSource:0}: Error finding container dcd8eb9a233df15d18af82a5a67d7a993f771593a0ab7889e3c78de46d9bd5bc: Status 404 returned error can't find the container with id dcd8eb9a233df15d18af82a5a67d7a993f771593a0ab7889e3c78de46d9bd5bc
Sep 18 19:58:58.085617 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:58.085579    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-9csv2" event={"ID":"35eb5c6b-e034-4ec9-baa7-2bbd78601a41","Type":"ContainerStarted","Data":"2c660cd1c2113ce3ce2f54249beb92c88a1b1864f1b4fa49eecf2cb425cc84b8"}
Sep 18 19:58:58.086060 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:58.085805    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-9csv2"
Sep 18 19:58:58.087643 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:58.087614    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-758b887fb4-mmjkk" event={"ID":"91c6036d-63f3-407e-bea7-18510ae4e95a","Type":"ContainerStarted","Data":"dcd8eb9a233df15d18af82a5a67d7a993f771593a0ab7889e3c78de46d9bd5bc"}
Sep 18 19:58:58.092002 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:58.091981    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-9csv2"
Sep 18 19:58:58.105628 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:58.105551    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-9csv2" podStartSLOduration=1.923514281 podStartE2EDuration="3.105535389s" totalImagesPullingTime="1.182021108s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:58:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:58:58.104072881 +0000 UTC m=+123.057814052" watchObservedRunningTime="2026-09-18 19:58:58.105535389 +0000 UTC m=+123.059276559"
Sep 18 19:58:58.771918 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:58.771785    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-fplht"]
Sep 18 19:58:58.774918 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:58.774892    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-fplht"
Sep 18 19:58:58.780117 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:58.778856    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 18 19:58:58.780117 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:58.779163    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 18 19:58:58.780117 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:58.779446    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 18 19:58:58.780117 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:58.779651    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-tw7bk\""
Sep 18 19:58:58.792768 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:58.792707    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-fplht"]
Sep 18 19:58:58.896024 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:58.895826    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fpg8\" (UniqueName: \"kubernetes.io/projected/fdcb2ae5-000a-4094-95b7-d6e56bbc904f-kube-api-access-6fpg8\") pod \"prometheus-operator-7bd5d56d6-fplht\" (UID: \"fdcb2ae5-000a-4094-95b7-d6e56bbc904f\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-fplht"
Sep 18 19:58:58.896024 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:58.895898    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/fdcb2ae5-000a-4094-95b7-d6e56bbc904f-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-fplht\" (UID: \"fdcb2ae5-000a-4094-95b7-d6e56bbc904f\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-fplht"
Sep 18 19:58:58.896024 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:58.895930    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fdcb2ae5-000a-4094-95b7-d6e56bbc904f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-fplht\" (UID: \"fdcb2ae5-000a-4094-95b7-d6e56bbc904f\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-fplht"
Sep 18 19:58:58.896024 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:58.895963    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fdcb2ae5-000a-4094-95b7-d6e56bbc904f-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-fplht\" (UID: \"fdcb2ae5-000a-4094-95b7-d6e56bbc904f\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-fplht"
Sep 18 19:58:58.997842 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:58.996765    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/fdcb2ae5-000a-4094-95b7-d6e56bbc904f-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-fplht\" (UID: \"fdcb2ae5-000a-4094-95b7-d6e56bbc904f\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-fplht"
Sep 18 19:58:58.997842 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:58.996819    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fdcb2ae5-000a-4094-95b7-d6e56bbc904f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-fplht\" (UID: \"fdcb2ae5-000a-4094-95b7-d6e56bbc904f\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-fplht"
Sep 18 19:58:58.997842 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:58.996860    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fdcb2ae5-000a-4094-95b7-d6e56bbc904f-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-fplht\" (UID: \"fdcb2ae5-000a-4094-95b7-d6e56bbc904f\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-fplht"
Sep 18 19:58:58.997842 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:58.996955    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fpg8\" (UniqueName: \"kubernetes.io/projected/fdcb2ae5-000a-4094-95b7-d6e56bbc904f-kube-api-access-6fpg8\") pod \"prometheus-operator-7bd5d56d6-fplht\" (UID: \"fdcb2ae5-000a-4094-95b7-d6e56bbc904f\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-fplht"
Sep 18 19:58:58.997842 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:58:58.997166    2549 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Sep 18 19:58:58.997842 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:58:58.997235    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdcb2ae5-000a-4094-95b7-d6e56bbc904f-prometheus-operator-tls podName:fdcb2ae5-000a-4094-95b7-d6e56bbc904f nodeName:}" failed. No retries permitted until 2026-09-18 19:58:59.497213496 +0000 UTC m=+124.450954653 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/fdcb2ae5-000a-4094-95b7-d6e56bbc904f-prometheus-operator-tls") pod "prometheus-operator-7bd5d56d6-fplht" (UID: "fdcb2ae5-000a-4094-95b7-d6e56bbc904f") : secret "prometheus-operator-tls" not found
Sep 18 19:58:58.998303 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:58.998288    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fdcb2ae5-000a-4094-95b7-d6e56bbc904f-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-fplht\" (UID: \"fdcb2ae5-000a-4094-95b7-d6e56bbc904f\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-fplht"
Sep 18 19:58:59.004796 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:59.003244    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fdcb2ae5-000a-4094-95b7-d6e56bbc904f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-fplht\" (UID: \"fdcb2ae5-000a-4094-95b7-d6e56bbc904f\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-fplht"
Sep 18 19:58:59.016378 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:59.016343    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fpg8\" (UniqueName: \"kubernetes.io/projected/fdcb2ae5-000a-4094-95b7-d6e56bbc904f-kube-api-access-6fpg8\") pod \"prometheus-operator-7bd5d56d6-fplht\" (UID: \"fdcb2ae5-000a-4094-95b7-d6e56bbc904f\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-fplht"
Sep 18 19:58:59.096556 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:59.096434    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6fm6j" event={"ID":"0a2a67f2-15ec-472e-95a5-79667b3a8eef","Type":"ContainerStarted","Data":"536daed90346ce4961a4225725e35fe6b58234d56fc4017a983ae841ba8eeee7"}
Sep 18 19:58:59.501346 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:59.501259    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/fdcb2ae5-000a-4094-95b7-d6e56bbc904f-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-fplht\" (UID: \"fdcb2ae5-000a-4094-95b7-d6e56bbc904f\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-fplht"
Sep 18 19:58:59.504215 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:59.504192    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/fdcb2ae5-000a-4094-95b7-d6e56bbc904f-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-fplht\" (UID: \"fdcb2ae5-000a-4094-95b7-d6e56bbc904f\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-fplht"
Sep 18 19:58:59.687550 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:58:59.687506    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-fplht"
Sep 18 19:59:02.750105 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:02.750050    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-6fm6j" podStartSLOduration=37.797916761 podStartE2EDuration="39.750026165s" totalImagesPullingTime="1.952109404s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:58:23 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:58:59.121088682 +0000 UTC m=+124.074829878" watchObservedRunningTime="2026-09-18 19:59:02.750026165 +0000 UTC m=+127.703767329"
Sep 18 19:59:02.750738 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:02.750719    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-fplht"]
Sep 18 19:59:02.752192 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:59:02.752161    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdcb2ae5_000a_4094_95b7_d6e56bbc904f.slice/crio-c1f71fa56e4501e51b959332076ec3c05cb90f66bf0ee7f3b30f698257830a44 WatchSource:0}: Error finding container c1f71fa56e4501e51b959332076ec3c05cb90f66bf0ee7f3b30f698257830a44: Status 404 returned error can't find the container with id c1f71fa56e4501e51b959332076ec3c05cb90f66bf0ee7f3b30f698257830a44
Sep 18 19:59:03.113276 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:03.113236    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-758b887fb4-mmjkk" event={"ID":"91c6036d-63f3-407e-bea7-18510ae4e95a","Type":"ContainerStarted","Data":"3f0be08ce12ea8653fe8f7da8e99426f948328fbfd27d5b66211536bb9aaa313"}
Sep 18 19:59:03.114738 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:03.114713    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-fplht" event={"ID":"fdcb2ae5-000a-4094-95b7-d6e56bbc904f","Type":"ContainerStarted","Data":"c1f71fa56e4501e51b959332076ec3c05cb90f66bf0ee7f3b30f698257830a44"}
Sep 18 19:59:03.149780 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:03.149735    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-758b887fb4-mmjkk" podStartSLOduration=1.3165315309999999 podStartE2EDuration="6.149717218s" totalImagesPullingTime="4.833185687s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:58:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:59:03.149586741 +0000 UTC m=+128.103327910" watchObservedRunningTime="2026-09-18 19:59:03.149717218 +0000 UTC m=+128.103458389"
Sep 18 19:59:04.342899 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:04.342859    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/063d8ee5-35b4-401b-a0ff-d8b416e20175-metrics-certs\") pod \"network-metrics-daemon-2hw77\" (UID: \"063d8ee5-35b4-401b-a0ff-d8b416e20175\") " pod="openshift-multus/network-metrics-daemon-2hw77"
Sep 18 19:59:04.345427 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:04.345377    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/063d8ee5-35b4-401b-a0ff-d8b416e20175-metrics-certs\") pod \"network-metrics-daemon-2hw77\" (UID: \"063d8ee5-35b4-401b-a0ff-d8b416e20175\") " pod="openshift-multus/network-metrics-daemon-2hw77"
Sep 18 19:59:04.630691 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:04.630607    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6rwzk\""
Sep 18 19:59:04.638409 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:04.638371    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2hw77"
Sep 18 19:59:07.392050 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:07.392018    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-758b887fb4-mmjkk"
Sep 18 19:59:07.392512 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:07.392081    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-758b887fb4-mmjkk"
Sep 18 19:59:07.397198 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:07.397173    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-758b887fb4-mmjkk"
Sep 18 19:59:08.138706 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:08.138674    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-758b887fb4-mmjkk"
Sep 18 19:59:10.036064 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:10.035941    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2hw77"]
Sep 18 19:59:10.039915 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:59:10.039882    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod063d8ee5_35b4_401b_a0ff_d8b416e20175.slice/crio-5389b99ee3e602ce62b077a6859f68361e60e0be9f5c817c7fa02e5e8803c593 WatchSource:0}: Error finding container 5389b99ee3e602ce62b077a6859f68361e60e0be9f5c817c7fa02e5e8803c593: Status 404 returned error can't find the container with id 5389b99ee3e602ce62b077a6859f68361e60e0be9f5c817c7fa02e5e8803c593
Sep 18 19:59:10.148533 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:10.148416    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-fklnx" event={"ID":"3c4c58e4-a1ac-4a7a-8230-75567a27ce18","Type":"ContainerStarted","Data":"9e4fb8100b57e435a5814d7fe8ea46967afadf59fde3f531b3472bd229cd75b9"}
Sep 18 19:59:10.149261 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:10.149212    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6594f4c75-fklnx"
Sep 18 19:59:10.151546 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:10.151521    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2hw77" event={"ID":"063d8ee5-35b4-401b-a0ff-d8b416e20175","Type":"ContainerStarted","Data":"5389b99ee3e602ce62b077a6859f68361e60e0be9f5c817c7fa02e5e8803c593"}
Sep 18 19:59:10.160641 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:10.160618    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6594f4c75-fklnx"
Sep 18 19:59:10.171222 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:10.171164    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-6594f4c75-fklnx" podStartSLOduration=1.199297186 podStartE2EDuration="18.171147924s" totalImagesPullingTime="16.971850738s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:58:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:59:10.168794101 +0000 UTC m=+135.122535280" watchObservedRunningTime="2026-09-18 19:59:10.171147924 +0000 UTC m=+135.124889096"
Sep 18 19:59:12.161898 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:12.161860    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-fplht" event={"ID":"fdcb2ae5-000a-4094-95b7-d6e56bbc904f","Type":"ContainerStarted","Data":"e21b50cb5ec1c3923ab150163362150f975d463fc36b006ddf76fba87ba70086"}
Sep 18 19:59:12.162360 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:12.161921    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-fplht" event={"ID":"fdcb2ae5-000a-4094-95b7-d6e56bbc904f","Type":"ContainerStarted","Data":"ea18649017ce8f1b6214a6ef6b5d7927bc64261c1511651b369cf6b2d18b0789"}
Sep 18 19:59:12.164208 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:12.164182    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2hw77" event={"ID":"063d8ee5-35b4-401b-a0ff-d8b416e20175","Type":"ContainerStarted","Data":"e3b1645f6fb686729440d36cdd30559b009ffde3d6668b57d42cff23787b3396"}
Sep 18 19:59:12.164333 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:12.164230    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2hw77" event={"ID":"063d8ee5-35b4-401b-a0ff-d8b416e20175","Type":"ContainerStarted","Data":"c4a7fee7957b622d698d65f408e36d8509810b3135cb20f425bcde5896a87f4c"}
Sep 18 19:59:12.186195 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:12.186107    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-fplht" podStartSLOduration=5.272118433 podStartE2EDuration="14.186090872s" totalImagesPullingTime="8.913972439s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:58:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:59:12.184647865 +0000 UTC m=+137.138389044" watchObservedRunningTime="2026-09-18 19:59:12.186090872 +0000 UTC m=+137.139832042"
Sep 18 19:59:12.206116 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:12.206063    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-2hw77" podStartSLOduration=134.577352035 podStartE2EDuration="2m16.206050666s" totalImagesPullingTime="1.628698631s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:56:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:59:12.204687177 +0000 UTC m=+137.158428346" watchObservedRunningTime="2026-09-18 19:59:12.206050666 +0000 UTC m=+137.159791835"
Sep 18 19:59:14.900809 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:14.900776    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-zshqm"]
Sep 18 19:59:14.945507 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:14.945479    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-zshqm"]
Sep 18 19:59:14.945670 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:14.945619    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-zshqm"
Sep 18 19:59:14.952872 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:14.952687    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 18 19:59:14.958361 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:14.958342    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 18 19:59:14.958655 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:14.958638    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-xlt9w\""
Sep 18 19:59:15.040186 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:15.040149    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c22e65d0-1e2f-4a02-a878-2508729abe1f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-zshqm\" (UID: \"c22e65d0-1e2f-4a02-a878-2508729abe1f\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-zshqm"
Sep 18 19:59:15.040186 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:15.040191    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c22e65d0-1e2f-4a02-a878-2508729abe1f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-zshqm\" (UID: \"c22e65d0-1e2f-4a02-a878-2508729abe1f\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-zshqm"
Sep 18 19:59:15.040422 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:15.040217    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8mgl\" (UniqueName: \"kubernetes.io/projected/c22e65d0-1e2f-4a02-a878-2508729abe1f-kube-api-access-x8mgl\") pod \"openshift-state-metrics-86c5bc79d6-zshqm\" (UID: \"c22e65d0-1e2f-4a02-a878-2508729abe1f\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-zshqm"
Sep 18 19:59:15.040422 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:15.040305    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c22e65d0-1e2f-4a02-a878-2508729abe1f-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-zshqm\" (UID: \"c22e65d0-1e2f-4a02-a878-2508729abe1f\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-zshqm"
Sep 18 19:59:15.141619 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:15.141580    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c22e65d0-1e2f-4a02-a878-2508729abe1f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-zshqm\" (UID: \"c22e65d0-1e2f-4a02-a878-2508729abe1f\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-zshqm"
Sep 18 19:59:15.141619 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:15.141622    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c22e65d0-1e2f-4a02-a878-2508729abe1f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-zshqm\" (UID: \"c22e65d0-1e2f-4a02-a878-2508729abe1f\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-zshqm"
Sep 18 19:59:15.141856 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:15.141650    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8mgl\" (UniqueName: \"kubernetes.io/projected/c22e65d0-1e2f-4a02-a878-2508729abe1f-kube-api-access-x8mgl\") pod \"openshift-state-metrics-86c5bc79d6-zshqm\" (UID: \"c22e65d0-1e2f-4a02-a878-2508729abe1f\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-zshqm"
Sep 18 19:59:15.141856 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:15.141836    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c22e65d0-1e2f-4a02-a878-2508729abe1f-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-zshqm\" (UID: \"c22e65d0-1e2f-4a02-a878-2508729abe1f\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-zshqm"
Sep 18 19:59:15.142669 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:15.142642    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c22e65d0-1e2f-4a02-a878-2508729abe1f-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-zshqm\" (UID: \"c22e65d0-1e2f-4a02-a878-2508729abe1f\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-zshqm"
Sep 18 19:59:15.144320 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:15.144296    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c22e65d0-1e2f-4a02-a878-2508729abe1f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-zshqm\" (UID: \"c22e65d0-1e2f-4a02-a878-2508729abe1f\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-zshqm"
Sep 18 19:59:15.144511 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:15.144490    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c22e65d0-1e2f-4a02-a878-2508729abe1f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-zshqm\" (UID: \"c22e65d0-1e2f-4a02-a878-2508729abe1f\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-zshqm"
Sep 18 19:59:15.150545 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:15.150495    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-5bx25"]
Sep 18 19:59:15.170046 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:15.169982    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8mgl\" (UniqueName: \"kubernetes.io/projected/c22e65d0-1e2f-4a02-a878-2508729abe1f-kube-api-access-x8mgl\") pod \"openshift-state-metrics-86c5bc79d6-zshqm\" (UID: \"c22e65d0-1e2f-4a02-a878-2508729abe1f\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-zshqm"
Sep 18 19:59:15.181782 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:15.181759    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5bx25"
Sep 18 19:59:15.186223 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:15.186202    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 18 19:59:15.186625 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:15.186605    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 18 19:59:15.186825 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:15.186801    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-c6mwt\""
Sep 18 19:59:15.186909 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:15.186838    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 18 19:59:15.255929 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:15.255901    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-zshqm"
Sep 18 19:59:15.344632 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:15.343791    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8b2bb638-e334-44e1-9449-6645ca708da8-root\") pod \"node-exporter-5bx25\" (UID: \"8b2bb638-e334-44e1-9449-6645ca708da8\") " pod="openshift-monitoring/node-exporter-5bx25"
Sep 18 19:59:15.344632 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:15.343844    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8b2bb638-e334-44e1-9449-6645ca708da8-node-exporter-accelerators-collector-config\") pod \"node-exporter-5bx25\" (UID: \"8b2bb638-e334-44e1-9449-6645ca708da8\") " pod="openshift-monitoring/node-exporter-5bx25"
Sep 18 19:59:15.344632 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:15.343917    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8b2bb638-e334-44e1-9449-6645ca708da8-node-exporter-wtmp\") pod \"node-exporter-5bx25\" (UID: \"8b2bb638-e334-44e1-9449-6645ca708da8\") " pod="openshift-monitoring/node-exporter-5bx25"
Sep 18 19:59:15.344632 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:15.343965    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8b2bb638-e334-44e1-9449-6645ca708da8-metrics-client-ca\") pod \"node-exporter-5bx25\" (UID: \"8b2bb638-e334-44e1-9449-6645ca708da8\") " pod="openshift-monitoring/node-exporter-5bx25"
Sep 18 19:59:15.344632 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:15.344030    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8b2bb638-e334-44e1-9449-6645ca708da8-node-exporter-tls\") pod \"node-exporter-5bx25\" (UID: \"8b2bb638-e334-44e1-9449-6645ca708da8\") " pod="openshift-monitoring/node-exporter-5bx25"
Sep 18 19:59:15.344632 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:15.344055    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8b2bb638-e334-44e1-9449-6645ca708da8-sys\") pod \"node-exporter-5bx25\" (UID: \"8b2bb638-e334-44e1-9449-6645ca708da8\") " pod="openshift-monitoring/node-exporter-5bx25"
Sep 18 19:59:15.344632 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:15.344081    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxc8s\" (UniqueName: \"kubernetes.io/projected/8b2bb638-e334-44e1-9449-6645ca708da8-kube-api-access-kxc8s\") pod \"node-exporter-5bx25\" (UID: \"8b2bb638-e334-44e1-9449-6645ca708da8\") " pod="openshift-monitoring/node-exporter-5bx25"
Sep 18 19:59:15.344632 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:15.344120    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8b2bb638-e334-44e1-9449-6645ca708da8-node-exporter-textfile\") pod \"node-exporter-5bx25\" (UID: \"8b2bb638-e334-44e1-9449-6645ca708da8\") " pod="openshift-monitoring/node-exporter-5bx25"
Sep 18 19:59:15.344632 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:15.344152    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8b2bb638-e334-44e1-9449-6645ca708da8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5bx25\" (UID: \"8b2bb638-e334-44e1-9449-6645ca708da8\") " pod="openshift-monitoring/node-exporter-5bx25"
Sep 18 19:59:15.445216 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:15.445136    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8b2bb638-e334-44e1-9449-6645ca708da8-node-exporter-textfile\") pod \"node-exporter-5bx25\" (UID: \"8b2bb638-e334-44e1-9449-6645ca708da8\") " pod="openshift-monitoring/node-exporter-5bx25"
Sep 18 19:59:15.445216 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:15.445185    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8b2bb638-e334-44e1-9449-6645ca708da8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5bx25\" (UID: \"8b2bb638-e334-44e1-9449-6645ca708da8\") " pod="openshift-monitoring/node-exporter-5bx25"
Sep 18 19:59:15.445216 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:15.445214    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8b2bb638-e334-44e1-9449-6645ca708da8-root\") pod \"node-exporter-5bx25\" (UID: \"8b2bb638-e334-44e1-9449-6645ca708da8\") " pod="openshift-monitoring/node-exporter-5bx25"
Sep 18 19:59:15.445507 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:15.445231    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8b2bb638-e334-44e1-9449-6645ca708da8-node-exporter-accelerators-collector-config\") pod \"node-exporter-5bx25\" (UID: \"8b2bb638-e334-44e1-9449-6645ca708da8\") " pod="openshift-monitoring/node-exporter-5bx25"
Sep 18 19:59:15.445507 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:15.445292    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8b2bb638-e334-44e1-9449-6645ca708da8-node-exporter-wtmp\") pod \"node-exporter-5bx25\" (UID: \"8b2bb638-e334-44e1-9449-6645ca708da8\") " pod="openshift-monitoring/node-exporter-5bx25"
Sep 18 19:59:15.445507 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:15.445333    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8b2bb638-e334-44e1-9449-6645ca708da8-metrics-client-ca\") pod \"node-exporter-5bx25\" (UID: \"8b2bb638-e334-44e1-9449-6645ca708da8\") " pod="openshift-monitoring/node-exporter-5bx25"
Sep 18 19:59:15.445507 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:15.445421    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8b2bb638-e334-44e1-9449-6645ca708da8-node-exporter-tls\") pod \"node-exporter-5bx25\" (UID: \"8b2bb638-e334-44e1-9449-6645ca708da8\") " pod="openshift-monitoring/node-exporter-5bx25"
Sep 18 19:59:15.445507 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:15.445448    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8b2bb638-e334-44e1-9449-6645ca708da8-sys\") pod \"node-exporter-5bx25\" (UID: \"8b2bb638-e334-44e1-9449-6645ca708da8\") " pod="openshift-monitoring/node-exporter-5bx25"
Sep 18 19:59:15.445507 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:15.445473    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxc8s\" (UniqueName: \"kubernetes.io/projected/8b2bb638-e334-44e1-9449-6645ca708da8-kube-api-access-kxc8s\") pod \"node-exporter-5bx25\" (UID: \"8b2bb638-e334-44e1-9449-6645ca708da8\") " pod="openshift-monitoring/node-exporter-5bx25"
Sep 18 19:59:15.445841 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:15.445553    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8b2bb638-e334-44e1-9449-6645ca708da8-node-exporter-textfile\") pod \"node-exporter-5bx25\" (UID: \"8b2bb638-e334-44e1-9449-6645ca708da8\") " pod="openshift-monitoring/node-exporter-5bx25"
Sep 18 19:59:15.445841 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:15.445651    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8b2bb638-e334-44e1-9449-6645ca708da8-root\") pod \"node-exporter-5bx25\" (UID: \"8b2bb638-e334-44e1-9449-6645ca708da8\") " pod="openshift-monitoring/node-exporter-5bx25"
Sep 18 19:59:15.445841 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:15.445711    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8b2bb638-e334-44e1-9449-6645ca708da8-node-exporter-wtmp\") pod \"node-exporter-5bx25\" (UID: \"8b2bb638-e334-44e1-9449-6645ca708da8\") " pod="openshift-monitoring/node-exporter-5bx25"
Sep 18 19:59:15.445988 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:15.445856    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8b2bb638-e334-44e1-9449-6645ca708da8-sys\") pod \"node-exporter-5bx25\" (UID: \"8b2bb638-e334-44e1-9449-6645ca708da8\") " pod="openshift-monitoring/node-exporter-5bx25"
Sep 18 19:59:15.446262 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:15.446231    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8b2bb638-e334-44e1-9449-6645ca708da8-metrics-client-ca\") pod \"node-exporter-5bx25\" (UID: \"8b2bb638-e334-44e1-9449-6645ca708da8\") " pod="openshift-monitoring/node-exporter-5bx25"
Sep 18 19:59:15.446262 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:15.446238    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8b2bb638-e334-44e1-9449-6645ca708da8-node-exporter-accelerators-collector-config\") pod \"node-exporter-5bx25\" (UID: \"8b2bb638-e334-44e1-9449-6645ca708da8\") " pod="openshift-monitoring/node-exporter-5bx25"
Sep 18 19:59:15.448506 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:15.448463    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8b2bb638-e334-44e1-9449-6645ca708da8-node-exporter-tls\") pod \"node-exporter-5bx25\" (UID: \"8b2bb638-e334-44e1-9449-6645ca708da8\") " pod="openshift-monitoring/node-exporter-5bx25"
Sep 18 19:59:15.448606 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:15.448504    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8b2bb638-e334-44e1-9449-6645ca708da8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5bx25\" (UID: \"8b2bb638-e334-44e1-9449-6645ca708da8\") " pod="openshift-monitoring/node-exporter-5bx25"
Sep 18 19:59:15.458764 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:15.458740    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxc8s\" (UniqueName: \"kubernetes.io/projected/8b2bb638-e334-44e1-9449-6645ca708da8-kube-api-access-kxc8s\") pod \"node-exporter-5bx25\" (UID: \"8b2bb638-e334-44e1-9449-6645ca708da8\") " pod="openshift-monitoring/node-exporter-5bx25"
Sep 18 19:59:15.492305 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:15.492279    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5bx25"
Sep 18 19:59:15.533642 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:59:15.533617    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b2bb638_e334_44e1_9449_6645ca708da8.slice/crio-c26172d1d47372087896994b4d1c9108e149922316e11147e8bd73098fb8e6b1 WatchSource:0}: Error finding container c26172d1d47372087896994b4d1c9108e149922316e11147e8bd73098fb8e6b1: Status 404 returned error can't find the container with id c26172d1d47372087896994b4d1c9108e149922316e11147e8bd73098fb8e6b1
Sep 18 19:59:15.581551 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:15.581521    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-zshqm"]
Sep 18 19:59:15.584195 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:59:15.583967    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc22e65d0_1e2f_4a02_a878_2508729abe1f.slice/crio-b5212019be63353219db29dc315b1461c3054414a8bfdf67020e5c926d0eebf6 WatchSource:0}: Error finding container b5212019be63353219db29dc315b1461c3054414a8bfdf67020e5c926d0eebf6: Status 404 returned error can't find the container with id b5212019be63353219db29dc315b1461c3054414a8bfdf67020e5c926d0eebf6
Sep 18 19:59:16.182319 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:16.182278    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5bx25" event={"ID":"8b2bb638-e334-44e1-9449-6645ca708da8","Type":"ContainerStarted","Data":"c26172d1d47372087896994b4d1c9108e149922316e11147e8bd73098fb8e6b1"}
Sep 18 19:59:16.184957 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:16.184926    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-zshqm" event={"ID":"c22e65d0-1e2f-4a02-a878-2508729abe1f","Type":"ContainerStarted","Data":"2823c6375efafa66fa7a6a8a59681a4a369550b2b4a960dd31079c7997cbd0da"}
Sep 18 19:59:16.185174 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:16.185142    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-zshqm" event={"ID":"c22e65d0-1e2f-4a02-a878-2508729abe1f","Type":"ContainerStarted","Data":"1461868c124da0903b39bfb174b4b0724fd3ee78c33b78ad9cbd7d4012ab2fa2"}
Sep 18 19:59:16.185295 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:16.185182    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-zshqm" event={"ID":"c22e65d0-1e2f-4a02-a878-2508729abe1f","Type":"ContainerStarted","Data":"b5212019be63353219db29dc315b1461c3054414a8bfdf67020e5c926d0eebf6"}
Sep 18 19:59:17.582506 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:17.582470    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-758b887fb4-mmjkk"]
Sep 18 19:59:18.196711 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:18.196671    2549 generic.go:356] "Generic (PLEG): container finished" podID="8b2bb638-e334-44e1-9449-6645ca708da8" containerID="2e2a48d84c7466a3c1ccbd0c1dac002b9682d762e4e5f96c17375101b8c01fce" exitCode=0
Sep 18 19:59:18.196900 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:18.196757    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5bx25" event={"ID":"8b2bb638-e334-44e1-9449-6645ca708da8","Type":"ContainerDied","Data":"2e2a48d84c7466a3c1ccbd0c1dac002b9682d762e4e5f96c17375101b8c01fce"}
Sep 18 19:59:18.199540 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:18.199508    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-zshqm" event={"ID":"c22e65d0-1e2f-4a02-a878-2508729abe1f","Type":"ContainerStarted","Data":"685a499bf7fcba7f6aff98507d8bae698e787b9e2be25656445704763b8a1ea7"}
Sep 18 19:59:18.244181 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:18.244130    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-zshqm" podStartSLOduration=2.4612464 podStartE2EDuration="4.24410823s" totalImagesPullingTime="1.78286183s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:59:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:59:18.242682837 +0000 UTC m=+143.196424007" watchObservedRunningTime="2026-09-18 19:59:18.24410823 +0000 UTC m=+143.197849402"
Sep 18 19:59:19.204999 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:19.204962    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5bx25" event={"ID":"8b2bb638-e334-44e1-9449-6645ca708da8","Type":"ContainerStarted","Data":"e1448692321fa8685f63faa73251537ba26d1e39d2948401748a89c8d562b510"}
Sep 18 19:59:19.205491 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:19.205018    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5bx25" event={"ID":"8b2bb638-e334-44e1-9449-6645ca708da8","Type":"ContainerStarted","Data":"fe9d4f1209178da3f3699745c4a3e3f246ff4497c3fa421bf3cc94f74b9f8e26"}
Sep 18 19:59:19.242884 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:19.242843    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-5bx25" podStartSLOduration=2.092657423 podStartE2EDuration="4.242829255s" totalImagesPullingTime="2.065109662s" totalInitContainerRuntime="85.06217ms" isStatefulPod=true podCreationTimestamp="2026-09-18 19:59:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:59:19.241567233 +0000 UTC m=+144.195308404" watchObservedRunningTime="2026-09-18 19:59:19.242829255 +0000 UTC m=+144.196570422"
Sep 18 19:59:22.226738 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:22.226707    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-98b85f47-vrbpg"]
Sep 18 19:59:22.262825 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:22.262797    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-98b85f47-vrbpg"]
Sep 18 19:59:22.262954 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:22.262888    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-98b85f47-vrbpg"
Sep 18 19:59:22.272532 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:22.272509    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 18 19:59:22.304617 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:22.304595    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9144c0d3-c3f8-4dda-9501-a4799332c6da-console-serving-cert\") pod \"console-98b85f47-vrbpg\" (UID: \"9144c0d3-c3f8-4dda-9501-a4799332c6da\") " pod="openshift-console/console-98b85f47-vrbpg"
Sep 18 19:59:22.304721 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:22.304628    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9144c0d3-c3f8-4dda-9501-a4799332c6da-console-oauth-config\") pod \"console-98b85f47-vrbpg\" (UID: \"9144c0d3-c3f8-4dda-9501-a4799332c6da\") " pod="openshift-console/console-98b85f47-vrbpg"
Sep 18 19:59:22.304721 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:22.304647    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9144c0d3-c3f8-4dda-9501-a4799332c6da-service-ca\") pod \"console-98b85f47-vrbpg\" (UID: \"9144c0d3-c3f8-4dda-9501-a4799332c6da\") " pod="openshift-console/console-98b85f47-vrbpg"
Sep 18 19:59:22.304721 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:22.304664    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9144c0d3-c3f8-4dda-9501-a4799332c6da-oauth-serving-cert\") pod \"console-98b85f47-vrbpg\" (UID: \"9144c0d3-c3f8-4dda-9501-a4799332c6da\") " pod="openshift-console/console-98b85f47-vrbpg"
Sep 18 19:59:22.304861 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:22.304749    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9144c0d3-c3f8-4dda-9501-a4799332c6da-trusted-ca-bundle\") pod \"console-98b85f47-vrbpg\" (UID: \"9144c0d3-c3f8-4dda-9501-a4799332c6da\") " pod="openshift-console/console-98b85f47-vrbpg"
Sep 18 19:59:22.304861 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:22.304785    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9144c0d3-c3f8-4dda-9501-a4799332c6da-console-config\") pod \"console-98b85f47-vrbpg\" (UID: \"9144c0d3-c3f8-4dda-9501-a4799332c6da\") " pod="openshift-console/console-98b85f47-vrbpg"
Sep 18 19:59:22.304925 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:22.304884    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbrxn\" (UniqueName: \"kubernetes.io/projected/9144c0d3-c3f8-4dda-9501-a4799332c6da-kube-api-access-zbrxn\") pod \"console-98b85f47-vrbpg\" (UID: \"9144c0d3-c3f8-4dda-9501-a4799332c6da\") " pod="openshift-console/console-98b85f47-vrbpg"
Sep 18 19:59:22.405748 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:22.405722    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9144c0d3-c3f8-4dda-9501-a4799332c6da-trusted-ca-bundle\") pod \"console-98b85f47-vrbpg\" (UID: \"9144c0d3-c3f8-4dda-9501-a4799332c6da\") " pod="openshift-console/console-98b85f47-vrbpg"
Sep 18 19:59:22.405855 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:22.405751    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9144c0d3-c3f8-4dda-9501-a4799332c6da-console-config\") pod \"console-98b85f47-vrbpg\" (UID: \"9144c0d3-c3f8-4dda-9501-a4799332c6da\") " pod="openshift-console/console-98b85f47-vrbpg"
Sep 18 19:59:22.405855 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:22.405793    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbrxn\" (UniqueName: \"kubernetes.io/projected/9144c0d3-c3f8-4dda-9501-a4799332c6da-kube-api-access-zbrxn\") pod \"console-98b85f47-vrbpg\" (UID: \"9144c0d3-c3f8-4dda-9501-a4799332c6da\") " pod="openshift-console/console-98b85f47-vrbpg"
Sep 18 19:59:22.405855 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:22.405837    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9144c0d3-c3f8-4dda-9501-a4799332c6da-console-serving-cert\") pod \"console-98b85f47-vrbpg\" (UID: \"9144c0d3-c3f8-4dda-9501-a4799332c6da\") " pod="openshift-console/console-98b85f47-vrbpg"
Sep 18 19:59:22.405985 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:22.405879    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9144c0d3-c3f8-4dda-9501-a4799332c6da-console-oauth-config\") pod \"console-98b85f47-vrbpg\" (UID: \"9144c0d3-c3f8-4dda-9501-a4799332c6da\") " pod="openshift-console/console-98b85f47-vrbpg"
Sep 18 19:59:22.405985 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:22.405904    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9144c0d3-c3f8-4dda-9501-a4799332c6da-service-ca\") pod \"console-98b85f47-vrbpg\" (UID: \"9144c0d3-c3f8-4dda-9501-a4799332c6da\") " pod="openshift-console/console-98b85f47-vrbpg"
Sep 18 19:59:22.405985 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:22.405925    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9144c0d3-c3f8-4dda-9501-a4799332c6da-oauth-serving-cert\") pod \"console-98b85f47-vrbpg\" (UID: \"9144c0d3-c3f8-4dda-9501-a4799332c6da\") " pod="openshift-console/console-98b85f47-vrbpg"
Sep 18 19:59:22.406591 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:22.406563    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9144c0d3-c3f8-4dda-9501-a4799332c6da-console-config\") pod \"console-98b85f47-vrbpg\" (UID: \"9144c0d3-c3f8-4dda-9501-a4799332c6da\") " pod="openshift-console/console-98b85f47-vrbpg"
Sep 18 19:59:22.406699 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:22.406648    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9144c0d3-c3f8-4dda-9501-a4799332c6da-trusted-ca-bundle\") pod \"console-98b85f47-vrbpg\" (UID: \"9144c0d3-c3f8-4dda-9501-a4799332c6da\") " pod="openshift-console/console-98b85f47-vrbpg"
Sep 18 19:59:22.406699 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:22.406659    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9144c0d3-c3f8-4dda-9501-a4799332c6da-oauth-serving-cert\") pod \"console-98b85f47-vrbpg\" (UID: \"9144c0d3-c3f8-4dda-9501-a4799332c6da\") " pod="openshift-console/console-98b85f47-vrbpg"
Sep 18 19:59:22.406699 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:22.406654    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9144c0d3-c3f8-4dda-9501-a4799332c6da-service-ca\") pod \"console-98b85f47-vrbpg\" (UID: \"9144c0d3-c3f8-4dda-9501-a4799332c6da\") " pod="openshift-console/console-98b85f47-vrbpg"
Sep 18 19:59:22.408225 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:22.408199    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9144c0d3-c3f8-4dda-9501-a4799332c6da-console-serving-cert\") pod \"console-98b85f47-vrbpg\" (UID: \"9144c0d3-c3f8-4dda-9501-a4799332c6da\") " pod="openshift-console/console-98b85f47-vrbpg"
Sep 18 19:59:22.408303 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:22.408290    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9144c0d3-c3f8-4dda-9501-a4799332c6da-console-oauth-config\") pod \"console-98b85f47-vrbpg\" (UID: \"9144c0d3-c3f8-4dda-9501-a4799332c6da\") " pod="openshift-console/console-98b85f47-vrbpg"
Sep 18 19:59:22.415581 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:22.415563    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbrxn\" (UniqueName: \"kubernetes.io/projected/9144c0d3-c3f8-4dda-9501-a4799332c6da-kube-api-access-zbrxn\") pod \"console-98b85f47-vrbpg\" (UID: \"9144c0d3-c3f8-4dda-9501-a4799332c6da\") " pod="openshift-console/console-98b85f47-vrbpg"
Sep 18 19:59:22.572212 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:22.572178    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-98b85f47-vrbpg"
Sep 18 19:59:22.869766 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:22.869742    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-98b85f47-vrbpg"]
Sep 18 19:59:22.872091 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:59:22.872065    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9144c0d3_c3f8_4dda_9501_a4799332c6da.slice/crio-2f80db2b26e2583729bc5c556e54c9f0e64dcb0861be2c05743eaafc98f357b3 WatchSource:0}: Error finding container 2f80db2b26e2583729bc5c556e54c9f0e64dcb0861be2c05743eaafc98f357b3: Status 404 returned error can't find the container with id 2f80db2b26e2583729bc5c556e54c9f0e64dcb0861be2c05743eaafc98f357b3
Sep 18 19:59:23.222694 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:23.222623    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-98b85f47-vrbpg" event={"ID":"9144c0d3-c3f8-4dda-9501-a4799332c6da","Type":"ContainerStarted","Data":"b2832467d5988b587ed371b991410dea9a77ee1d8d593654fbf8c8ed408c7c72"}
Sep 18 19:59:23.222694 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:23.222671    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-98b85f47-vrbpg" event={"ID":"9144c0d3-c3f8-4dda-9501-a4799332c6da","Type":"ContainerStarted","Data":"2f80db2b26e2583729bc5c556e54c9f0e64dcb0861be2c05743eaafc98f357b3"}
Sep 18 19:59:23.258402 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:23.258335    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-98b85f47-vrbpg" podStartSLOduration=1.258314297 podStartE2EDuration="1.258314297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:59:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:59:23.257486278 +0000 UTC m=+148.211227447" watchObservedRunningTime="2026-09-18 19:59:23.258314297 +0000 UTC m=+148.212055465"
Sep 18 19:59:31.890836 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:59:31.890795    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-mbfbc" podUID="0b1d2455-d7fc-4bde-a435-7dfaedd8390e"
Sep 18 19:59:32.253304 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:32.253229    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mbfbc"
Sep 18 19:59:32.573045 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:32.573007    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-98b85f47-vrbpg"
Sep 18 19:59:32.573198 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:32.573065    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-98b85f47-vrbpg"
Sep 18 19:59:32.578354 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:32.578332    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-98b85f47-vrbpg"
Sep 18 19:59:33.261176 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:33.261148    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-98b85f47-vrbpg"
Sep 18 19:59:36.715633 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:36.715594    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0b1d2455-d7fc-4bde-a435-7dfaedd8390e-metrics-tls\") pod \"dns-default-mbfbc\" (UID: \"0b1d2455-d7fc-4bde-a435-7dfaedd8390e\") " pod="openshift-dns/dns-default-mbfbc"
Sep 18 19:59:36.717821 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:36.717799    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0b1d2455-d7fc-4bde-a435-7dfaedd8390e-metrics-tls\") pod \"dns-default-mbfbc\" (UID: \"0b1d2455-d7fc-4bde-a435-7dfaedd8390e\") " pod="openshift-dns/dns-default-mbfbc"
Sep 18 19:59:36.757397 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:36.757358    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-r2wjd\""
Sep 18 19:59:36.764878 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:36.764864    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mbfbc"
Sep 18 19:59:37.047200 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:37.047159    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mbfbc"]
Sep 18 19:59:37.049725 ip-10-0-140-178 kubenswrapper[2549]: W0918 19:59:37.049694    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b1d2455_d7fc_4bde_a435_7dfaedd8390e.slice/crio-f95b57d7eb6e26a2bbeaa4b5ad26fa4203e090278873caf83b0301ffaa95586b WatchSource:0}: Error finding container f95b57d7eb6e26a2bbeaa4b5ad26fa4203e090278873caf83b0301ffaa95586b: Status 404 returned error can't find the container with id f95b57d7eb6e26a2bbeaa4b5ad26fa4203e090278873caf83b0301ffaa95586b
Sep 18 19:59:37.272877 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:37.272843    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mbfbc" event={"ID":"0b1d2455-d7fc-4bde-a435-7dfaedd8390e","Type":"ContainerStarted","Data":"f95b57d7eb6e26a2bbeaa4b5ad26fa4203e090278873caf83b0301ffaa95586b"}
Sep 18 19:59:40.285610 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:40.285573    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mbfbc" event={"ID":"0b1d2455-d7fc-4bde-a435-7dfaedd8390e","Type":"ContainerStarted","Data":"d7d29bb7b5136e536603b690052f0bd67f145baec21b327328c08f493cd502ff"}
Sep 18 19:59:40.285998 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:40.285653    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mbfbc" event={"ID":"0b1d2455-d7fc-4bde-a435-7dfaedd8390e","Type":"ContainerStarted","Data":"9dca14b65f57321bffd0109eb95cce733dcf932753f46f2a78f8532d0cb71a69"}
Sep 18 19:59:40.285998 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:40.285717    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-mbfbc"
Sep 18 19:59:42.607429 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:42.607377    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-758b887fb4-mmjkk" podUID="91c6036d-63f3-407e-bea7-18510ae4e95a" containerName="console" containerID="cri-o://3f0be08ce12ea8653fe8f7da8e99426f948328fbfd27d5b66211536bb9aaa313" gracePeriod=15
Sep 18 19:59:42.885716 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:42.885659    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-758b887fb4-mmjkk"
Sep 18 19:59:42.907985 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:42.907943    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-mbfbc" podStartSLOduration=132.662086425 podStartE2EDuration="2m14.907923357s" totalImagesPullingTime="2.245836932s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:57:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:59:40.306647825 +0000 UTC m=+165.260388992" watchObservedRunningTime="2026-09-18 19:59:42.907923357 +0000 UTC m=+167.861664535"
Sep 18 19:59:42.964310 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:42.964287    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/91c6036d-63f3-407e-bea7-18510ae4e95a-service-ca\") pod \"91c6036d-63f3-407e-bea7-18510ae4e95a\" (UID: \"91c6036d-63f3-407e-bea7-18510ae4e95a\") "
Sep 18 19:59:42.964469 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:42.964364    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/91c6036d-63f3-407e-bea7-18510ae4e95a-console-oauth-config\") pod \"91c6036d-63f3-407e-bea7-18510ae4e95a\" (UID: \"91c6036d-63f3-407e-bea7-18510ae4e95a\") "
Sep 18 19:59:42.964469 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:42.964398    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c2ffq\" (UniqueName: \"kubernetes.io/projected/91c6036d-63f3-407e-bea7-18510ae4e95a-kube-api-access-c2ffq\") pod \"91c6036d-63f3-407e-bea7-18510ae4e95a\" (UID: \"91c6036d-63f3-407e-bea7-18510ae4e95a\") "
Sep 18 19:59:42.964469 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:42.964424    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/91c6036d-63f3-407e-bea7-18510ae4e95a-console-config\") pod \"91c6036d-63f3-407e-bea7-18510ae4e95a\" (UID: \"91c6036d-63f3-407e-bea7-18510ae4e95a\") "
Sep 18 19:59:42.964469 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:42.964445    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/91c6036d-63f3-407e-bea7-18510ae4e95a-oauth-serving-cert\") pod \"91c6036d-63f3-407e-bea7-18510ae4e95a\" (UID: \"91c6036d-63f3-407e-bea7-18510ae4e95a\") "
Sep 18 19:59:42.964680 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:42.964476    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/91c6036d-63f3-407e-bea7-18510ae4e95a-console-serving-cert\") pod \"91c6036d-63f3-407e-bea7-18510ae4e95a\" (UID: \"91c6036d-63f3-407e-bea7-18510ae4e95a\") "
Sep 18 19:59:42.964829 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:42.964796    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91c6036d-63f3-407e-bea7-18510ae4e95a-service-ca" (OuterVolumeSpecName: "service-ca") pod "91c6036d-63f3-407e-bea7-18510ae4e95a" (UID: "91c6036d-63f3-407e-bea7-18510ae4e95a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:59:42.964829 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:42.964804    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91c6036d-63f3-407e-bea7-18510ae4e95a-console-config" (OuterVolumeSpecName: "console-config") pod "91c6036d-63f3-407e-bea7-18510ae4e95a" (UID: "91c6036d-63f3-407e-bea7-18510ae4e95a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:59:42.964996 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:42.964852    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91c6036d-63f3-407e-bea7-18510ae4e95a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "91c6036d-63f3-407e-bea7-18510ae4e95a" (UID: "91c6036d-63f3-407e-bea7-18510ae4e95a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:59:42.966719 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:42.966690    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91c6036d-63f3-407e-bea7-18510ae4e95a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "91c6036d-63f3-407e-bea7-18510ae4e95a" (UID: "91c6036d-63f3-407e-bea7-18510ae4e95a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:59:42.966814 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:42.966764    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91c6036d-63f3-407e-bea7-18510ae4e95a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "91c6036d-63f3-407e-bea7-18510ae4e95a" (UID: "91c6036d-63f3-407e-bea7-18510ae4e95a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:59:42.966814 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:42.966785    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91c6036d-63f3-407e-bea7-18510ae4e95a-kube-api-access-c2ffq" (OuterVolumeSpecName: "kube-api-access-c2ffq") pod "91c6036d-63f3-407e-bea7-18510ae4e95a" (UID: "91c6036d-63f3-407e-bea7-18510ae4e95a"). InnerVolumeSpecName "kube-api-access-c2ffq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:59:43.065266 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:43.065240    2549 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/91c6036d-63f3-407e-bea7-18510ae4e95a-console-oauth-config\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 19:59:43.065266 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:43.065267    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2ffq\" (UniqueName: \"kubernetes.io/projected/91c6036d-63f3-407e-bea7-18510ae4e95a-kube-api-access-c2ffq\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 19:59:43.065437 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:43.065280    2549 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/91c6036d-63f3-407e-bea7-18510ae4e95a-console-config\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 19:59:43.065437 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:43.065293    2549 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/91c6036d-63f3-407e-bea7-18510ae4e95a-oauth-serving-cert\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 19:59:43.065437 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:43.065305    2549 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/91c6036d-63f3-407e-bea7-18510ae4e95a-console-serving-cert\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 19:59:43.065437 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:43.065317    2549 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/91c6036d-63f3-407e-bea7-18510ae4e95a-service-ca\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 19:59:43.297710 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:43.297631    2549 generic.go:356] "Generic (PLEG): container finished" podID="91c6036d-63f3-407e-bea7-18510ae4e95a" containerID="3f0be08ce12ea8653fe8f7da8e99426f948328fbfd27d5b66211536bb9aaa313" exitCode=0
Sep 18 19:59:43.297710 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:43.297673    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-758b887fb4-mmjkk" event={"ID":"91c6036d-63f3-407e-bea7-18510ae4e95a","Type":"ContainerDied","Data":"3f0be08ce12ea8653fe8f7da8e99426f948328fbfd27d5b66211536bb9aaa313"}
Sep 18 19:59:43.297710 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:43.297680    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-758b887fb4-mmjkk"
Sep 18 19:59:43.297928 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:43.297721    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-758b887fb4-mmjkk" event={"ID":"91c6036d-63f3-407e-bea7-18510ae4e95a","Type":"ContainerDied","Data":"dcd8eb9a233df15d18af82a5a67d7a993f771593a0ab7889e3c78de46d9bd5bc"}
Sep 18 19:59:43.297928 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:43.297749    2549 scope.go:117] "RemoveContainer" containerID="3f0be08ce12ea8653fe8f7da8e99426f948328fbfd27d5b66211536bb9aaa313"
Sep 18 19:59:43.308359 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:43.308339    2549 scope.go:117] "RemoveContainer" containerID="3f0be08ce12ea8653fe8f7da8e99426f948328fbfd27d5b66211536bb9aaa313"
Sep 18 19:59:43.308764 ip-10-0-140-178 kubenswrapper[2549]: E0918 19:59:43.308743    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f0be08ce12ea8653fe8f7da8e99426f948328fbfd27d5b66211536bb9aaa313\": container with ID starting with 3f0be08ce12ea8653fe8f7da8e99426f948328fbfd27d5b66211536bb9aaa313 not found: ID does not exist" containerID="3f0be08ce12ea8653fe8f7da8e99426f948328fbfd27d5b66211536bb9aaa313"
Sep 18 19:59:43.308843 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:43.308773    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f0be08ce12ea8653fe8f7da8e99426f948328fbfd27d5b66211536bb9aaa313"} err="failed to get container status \"3f0be08ce12ea8653fe8f7da8e99426f948328fbfd27d5b66211536bb9aaa313\": rpc error: code = NotFound desc = could not find container \"3f0be08ce12ea8653fe8f7da8e99426f948328fbfd27d5b66211536bb9aaa313\": container with ID starting with 3f0be08ce12ea8653fe8f7da8e99426f948328fbfd27d5b66211536bb9aaa313 not found: ID does not exist"
Sep 18 19:59:43.327129 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:43.327105    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-758b887fb4-mmjkk"]
Sep 18 19:59:43.329834 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:43.329808    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-758b887fb4-mmjkk"]
Sep 18 19:59:43.519868 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:43.519843    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91c6036d-63f3-407e-bea7-18510ae4e95a" path="/var/lib/kubelet/pods/91c6036d-63f3-407e-bea7-18510ae4e95a/volumes"
Sep 18 19:59:44.302988 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:44.302956    2549 generic.go:356] "Generic (PLEG): container finished" podID="174ef0f9-1b9f-48b9-b6c1-1fb01a1f7b1b" containerID="485d9c5b2ddaea18ad1b2a8d38d09d1fb9f41c7d77307efb7e83f5563124f7a9" exitCode=0
Sep 18 19:59:44.303432 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:44.303036    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-sfdnq" event={"ID":"174ef0f9-1b9f-48b9-b6c1-1fb01a1f7b1b","Type":"ContainerDied","Data":"485d9c5b2ddaea18ad1b2a8d38d09d1fb9f41c7d77307efb7e83f5563124f7a9"}
Sep 18 19:59:44.303432 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:44.303358    2549 scope.go:117] "RemoveContainer" containerID="485d9c5b2ddaea18ad1b2a8d38d09d1fb9f41c7d77307efb7e83f5563124f7a9"
Sep 18 19:59:45.309983 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:45.309948    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-sfdnq" event={"ID":"174ef0f9-1b9f-48b9-b6c1-1fb01a1f7b1b","Type":"ContainerStarted","Data":"38058c7de736e6577eba20c9cf8ae0c1f67eb966aec5b7edbb89c87af6ac02b3"}
Sep 18 19:59:50.291918 ip-10-0-140-178 kubenswrapper[2549]: I0918 19:59:50.291883    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-mbfbc"
Sep 18 20:00:39.234925 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:00:39.234886    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-b648968bd-mxxcn"]
Sep 18 20:00:39.235374 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:00:39.235164    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91c6036d-63f3-407e-bea7-18510ae4e95a" containerName="console"
Sep 18 20:00:39.235374 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:00:39.235174    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="91c6036d-63f3-407e-bea7-18510ae4e95a" containerName="console"
Sep 18 20:00:39.235374 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:00:39.235230    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="91c6036d-63f3-407e-bea7-18510ae4e95a" containerName="console"
Sep 18 20:00:39.237154 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:00:39.237137    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-b648968bd-mxxcn"
Sep 18 20:00:39.239994 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:00:39.239967    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 18 20:00:39.240129 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:00:39.240018    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-bhxvt\""
Sep 18 20:00:39.240188 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:00:39.240158    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 18 20:00:39.240630 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:00:39.240613    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 18 20:00:39.240951 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:00:39.240934    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 18 20:00:39.248215 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:00:39.248195    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 18 20:00:39.251344 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:00:39.251323    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 18 20:00:39.261420 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:00:39.261395    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-b648968bd-mxxcn"]
Sep 18 20:00:39.279253 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:00:39.279202    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/79dc0bc7-5103-416d-a44f-fb5efacc8867-telemeter-client-tls\") pod \"telemeter-client-b648968bd-mxxcn\" (UID: \"79dc0bc7-5103-416d-a44f-fb5efacc8867\") " pod="openshift-monitoring/telemeter-client-b648968bd-mxxcn"
Sep 18 20:00:39.279253 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:00:39.279256    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/79dc0bc7-5103-416d-a44f-fb5efacc8867-federate-client-tls\") pod \"telemeter-client-b648968bd-mxxcn\" (UID: \"79dc0bc7-5103-416d-a44f-fb5efacc8867\") " pod="openshift-monitoring/telemeter-client-b648968bd-mxxcn"
Sep 18 20:00:39.279465 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:00:39.279321    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/79dc0bc7-5103-416d-a44f-fb5efacc8867-secret-telemeter-client\") pod \"telemeter-client-b648968bd-mxxcn\" (UID: \"79dc0bc7-5103-416d-a44f-fb5efacc8867\") " pod="openshift-monitoring/telemeter-client-b648968bd-mxxcn"
Sep 18 20:00:39.279465 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:00:39.279348    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79dc0bc7-5103-416d-a44f-fb5efacc8867-serving-certs-ca-bundle\") pod \"telemeter-client-b648968bd-mxxcn\" (UID: \"79dc0bc7-5103-416d-a44f-fb5efacc8867\") " pod="openshift-monitoring/telemeter-client-b648968bd-mxxcn"
Sep 18 20:00:39.279465 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:00:39.279392    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/79dc0bc7-5103-416d-a44f-fb5efacc8867-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-b648968bd-mxxcn\" (UID: \"79dc0bc7-5103-416d-a44f-fb5efacc8867\") " pod="openshift-monitoring/telemeter-client-b648968bd-mxxcn"
Sep 18 20:00:39.279465 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:00:39.279448    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79dc0bc7-5103-416d-a44f-fb5efacc8867-telemeter-trusted-ca-bundle\") pod \"telemeter-client-b648968bd-mxxcn\" (UID: \"79dc0bc7-5103-416d-a44f-fb5efacc8867\") " pod="openshift-monitoring/telemeter-client-b648968bd-mxxcn"
Sep 18 20:00:39.279628 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:00:39.279492    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59k7k\" (UniqueName: \"kubernetes.io/projected/79dc0bc7-5103-416d-a44f-fb5efacc8867-kube-api-access-59k7k\") pod \"telemeter-client-b648968bd-mxxcn\" (UID: \"79dc0bc7-5103-416d-a44f-fb5efacc8867\") " pod="openshift-monitoring/telemeter-client-b648968bd-mxxcn"
Sep 18 20:00:39.279628 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:00:39.279551    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/79dc0bc7-5103-416d-a44f-fb5efacc8867-metrics-client-ca\") pod \"telemeter-client-b648968bd-mxxcn\" (UID: \"79dc0bc7-5103-416d-a44f-fb5efacc8867\") " pod="openshift-monitoring/telemeter-client-b648968bd-mxxcn"
Sep 18 20:00:39.380283 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:00:39.380244    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/79dc0bc7-5103-416d-a44f-fb5efacc8867-secret-telemeter-client\") pod \"telemeter-client-b648968bd-mxxcn\" (UID: \"79dc0bc7-5103-416d-a44f-fb5efacc8867\") " pod="openshift-monitoring/telemeter-client-b648968bd-mxxcn"
Sep 18 20:00:39.380283 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:00:39.380287    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79dc0bc7-5103-416d-a44f-fb5efacc8867-serving-certs-ca-bundle\") pod \"telemeter-client-b648968bd-mxxcn\" (UID: \"79dc0bc7-5103-416d-a44f-fb5efacc8867\") " pod="openshift-monitoring/telemeter-client-b648968bd-mxxcn"
Sep 18 20:00:39.380542 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:00:39.380329    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/79dc0bc7-5103-416d-a44f-fb5efacc8867-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-b648968bd-mxxcn\" (UID: \"79dc0bc7-5103-416d-a44f-fb5efacc8867\") " pod="openshift-monitoring/telemeter-client-b648968bd-mxxcn"
Sep 18 20:00:39.380542 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:00:39.380360    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79dc0bc7-5103-416d-a44f-fb5efacc8867-telemeter-trusted-ca-bundle\") pod \"telemeter-client-b648968bd-mxxcn\" (UID: \"79dc0bc7-5103-416d-a44f-fb5efacc8867\") " pod="openshift-monitoring/telemeter-client-b648968bd-mxxcn"
Sep 18 20:00:39.380542 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:00:39.380403    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59k7k\" (UniqueName: \"kubernetes.io/projected/79dc0bc7-5103-416d-a44f-fb5efacc8867-kube-api-access-59k7k\") pod \"telemeter-client-b648968bd-mxxcn\" (UID: \"79dc0bc7-5103-416d-a44f-fb5efacc8867\") " pod="openshift-monitoring/telemeter-client-b648968bd-mxxcn"
Sep 18 20:00:39.380542 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:00:39.380426    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/79dc0bc7-5103-416d-a44f-fb5efacc8867-metrics-client-ca\") pod \"telemeter-client-b648968bd-mxxcn\" (UID: \"79dc0bc7-5103-416d-a44f-fb5efacc8867\") " pod="openshift-monitoring/telemeter-client-b648968bd-mxxcn"
Sep 18 20:00:39.380542 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:00:39.380478    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/79dc0bc7-5103-416d-a44f-fb5efacc8867-telemeter-client-tls\") pod \"telemeter-client-b648968bd-mxxcn\" (UID: \"79dc0bc7-5103-416d-a44f-fb5efacc8867\") " pod="openshift-monitoring/telemeter-client-b648968bd-mxxcn"
Sep 18 20:00:39.380542 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:00:39.380511    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/79dc0bc7-5103-416d-a44f-fb5efacc8867-federate-client-tls\") pod \"telemeter-client-b648968bd-mxxcn\" (UID: \"79dc0bc7-5103-416d-a44f-fb5efacc8867\") " pod="openshift-monitoring/telemeter-client-b648968bd-mxxcn"
Sep 18 20:00:39.381595 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:00:39.381570    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/79dc0bc7-5103-416d-a44f-fb5efacc8867-metrics-client-ca\") pod \"telemeter-client-b648968bd-mxxcn\" (UID: \"79dc0bc7-5103-416d-a44f-fb5efacc8867\") " pod="openshift-monitoring/telemeter-client-b648968bd-mxxcn"
Sep 18 20:00:39.381788 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:00:39.381694    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79dc0bc7-5103-416d-a44f-fb5efacc8867-serving-certs-ca-bundle\") pod \"telemeter-client-b648968bd-mxxcn\" (UID: \"79dc0bc7-5103-416d-a44f-fb5efacc8867\") " pod="openshift-monitoring/telemeter-client-b648968bd-mxxcn"
Sep 18 20:00:39.382909 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:00:39.382883    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79dc0bc7-5103-416d-a44f-fb5efacc8867-telemeter-trusted-ca-bundle\") pod \"telemeter-client-b648968bd-mxxcn\" (UID: \"79dc0bc7-5103-416d-a44f-fb5efacc8867\") " pod="openshift-monitoring/telemeter-client-b648968bd-mxxcn"
Sep 18 20:00:39.383671 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:00:39.383647    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/79dc0bc7-5103-416d-a44f-fb5efacc8867-federate-client-tls\") pod \"telemeter-client-b648968bd-mxxcn\" (UID: \"79dc0bc7-5103-416d-a44f-fb5efacc8867\") " pod="openshift-monitoring/telemeter-client-b648968bd-mxxcn"
Sep 18 20:00:39.383811 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:00:39.383788    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/79dc0bc7-5103-416d-a44f-fb5efacc8867-secret-telemeter-client\") pod \"telemeter-client-b648968bd-mxxcn\" (UID: \"79dc0bc7-5103-416d-a44f-fb5efacc8867\") " pod="openshift-monitoring/telemeter-client-b648968bd-mxxcn"
Sep 18 20:00:39.384051 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:00:39.384034    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/79dc0bc7-5103-416d-a44f-fb5efacc8867-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-b648968bd-mxxcn\" (UID: \"79dc0bc7-5103-416d-a44f-fb5efacc8867\") " pod="openshift-monitoring/telemeter-client-b648968bd-mxxcn"
Sep 18 20:00:39.384453 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:00:39.384431    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/79dc0bc7-5103-416d-a44f-fb5efacc8867-telemeter-client-tls\") pod \"telemeter-client-b648968bd-mxxcn\" (UID: \"79dc0bc7-5103-416d-a44f-fb5efacc8867\") " pod="openshift-monitoring/telemeter-client-b648968bd-mxxcn"
Sep 18 20:00:39.390554 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:00:39.390534    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59k7k\" (UniqueName: \"kubernetes.io/projected/79dc0bc7-5103-416d-a44f-fb5efacc8867-kube-api-access-59k7k\") pod \"telemeter-client-b648968bd-mxxcn\" (UID: \"79dc0bc7-5103-416d-a44f-fb5efacc8867\") " pod="openshift-monitoring/telemeter-client-b648968bd-mxxcn"
Sep 18 20:00:39.546636 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:00:39.546543    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-b648968bd-mxxcn"
Sep 18 20:00:39.837311 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:00:39.837275    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-b648968bd-mxxcn"]
Sep 18 20:00:39.841250 ip-10-0-140-178 kubenswrapper[2549]: W0918 20:00:39.841200    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79dc0bc7_5103_416d_a44f_fb5efacc8867.slice/crio-563d5185ad31e08067ae6132d9f640ca7bfee76f56048dd8afcae8d8738d9810 WatchSource:0}: Error finding container 563d5185ad31e08067ae6132d9f640ca7bfee76f56048dd8afcae8d8738d9810: Status 404 returned error can't find the container with id 563d5185ad31e08067ae6132d9f640ca7bfee76f56048dd8afcae8d8738d9810
Sep 18 20:00:40.522909 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:00:40.522870    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-b648968bd-mxxcn" event={"ID":"79dc0bc7-5103-416d-a44f-fb5efacc8867","Type":"ContainerStarted","Data":"563d5185ad31e08067ae6132d9f640ca7bfee76f56048dd8afcae8d8738d9810"}
Sep 18 20:00:42.532312 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:00:42.532278    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-b648968bd-mxxcn" event={"ID":"79dc0bc7-5103-416d-a44f-fb5efacc8867","Type":"ContainerStarted","Data":"555d1a57d79047347714529bad7c18d23a2d0bd56a6886dc29fb0766a9108e98"}
Sep 18 20:00:43.538565 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:00:43.538485    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-b648968bd-mxxcn" event={"ID":"79dc0bc7-5103-416d-a44f-fb5efacc8867","Type":"ContainerStarted","Data":"18cbc61c39780504a5471dbef1219497301fd8d45003a0cd0ba8e8c6da7de535"}
Sep 18 20:00:43.538565 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:00:43.538537    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-b648968bd-mxxcn" event={"ID":"79dc0bc7-5103-416d-a44f-fb5efacc8867","Type":"ContainerStarted","Data":"bf9c720f62280254e1610693f93b0b3fe6523caa3a4aff3fef8c363bdbf3a50c"}
Sep 18 20:00:43.691004 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:00:43.690958    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-b648968bd-mxxcn" podStartSLOduration=1.483226073 podStartE2EDuration="4.690942234s" totalImagesPullingTime="3.207716161s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:00:39 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:00:43.567528963 +0000 UTC m=+228.521270131" watchObservedRunningTime="2026-09-18 20:00:43.690942234 +0000 UTC m=+228.644683401"
Sep 18 20:00:43.691466 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:00:43.691451    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-85c6769964-xw6b7"]
Sep 18 20:00:43.693441 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:00:43.693426    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85c6769964-xw6b7"
Sep 18 20:00:43.708718 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:00:43.708363    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f12daa1a-924a-49f0-9000-b08c97a7ec17-console-serving-cert\") pod \"console-85c6769964-xw6b7\" (UID: \"f12daa1a-924a-49f0-9000-b08c97a7ec17\") " pod="openshift-console/console-85c6769964-xw6b7"
Sep 18 20:00:43.708718 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:00:43.708457    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f12daa1a-924a-49f0-9000-b08c97a7ec17-console-oauth-config\") pod \"console-85c6769964-xw6b7\" (UID: \"f12daa1a-924a-49f0-9000-b08c97a7ec17\") " pod="openshift-console/console-85c6769964-xw6b7"
Sep 18 20:00:43.708718 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:00:43.708491    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f12daa1a-924a-49f0-9000-b08c97a7ec17-console-config\") pod \"console-85c6769964-xw6b7\" (UID: \"f12daa1a-924a-49f0-9000-b08c97a7ec17\") " pod="openshift-console/console-85c6769964-xw6b7"
Sep 18 20:00:43.708718 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:00:43.708550    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f12daa1a-924a-49f0-9000-b08c97a7ec17-service-ca\") pod \"console-85c6769964-xw6b7\" (UID: \"f12daa1a-924a-49f0-9000-b08c97a7ec17\") " pod="openshift-console/console-85c6769964-xw6b7"
Sep 18 20:00:43.708718 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:00:43.708582    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxbkb\" (UniqueName: \"kubernetes.io/projected/f12daa1a-924a-49f0-9000-b08c97a7ec17-kube-api-access-bxbkb\") pod \"console-85c6769964-xw6b7\" (UID: \"f12daa1a-924a-49f0-9000-b08c97a7ec17\") " pod="openshift-console/console-85c6769964-xw6b7"
Sep 18 20:00:43.708718 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:00:43.708605    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f12daa1a-924a-49f0-9000-b08c97a7ec17-oauth-serving-cert\") pod \"console-85c6769964-xw6b7\" (UID: \"f12daa1a-924a-49f0-9000-b08c97a7ec17\") " pod="openshift-console/console-85c6769964-xw6b7"
Sep 18 20:00:43.708718 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:00:43.708633    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f12daa1a-924a-49f0-9000-b08c97a7ec17-trusted-ca-bundle\") pod \"console-85c6769964-xw6b7\" (UID: \"f12daa1a-924a-49f0-9000-b08c97a7ec17\") " pod="openshift-console/console-85c6769964-xw6b7"
Sep 18 20:00:43.717506 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:00:43.717485    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85c6769964-xw6b7"]
Sep 18 20:00:43.809141 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:00:43.809061    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f12daa1a-924a-49f0-9000-b08c97a7ec17-console-config\") pod \"console-85c6769964-xw6b7\" (UID: \"f12daa1a-924a-49f0-9000-b08c97a7ec17\") " pod="openshift-console/console-85c6769964-xw6b7"
Sep 18 20:00:43.809141 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:00:43.809125    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f12daa1a-924a-49f0-9000-b08c97a7ec17-service-ca\") pod \"console-85c6769964-xw6b7\" (UID: \"f12daa1a-924a-49f0-9000-b08c97a7ec17\") " pod="openshift-console/console-85c6769964-xw6b7"
Sep 18 20:00:43.809344 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:00:43.809159    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxbkb\" (UniqueName: \"kubernetes.io/projected/f12daa1a-924a-49f0-9000-b08c97a7ec17-kube-api-access-bxbkb\") pod \"console-85c6769964-xw6b7\" (UID: \"f12daa1a-924a-49f0-9000-b08c97a7ec17\") " pod="openshift-console/console-85c6769964-xw6b7"
Sep 18 20:00:43.809344 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:00:43.809182    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f12daa1a-924a-49f0-9000-b08c97a7ec17-oauth-serving-cert\") pod \"console-85c6769964-xw6b7\" (UID: \"f12daa1a-924a-49f0-9000-b08c97a7ec17\") " pod="openshift-console/console-85c6769964-xw6b7"
Sep 18 20:00:43.809344 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:00:43.809315    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f12daa1a-924a-49f0-9000-b08c97a7ec17-trusted-ca-bundle\") pod \"console-85c6769964-xw6b7\" (UID: \"f12daa1a-924a-49f0-9000-b08c97a7ec17\") " pod="openshift-console/console-85c6769964-xw6b7"
Sep 18 20:00:43.809524 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:00:43.809366    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f12daa1a-924a-49f0-9000-b08c97a7ec17-console-serving-cert\") pod \"console-85c6769964-xw6b7\" (UID: \"f12daa1a-924a-49f0-9000-b08c97a7ec17\") " pod="openshift-console/console-85c6769964-xw6b7"
Sep 18 20:00:43.809524 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:00:43.809437    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f12daa1a-924a-49f0-9000-b08c97a7ec17-console-oauth-config\") pod \"console-85c6769964-xw6b7\" (UID: \"f12daa1a-924a-49f0-9000-b08c97a7ec17\") " pod="openshift-console/console-85c6769964-xw6b7"
Sep 18 20:00:43.809900 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:00:43.809872    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f12daa1a-924a-49f0-9000-b08c97a7ec17-console-config\") pod \"console-85c6769964-xw6b7\" (UID: \"f12daa1a-924a-49f0-9000-b08c97a7ec17\") " pod="openshift-console/console-85c6769964-xw6b7"
Sep 18 20:00:43.809992 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:00:43.809905    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f12daa1a-924a-49f0-9000-b08c97a7ec17-service-ca\") pod \"console-85c6769964-xw6b7\" (UID: \"f12daa1a-924a-49f0-9000-b08c97a7ec17\") " pod="openshift-console/console-85c6769964-xw6b7"
Sep 18 20:00:43.809992 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:00:43.809916    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f12daa1a-924a-49f0-9000-b08c97a7ec17-oauth-serving-cert\") pod \"console-85c6769964-xw6b7\" (UID: \"f12daa1a-924a-49f0-9000-b08c97a7ec17\") " pod="openshift-console/console-85c6769964-xw6b7"
Sep 18 20:00:43.810143 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:00:43.810125    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f12daa1a-924a-49f0-9000-b08c97a7ec17-trusted-ca-bundle\") pod \"console-85c6769964-xw6b7\" (UID: \"f12daa1a-924a-49f0-9000-b08c97a7ec17\") " pod="openshift-console/console-85c6769964-xw6b7"
Sep 18 20:00:43.811649 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:00:43.811628    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f12daa1a-924a-49f0-9000-b08c97a7ec17-console-serving-cert\") pod \"console-85c6769964-xw6b7\" (UID: \"f12daa1a-924a-49f0-9000-b08c97a7ec17\") " pod="openshift-console/console-85c6769964-xw6b7"
Sep 18 20:00:43.811738 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:00:43.811632    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f12daa1a-924a-49f0-9000-b08c97a7ec17-console-oauth-config\") pod \"console-85c6769964-xw6b7\" (UID: \"f12daa1a-924a-49f0-9000-b08c97a7ec17\") " pod="openshift-console/console-85c6769964-xw6b7"
Sep 18 20:00:43.819505 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:00:43.819488    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxbkb\" (UniqueName: \"kubernetes.io/projected/f12daa1a-924a-49f0-9000-b08c97a7ec17-kube-api-access-bxbkb\") pod \"console-85c6769964-xw6b7\" (UID: \"f12daa1a-924a-49f0-9000-b08c97a7ec17\") " pod="openshift-console/console-85c6769964-xw6b7"
Sep 18 20:00:44.002591 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:00:44.002551    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85c6769964-xw6b7"
Sep 18 20:00:44.280707 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:00:44.280595    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85c6769964-xw6b7"]
Sep 18 20:00:44.288555 ip-10-0-140-178 kubenswrapper[2549]: W0918 20:00:44.287249    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf12daa1a_924a_49f0_9000_b08c97a7ec17.slice/crio-eb3d307fc9915415f3bcbe4d044f26ffd0901e4d9b8eaabde55588d654a955f2 WatchSource:0}: Error finding container eb3d307fc9915415f3bcbe4d044f26ffd0901e4d9b8eaabde55588d654a955f2: Status 404 returned error can't find the container with id eb3d307fc9915415f3bcbe4d044f26ffd0901e4d9b8eaabde55588d654a955f2
Sep 18 20:00:44.544674 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:00:44.544571    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85c6769964-xw6b7" event={"ID":"f12daa1a-924a-49f0-9000-b08c97a7ec17","Type":"ContainerStarted","Data":"e881d335cb19c7cb1bcb3adceed632c814431ccc92c28429cf38cc189a9b5ade"}
Sep 18 20:00:44.544674 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:00:44.544623    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85c6769964-xw6b7" event={"ID":"f12daa1a-924a-49f0-9000-b08c97a7ec17","Type":"ContainerStarted","Data":"eb3d307fc9915415f3bcbe4d044f26ffd0901e4d9b8eaabde55588d654a955f2"}
Sep 18 20:00:44.567811 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:00:44.567765    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-85c6769964-xw6b7" podStartSLOduration=1.567736944 podStartE2EDuration="1.567736944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:00:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:00:44.56626655 +0000 UTC m=+229.520007729" watchObservedRunningTime="2026-09-18 20:00:44.567736944 +0000 UTC m=+229.521478111"
Sep 18 20:00:54.002891 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:00:54.002836    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-85c6769964-xw6b7"
Sep 18 20:00:54.003420 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:00:54.002926    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-85c6769964-xw6b7"
Sep 18 20:00:54.007678 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:00:54.007653    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-85c6769964-xw6b7"
Sep 18 20:00:54.583936 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:00:54.583909    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-85c6769964-xw6b7"
Sep 18 20:00:54.697163 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:00:54.697134    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-98b85f47-vrbpg"]
Sep 18 20:01:19.715433 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:01:19.715323    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-98b85f47-vrbpg" podUID="9144c0d3-c3f8-4dda-9501-a4799332c6da" containerName="console" containerID="cri-o://b2832467d5988b587ed371b991410dea9a77ee1d8d593654fbf8c8ed408c7c72" gracePeriod=15
Sep 18 20:01:19.947083 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:01:19.947058    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-98b85f47-vrbpg"
Sep 18 20:01:20.069356 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:01:20.069278    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9144c0d3-c3f8-4dda-9501-a4799332c6da-console-config\") pod \"9144c0d3-c3f8-4dda-9501-a4799332c6da\" (UID: \"9144c0d3-c3f8-4dda-9501-a4799332c6da\") "
Sep 18 20:01:20.069356 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:01:20.069311    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zbrxn\" (UniqueName: \"kubernetes.io/projected/9144c0d3-c3f8-4dda-9501-a4799332c6da-kube-api-access-zbrxn\") pod \"9144c0d3-c3f8-4dda-9501-a4799332c6da\" (UID: \"9144c0d3-c3f8-4dda-9501-a4799332c6da\") "
Sep 18 20:01:20.069356 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:01:20.069332    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9144c0d3-c3f8-4dda-9501-a4799332c6da-oauth-serving-cert\") pod \"9144c0d3-c3f8-4dda-9501-a4799332c6da\" (UID: \"9144c0d3-c3f8-4dda-9501-a4799332c6da\") "
Sep 18 20:01:20.069356 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:01:20.069359    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9144c0d3-c3f8-4dda-9501-a4799332c6da-trusted-ca-bundle\") pod \"9144c0d3-c3f8-4dda-9501-a4799332c6da\" (UID: \"9144c0d3-c3f8-4dda-9501-a4799332c6da\") "
Sep 18 20:01:20.069696 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:01:20.069478    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9144c0d3-c3f8-4dda-9501-a4799332c6da-service-ca\") pod \"9144c0d3-c3f8-4dda-9501-a4799332c6da\" (UID: \"9144c0d3-c3f8-4dda-9501-a4799332c6da\") "
Sep 18 20:01:20.069696 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:01:20.069547    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9144c0d3-c3f8-4dda-9501-a4799332c6da-console-serving-cert\") pod \"9144c0d3-c3f8-4dda-9501-a4799332c6da\" (UID: \"9144c0d3-c3f8-4dda-9501-a4799332c6da\") "
Sep 18 20:01:20.069696 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:01:20.069600    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9144c0d3-c3f8-4dda-9501-a4799332c6da-console-oauth-config\") pod \"9144c0d3-c3f8-4dda-9501-a4799332c6da\" (UID: \"9144c0d3-c3f8-4dda-9501-a4799332c6da\") "
Sep 18 20:01:20.069864 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:01:20.069832    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9144c0d3-c3f8-4dda-9501-a4799332c6da-console-config" (OuterVolumeSpecName: "console-config") pod "9144c0d3-c3f8-4dda-9501-a4799332c6da" (UID: "9144c0d3-c3f8-4dda-9501-a4799332c6da"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:01:20.069864 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:01:20.069852    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9144c0d3-c3f8-4dda-9501-a4799332c6da-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "9144c0d3-c3f8-4dda-9501-a4799332c6da" (UID: "9144c0d3-c3f8-4dda-9501-a4799332c6da"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:01:20.069967 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:01:20.069844    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9144c0d3-c3f8-4dda-9501-a4799332c6da-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "9144c0d3-c3f8-4dda-9501-a4799332c6da" (UID: "9144c0d3-c3f8-4dda-9501-a4799332c6da"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:01:20.069967 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:01:20.069904    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9144c0d3-c3f8-4dda-9501-a4799332c6da-service-ca" (OuterVolumeSpecName: "service-ca") pod "9144c0d3-c3f8-4dda-9501-a4799332c6da" (UID: "9144c0d3-c3f8-4dda-9501-a4799332c6da"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:01:20.071661 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:01:20.071639    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9144c0d3-c3f8-4dda-9501-a4799332c6da-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "9144c0d3-c3f8-4dda-9501-a4799332c6da" (UID: "9144c0d3-c3f8-4dda-9501-a4799332c6da"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:01:20.071661 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:01:20.071641    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9144c0d3-c3f8-4dda-9501-a4799332c6da-kube-api-access-zbrxn" (OuterVolumeSpecName: "kube-api-access-zbrxn") pod "9144c0d3-c3f8-4dda-9501-a4799332c6da" (UID: "9144c0d3-c3f8-4dda-9501-a4799332c6da"). InnerVolumeSpecName "kube-api-access-zbrxn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:01:20.071797 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:01:20.071683    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9144c0d3-c3f8-4dda-9501-a4799332c6da-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "9144c0d3-c3f8-4dda-9501-a4799332c6da" (UID: "9144c0d3-c3f8-4dda-9501-a4799332c6da"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:01:20.170254 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:01:20.170231    2549 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9144c0d3-c3f8-4dda-9501-a4799332c6da-oauth-serving-cert\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:01:20.170254 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:01:20.170251    2549 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9144c0d3-c3f8-4dda-9501-a4799332c6da-trusted-ca-bundle\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:01:20.170415 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:01:20.170261    2549 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9144c0d3-c3f8-4dda-9501-a4799332c6da-service-ca\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:01:20.170415 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:01:20.170270    2549 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9144c0d3-c3f8-4dda-9501-a4799332c6da-console-serving-cert\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:01:20.170415 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:01:20.170279    2549 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9144c0d3-c3f8-4dda-9501-a4799332c6da-console-oauth-config\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:01:20.170415 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:01:20.170287    2549 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9144c0d3-c3f8-4dda-9501-a4799332c6da-console-config\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:01:20.170415 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:01:20.170295    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbrxn\" (UniqueName: \"kubernetes.io/projected/9144c0d3-c3f8-4dda-9501-a4799332c6da-kube-api-access-zbrxn\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:01:20.672993 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:01:20.672963    2549 generic.go:356] "Generic (PLEG): container finished" podID="9144c0d3-c3f8-4dda-9501-a4799332c6da" containerID="b2832467d5988b587ed371b991410dea9a77ee1d8d593654fbf8c8ed408c7c72" exitCode=0
Sep 18 20:01:20.673157 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:01:20.673005    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-98b85f47-vrbpg"
Sep 18 20:01:20.673157 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:01:20.673046    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-98b85f47-vrbpg" event={"ID":"9144c0d3-c3f8-4dda-9501-a4799332c6da","Type":"ContainerDied","Data":"b2832467d5988b587ed371b991410dea9a77ee1d8d593654fbf8c8ed408c7c72"}
Sep 18 20:01:20.673157 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:01:20.673098    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-98b85f47-vrbpg" event={"ID":"9144c0d3-c3f8-4dda-9501-a4799332c6da","Type":"ContainerDied","Data":"2f80db2b26e2583729bc5c556e54c9f0e64dcb0861be2c05743eaafc98f357b3"}
Sep 18 20:01:20.673157 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:01:20.673119    2549 scope.go:117] "RemoveContainer" containerID="b2832467d5988b587ed371b991410dea9a77ee1d8d593654fbf8c8ed408c7c72"
Sep 18 20:01:20.681361 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:01:20.681347    2549 scope.go:117] "RemoveContainer" containerID="b2832467d5988b587ed371b991410dea9a77ee1d8d593654fbf8c8ed408c7c72"
Sep 18 20:01:20.681641 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:01:20.681626    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2832467d5988b587ed371b991410dea9a77ee1d8d593654fbf8c8ed408c7c72\": container with ID starting with b2832467d5988b587ed371b991410dea9a77ee1d8d593654fbf8c8ed408c7c72 not found: ID does not exist" containerID="b2832467d5988b587ed371b991410dea9a77ee1d8d593654fbf8c8ed408c7c72"
Sep 18 20:01:20.681691 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:01:20.681648    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2832467d5988b587ed371b991410dea9a77ee1d8d593654fbf8c8ed408c7c72"} err="failed to get container status \"b2832467d5988b587ed371b991410dea9a77ee1d8d593654fbf8c8ed408c7c72\": rpc error: code = NotFound desc = could not find container \"b2832467d5988b587ed371b991410dea9a77ee1d8d593654fbf8c8ed408c7c72\": container with ID starting with b2832467d5988b587ed371b991410dea9a77ee1d8d593654fbf8c8ed408c7c72 not found: ID does not exist"
Sep 18 20:01:20.695299 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:01:20.695266    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-98b85f47-vrbpg"]
Sep 18 20:01:20.699590 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:01:20.699570    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-98b85f47-vrbpg"]
Sep 18 20:01:21.520312 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:01:21.520275    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9144c0d3-c3f8-4dda-9501-a4799332c6da" path="/var/lib/kubelet/pods/9144c0d3-c3f8-4dda-9501-a4799332c6da/volumes"
Sep 18 20:01:55.397841 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:01:55.397804    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-scn7p_a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48/ovn-acl-logging/0.log"
Sep 18 20:01:55.401467 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:01:55.401202    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-scn7p_a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48/ovn-acl-logging/0.log"
Sep 18 20:01:55.403976 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:01:55.403945    2549 kubelet.go:1648] "Image garbage collection succeeded"
Sep 18 20:01:58.509120 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:01:58.509083    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6c4b8dd4d8-mtlrh"]
Sep 18 20:01:58.511372 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:01:58.509356    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9144c0d3-c3f8-4dda-9501-a4799332c6da" containerName="console"
Sep 18 20:01:58.511372 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:01:58.509366    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="9144c0d3-c3f8-4dda-9501-a4799332c6da" containerName="console"
Sep 18 20:01:58.511372 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:01:58.509448    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="9144c0d3-c3f8-4dda-9501-a4799332c6da" containerName="console"
Sep 18 20:01:58.512219 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:01:58.512204    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c4b8dd4d8-mtlrh"
Sep 18 20:01:58.535011 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:01:58.534984    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c4b8dd4d8-mtlrh"]
Sep 18 20:01:58.656670 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:01:58.656642    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4e6a9e26-e76d-44c3-bb37-6258c7bf8907-console-oauth-config\") pod \"console-6c4b8dd4d8-mtlrh\" (UID: \"4e6a9e26-e76d-44c3-bb37-6258c7bf8907\") " pod="openshift-console/console-6c4b8dd4d8-mtlrh"
Sep 18 20:01:58.656670 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:01:58.656670    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e6a9e26-e76d-44c3-bb37-6258c7bf8907-trusted-ca-bundle\") pod \"console-6c4b8dd4d8-mtlrh\" (UID: \"4e6a9e26-e76d-44c3-bb37-6258c7bf8907\") " pod="openshift-console/console-6c4b8dd4d8-mtlrh"
Sep 18 20:01:58.656859 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:01:58.656694    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4e6a9e26-e76d-44c3-bb37-6258c7bf8907-oauth-serving-cert\") pod \"console-6c4b8dd4d8-mtlrh\" (UID: \"4e6a9e26-e76d-44c3-bb37-6258c7bf8907\") " pod="openshift-console/console-6c4b8dd4d8-mtlrh"
Sep 18 20:01:58.656859 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:01:58.656723    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4e6a9e26-e76d-44c3-bb37-6258c7bf8907-console-config\") pod \"console-6c4b8dd4d8-mtlrh\" (UID: \"4e6a9e26-e76d-44c3-bb37-6258c7bf8907\") " pod="openshift-console/console-6c4b8dd4d8-mtlrh"
Sep 18 20:01:58.656859 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:01:58.656836    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4e6a9e26-e76d-44c3-bb37-6258c7bf8907-service-ca\") pod \"console-6c4b8dd4d8-mtlrh\" (UID: \"4e6a9e26-e76d-44c3-bb37-6258c7bf8907\") " pod="openshift-console/console-6c4b8dd4d8-mtlrh"
Sep 18 20:01:58.657006 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:01:58.656897    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfjfq\" (UniqueName: \"kubernetes.io/projected/4e6a9e26-e76d-44c3-bb37-6258c7bf8907-kube-api-access-xfjfq\") pod \"console-6c4b8dd4d8-mtlrh\" (UID: \"4e6a9e26-e76d-44c3-bb37-6258c7bf8907\") " pod="openshift-console/console-6c4b8dd4d8-mtlrh"
Sep 18 20:01:58.657006 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:01:58.656930    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4e6a9e26-e76d-44c3-bb37-6258c7bf8907-console-serving-cert\") pod \"console-6c4b8dd4d8-mtlrh\" (UID: \"4e6a9e26-e76d-44c3-bb37-6258c7bf8907\") " pod="openshift-console/console-6c4b8dd4d8-mtlrh"
Sep 18 20:01:58.757515 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:01:58.757483    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4e6a9e26-e76d-44c3-bb37-6258c7bf8907-oauth-serving-cert\") pod \"console-6c4b8dd4d8-mtlrh\" (UID: \"4e6a9e26-e76d-44c3-bb37-6258c7bf8907\") " pod="openshift-console/console-6c4b8dd4d8-mtlrh"
Sep 18 20:01:58.757651 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:01:58.757523    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4e6a9e26-e76d-44c3-bb37-6258c7bf8907-console-config\") pod \"console-6c4b8dd4d8-mtlrh\" (UID: \"4e6a9e26-e76d-44c3-bb37-6258c7bf8907\") " pod="openshift-console/console-6c4b8dd4d8-mtlrh"
Sep 18 20:01:58.757651 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:01:58.757550    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4e6a9e26-e76d-44c3-bb37-6258c7bf8907-service-ca\") pod \"console-6c4b8dd4d8-mtlrh\" (UID: \"4e6a9e26-e76d-44c3-bb37-6258c7bf8907\") " pod="openshift-console/console-6c4b8dd4d8-mtlrh"
Sep 18 20:01:58.757651 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:01:58.757584    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfjfq\" (UniqueName: \"kubernetes.io/projected/4e6a9e26-e76d-44c3-bb37-6258c7bf8907-kube-api-access-xfjfq\") pod \"console-6c4b8dd4d8-mtlrh\" (UID: \"4e6a9e26-e76d-44c3-bb37-6258c7bf8907\") " pod="openshift-console/console-6c4b8dd4d8-mtlrh"
Sep 18 20:01:58.757651 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:01:58.757609    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4e6a9e26-e76d-44c3-bb37-6258c7bf8907-console-serving-cert\") pod \"console-6c4b8dd4d8-mtlrh\" (UID: \"4e6a9e26-e76d-44c3-bb37-6258c7bf8907\") " pod="openshift-console/console-6c4b8dd4d8-mtlrh"
Sep 18 20:01:58.757651 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:01:58.757635    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4e6a9e26-e76d-44c3-bb37-6258c7bf8907-console-oauth-config\") pod \"console-6c4b8dd4d8-mtlrh\" (UID: \"4e6a9e26-e76d-44c3-bb37-6258c7bf8907\") " pod="openshift-console/console-6c4b8dd4d8-mtlrh"
Sep 18 20:01:58.757909 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:01:58.757657    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e6a9e26-e76d-44c3-bb37-6258c7bf8907-trusted-ca-bundle\") pod \"console-6c4b8dd4d8-mtlrh\" (UID: \"4e6a9e26-e76d-44c3-bb37-6258c7bf8907\") " pod="openshift-console/console-6c4b8dd4d8-mtlrh"
Sep 18 20:01:58.758295 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:01:58.758269    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4e6a9e26-e76d-44c3-bb37-6258c7bf8907-oauth-serving-cert\") pod \"console-6c4b8dd4d8-mtlrh\" (UID: \"4e6a9e26-e76d-44c3-bb37-6258c7bf8907\") " pod="openshift-console/console-6c4b8dd4d8-mtlrh"
Sep 18 20:01:58.758374 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:01:58.758315    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4e6a9e26-e76d-44c3-bb37-6258c7bf8907-service-ca\") pod \"console-6c4b8dd4d8-mtlrh\" (UID: \"4e6a9e26-e76d-44c3-bb37-6258c7bf8907\") " pod="openshift-console/console-6c4b8dd4d8-mtlrh"
Sep 18 20:01:58.758755 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:01:58.758730    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4e6a9e26-e76d-44c3-bb37-6258c7bf8907-console-config\") pod \"console-6c4b8dd4d8-mtlrh\" (UID: \"4e6a9e26-e76d-44c3-bb37-6258c7bf8907\") " pod="openshift-console/console-6c4b8dd4d8-mtlrh"
Sep 18 20:01:58.758929 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:01:58.758913    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e6a9e26-e76d-44c3-bb37-6258c7bf8907-trusted-ca-bundle\") pod \"console-6c4b8dd4d8-mtlrh\" (UID: \"4e6a9e26-e76d-44c3-bb37-6258c7bf8907\") " pod="openshift-console/console-6c4b8dd4d8-mtlrh"
Sep 18 20:01:58.760094 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:01:58.760033    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4e6a9e26-e76d-44c3-bb37-6258c7bf8907-console-serving-cert\") pod \"console-6c4b8dd4d8-mtlrh\" (UID: \"4e6a9e26-e76d-44c3-bb37-6258c7bf8907\") " pod="openshift-console/console-6c4b8dd4d8-mtlrh"
Sep 18 20:01:58.760224 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:01:58.760202    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4e6a9e26-e76d-44c3-bb37-6258c7bf8907-console-oauth-config\") pod \"console-6c4b8dd4d8-mtlrh\" (UID: \"4e6a9e26-e76d-44c3-bb37-6258c7bf8907\") " pod="openshift-console/console-6c4b8dd4d8-mtlrh"
Sep 18 20:01:58.767463 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:01:58.767440    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfjfq\" (UniqueName: \"kubernetes.io/projected/4e6a9e26-e76d-44c3-bb37-6258c7bf8907-kube-api-access-xfjfq\") pod \"console-6c4b8dd4d8-mtlrh\" (UID: \"4e6a9e26-e76d-44c3-bb37-6258c7bf8907\") " pod="openshift-console/console-6c4b8dd4d8-mtlrh"
Sep 18 20:01:58.820859 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:01:58.820832    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c4b8dd4d8-mtlrh"
Sep 18 20:01:59.098051 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:01:59.098019    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c4b8dd4d8-mtlrh"]
Sep 18 20:01:59.101321 ip-10-0-140-178 kubenswrapper[2549]: W0918 20:01:59.101296    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e6a9e26_e76d_44c3_bb37_6258c7bf8907.slice/crio-d1a054dbe430e8280c015a710b186bd2adf8acb7fb0a25843f9c9a63fca07276 WatchSource:0}: Error finding container d1a054dbe430e8280c015a710b186bd2adf8acb7fb0a25843f9c9a63fca07276: Status 404 returned error can't find the container with id d1a054dbe430e8280c015a710b186bd2adf8acb7fb0a25843f9c9a63fca07276
Sep 18 20:01:59.826319 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:01:59.826282    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c4b8dd4d8-mtlrh" event={"ID":"4e6a9e26-e76d-44c3-bb37-6258c7bf8907","Type":"ContainerStarted","Data":"bd18a810f90dbd8d1729ed3e6850f236d4a4dbf6aa3126787e2411325a5e6a78"}
Sep 18 20:01:59.826792 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:01:59.826329    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c4b8dd4d8-mtlrh" event={"ID":"4e6a9e26-e76d-44c3-bb37-6258c7bf8907","Type":"ContainerStarted","Data":"d1a054dbe430e8280c015a710b186bd2adf8acb7fb0a25843f9c9a63fca07276"}
Sep 18 20:01:59.857977 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:01:59.857931    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6c4b8dd4d8-mtlrh" podStartSLOduration=1.857915224 podStartE2EDuration="1.857915224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:01:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:01:59.856066141 +0000 UTC m=+304.809807320" watchObservedRunningTime="2026-09-18 20:01:59.857915224 +0000 UTC m=+304.811656451"
Sep 18 20:02:08.821495 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:02:08.821455    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6c4b8dd4d8-mtlrh"
Sep 18 20:02:08.821952 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:02:08.821529    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6c4b8dd4d8-mtlrh"
Sep 18 20:02:08.826055 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:02:08.826031    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6c4b8dd4d8-mtlrh"
Sep 18 20:02:08.863978 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:02:08.863956    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6c4b8dd4d8-mtlrh"
Sep 18 20:02:08.921842 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:02:08.921801    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-85c6769964-xw6b7"]
Sep 18 20:02:33.942598 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:02:33.942474    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-85c6769964-xw6b7" podUID="f12daa1a-924a-49f0-9000-b08c97a7ec17" containerName="console" containerID="cri-o://e881d335cb19c7cb1bcb3adceed632c814431ccc92c28429cf38cc189a9b5ade" gracePeriod=15
Sep 18 20:02:34.186713 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:02:34.186691    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85c6769964-xw6b7"
Sep 18 20:02:34.318509 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:02:34.318422    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f12daa1a-924a-49f0-9000-b08c97a7ec17-trusted-ca-bundle\") pod \"f12daa1a-924a-49f0-9000-b08c97a7ec17\" (UID: \"f12daa1a-924a-49f0-9000-b08c97a7ec17\") "
Sep 18 20:02:34.318509 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:02:34.318482    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f12daa1a-924a-49f0-9000-b08c97a7ec17-console-config\") pod \"f12daa1a-924a-49f0-9000-b08c97a7ec17\" (UID: \"f12daa1a-924a-49f0-9000-b08c97a7ec17\") "
Sep 18 20:02:34.318509 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:02:34.318503    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f12daa1a-924a-49f0-9000-b08c97a7ec17-console-oauth-config\") pod \"f12daa1a-924a-49f0-9000-b08c97a7ec17\" (UID: \"f12daa1a-924a-49f0-9000-b08c97a7ec17\") "
Sep 18 20:02:34.318766 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:02:34.318533    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f12daa1a-924a-49f0-9000-b08c97a7ec17-oauth-serving-cert\") pod \"f12daa1a-924a-49f0-9000-b08c97a7ec17\" (UID: \"f12daa1a-924a-49f0-9000-b08c97a7ec17\") "
Sep 18 20:02:34.318766 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:02:34.318561    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bxbkb\" (UniqueName: \"kubernetes.io/projected/f12daa1a-924a-49f0-9000-b08c97a7ec17-kube-api-access-bxbkb\") pod \"f12daa1a-924a-49f0-9000-b08c97a7ec17\" (UID: \"f12daa1a-924a-49f0-9000-b08c97a7ec17\") "
Sep 18 20:02:34.318766 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:02:34.318579    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f12daa1a-924a-49f0-9000-b08c97a7ec17-console-serving-cert\") pod \"f12daa1a-924a-49f0-9000-b08c97a7ec17\" (UID: \"f12daa1a-924a-49f0-9000-b08c97a7ec17\") "
Sep 18 20:02:34.318766 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:02:34.318595    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f12daa1a-924a-49f0-9000-b08c97a7ec17-service-ca\") pod \"f12daa1a-924a-49f0-9000-b08c97a7ec17\" (UID: \"f12daa1a-924a-49f0-9000-b08c97a7ec17\") "
Sep 18 20:02:34.319024 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:02:34.318947    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f12daa1a-924a-49f0-9000-b08c97a7ec17-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "f12daa1a-924a-49f0-9000-b08c97a7ec17" (UID: "f12daa1a-924a-49f0-9000-b08c97a7ec17"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:02:34.319024 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:02:34.318978    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f12daa1a-924a-49f0-9000-b08c97a7ec17-console-config" (OuterVolumeSpecName: "console-config") pod "f12daa1a-924a-49f0-9000-b08c97a7ec17" (UID: "f12daa1a-924a-49f0-9000-b08c97a7ec17"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:02:34.319024 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:02:34.318986    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f12daa1a-924a-49f0-9000-b08c97a7ec17-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "f12daa1a-924a-49f0-9000-b08c97a7ec17" (UID: "f12daa1a-924a-49f0-9000-b08c97a7ec17"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:02:34.319296 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:02:34.319256    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f12daa1a-924a-49f0-9000-b08c97a7ec17-service-ca" (OuterVolumeSpecName: "service-ca") pod "f12daa1a-924a-49f0-9000-b08c97a7ec17" (UID: "f12daa1a-924a-49f0-9000-b08c97a7ec17"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:02:34.320685 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:02:34.320658    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f12daa1a-924a-49f0-9000-b08c97a7ec17-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "f12daa1a-924a-49f0-9000-b08c97a7ec17" (UID: "f12daa1a-924a-49f0-9000-b08c97a7ec17"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:02:34.321095 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:02:34.321076    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f12daa1a-924a-49f0-9000-b08c97a7ec17-kube-api-access-bxbkb" (OuterVolumeSpecName: "kube-api-access-bxbkb") pod "f12daa1a-924a-49f0-9000-b08c97a7ec17" (UID: "f12daa1a-924a-49f0-9000-b08c97a7ec17"). InnerVolumeSpecName "kube-api-access-bxbkb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:02:34.321095 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:02:34.321082    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f12daa1a-924a-49f0-9000-b08c97a7ec17-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "f12daa1a-924a-49f0-9000-b08c97a7ec17" (UID: "f12daa1a-924a-49f0-9000-b08c97a7ec17"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:02:34.419880 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:02:34.419836    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxbkb\" (UniqueName: \"kubernetes.io/projected/f12daa1a-924a-49f0-9000-b08c97a7ec17-kube-api-access-bxbkb\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:02:34.419880 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:02:34.419875    2549 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f12daa1a-924a-49f0-9000-b08c97a7ec17-console-serving-cert\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:02:34.419880 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:02:34.419884    2549 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f12daa1a-924a-49f0-9000-b08c97a7ec17-service-ca\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:02:34.419880 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:02:34.419894    2549 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f12daa1a-924a-49f0-9000-b08c97a7ec17-trusted-ca-bundle\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:02:34.420138 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:02:34.419903    2549 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f12daa1a-924a-49f0-9000-b08c97a7ec17-console-config\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:02:34.420138 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:02:34.419911    2549 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f12daa1a-924a-49f0-9000-b08c97a7ec17-console-oauth-config\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:02:34.420138 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:02:34.419919    2549 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f12daa1a-924a-49f0-9000-b08c97a7ec17-oauth-serving-cert\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:02:34.958079 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:02:34.958049    2549 generic.go:356] "Generic (PLEG): container finished" podID="f12daa1a-924a-49f0-9000-b08c97a7ec17" containerID="e881d335cb19c7cb1bcb3adceed632c814431ccc92c28429cf38cc189a9b5ade" exitCode=0
Sep 18 20:02:34.958505 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:02:34.958086    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85c6769964-xw6b7" event={"ID":"f12daa1a-924a-49f0-9000-b08c97a7ec17","Type":"ContainerDied","Data":"e881d335cb19c7cb1bcb3adceed632c814431ccc92c28429cf38cc189a9b5ade"}
Sep 18 20:02:34.958505 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:02:34.958092    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85c6769964-xw6b7"
Sep 18 20:02:34.958505 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:02:34.958122    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85c6769964-xw6b7" event={"ID":"f12daa1a-924a-49f0-9000-b08c97a7ec17","Type":"ContainerDied","Data":"eb3d307fc9915415f3bcbe4d044f26ffd0901e4d9b8eaabde55588d654a955f2"}
Sep 18 20:02:34.958505 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:02:34.958142    2549 scope.go:117] "RemoveContainer" containerID="e881d335cb19c7cb1bcb3adceed632c814431ccc92c28429cf38cc189a9b5ade"
Sep 18 20:02:34.966548 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:02:34.966528    2549 scope.go:117] "RemoveContainer" containerID="e881d335cb19c7cb1bcb3adceed632c814431ccc92c28429cf38cc189a9b5ade"
Sep 18 20:02:34.966818 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:02:34.966798    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e881d335cb19c7cb1bcb3adceed632c814431ccc92c28429cf38cc189a9b5ade\": container with ID starting with e881d335cb19c7cb1bcb3adceed632c814431ccc92c28429cf38cc189a9b5ade not found: ID does not exist" containerID="e881d335cb19c7cb1bcb3adceed632c814431ccc92c28429cf38cc189a9b5ade"
Sep 18 20:02:34.966872 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:02:34.966826    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e881d335cb19c7cb1bcb3adceed632c814431ccc92c28429cf38cc189a9b5ade"} err="failed to get container status \"e881d335cb19c7cb1bcb3adceed632c814431ccc92c28429cf38cc189a9b5ade\": rpc error: code = NotFound desc = could not find container \"e881d335cb19c7cb1bcb3adceed632c814431ccc92c28429cf38cc189a9b5ade\": container with ID starting with e881d335cb19c7cb1bcb3adceed632c814431ccc92c28429cf38cc189a9b5ade not found: ID does not exist"
Sep 18 20:02:34.980131 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:02:34.980109    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-85c6769964-xw6b7"]
Sep 18 20:02:34.984156 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:02:34.984136    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-85c6769964-xw6b7"]
Sep 18 20:02:35.519797 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:02:35.519769    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f12daa1a-924a-49f0-9000-b08c97a7ec17" path="/var/lib/kubelet/pods/f12daa1a-924a-49f0-9000-b08c97a7ec17/volumes"
Sep 18 20:02:51.115124 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:02:51.115090    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wzbg2"]
Sep 18 20:02:51.115569 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:02:51.115363    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f12daa1a-924a-49f0-9000-b08c97a7ec17" containerName="console"
Sep 18 20:02:51.115569 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:02:51.115373    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="f12daa1a-924a-49f0-9000-b08c97a7ec17" containerName="console"
Sep 18 20:02:51.115569 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:02:51.115447    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="f12daa1a-924a-49f0-9000-b08c97a7ec17" containerName="console"
Sep 18 20:02:51.118252 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:02:51.118235    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wzbg2"
Sep 18 20:02:51.121274 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:02:51.121250    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 18 20:02:51.121429 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:02:51.121344    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 18 20:02:51.121429 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:02:51.121403    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-zjfq9\""
Sep 18 20:02:51.128348 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:02:51.128325    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wzbg2"]
Sep 18 20:02:51.248761 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:02:51.248731    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f20e6aa4-14c2-4931-bcd7-5bb26c849eca-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wzbg2\" (UID: \"f20e6aa4-14c2-4931-bcd7-5bb26c849eca\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wzbg2"
Sep 18 20:02:51.248939 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:02:51.248793    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f20e6aa4-14c2-4931-bcd7-5bb26c849eca-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wzbg2\" (UID: \"f20e6aa4-14c2-4931-bcd7-5bb26c849eca\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wzbg2"
Sep 18 20:02:51.248939 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:02:51.248866    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhwlp\" (UniqueName: \"kubernetes.io/projected/f20e6aa4-14c2-4931-bcd7-5bb26c849eca-kube-api-access-xhwlp\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wzbg2\" (UID: \"f20e6aa4-14c2-4931-bcd7-5bb26c849eca\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wzbg2"
Sep 18 20:02:51.350070 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:02:51.350043    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f20e6aa4-14c2-4931-bcd7-5bb26c849eca-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wzbg2\" (UID: \"f20e6aa4-14c2-4931-bcd7-5bb26c849eca\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wzbg2"
Sep 18 20:02:51.350158 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:02:51.350082    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhwlp\" (UniqueName: \"kubernetes.io/projected/f20e6aa4-14c2-4931-bcd7-5bb26c849eca-kube-api-access-xhwlp\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wzbg2\" (UID: \"f20e6aa4-14c2-4931-bcd7-5bb26c849eca\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wzbg2"
Sep 18 20:02:51.350158 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:02:51.350108    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f20e6aa4-14c2-4931-bcd7-5bb26c849eca-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wzbg2\" (UID: \"f20e6aa4-14c2-4931-bcd7-5bb26c849eca\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wzbg2"
Sep 18 20:02:51.350533 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:02:51.350506    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f20e6aa4-14c2-4931-bcd7-5bb26c849eca-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wzbg2\" (UID: \"f20e6aa4-14c2-4931-bcd7-5bb26c849eca\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wzbg2"
Sep 18 20:02:51.350614 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:02:51.350539    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f20e6aa4-14c2-4931-bcd7-5bb26c849eca-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wzbg2\" (UID: \"f20e6aa4-14c2-4931-bcd7-5bb26c849eca\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wzbg2"
Sep 18 20:02:51.359437 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:02:51.359413    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhwlp\" (UniqueName: \"kubernetes.io/projected/f20e6aa4-14c2-4931-bcd7-5bb26c849eca-kube-api-access-xhwlp\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wzbg2\" (UID: \"f20e6aa4-14c2-4931-bcd7-5bb26c849eca\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wzbg2"
Sep 18 20:02:51.427430 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:02:51.427356    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wzbg2"
Sep 18 20:02:51.698746 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:02:51.698721    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wzbg2"]
Sep 18 20:02:51.700599 ip-10-0-140-178 kubenswrapper[2549]: W0918 20:02:51.700564    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf20e6aa4_14c2_4931_bcd7_5bb26c849eca.slice/crio-9741ca322292dfd68820a99fc8afaf51ba40ba13227dc9ec18b7eac49bb30801 WatchSource:0}: Error finding container 9741ca322292dfd68820a99fc8afaf51ba40ba13227dc9ec18b7eac49bb30801: Status 404 returned error can't find the container with id 9741ca322292dfd68820a99fc8afaf51ba40ba13227dc9ec18b7eac49bb30801
Sep 18 20:02:51.702712 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:02:51.702683    2549 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 20:02:52.029605 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:02:52.029523    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wzbg2" event={"ID":"f20e6aa4-14c2-4931-bcd7-5bb26c849eca","Type":"ContainerStarted","Data":"9741ca322292dfd68820a99fc8afaf51ba40ba13227dc9ec18b7eac49bb30801"}
Sep 18 20:02:58.055020 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:02:58.054987    2549 generic.go:356] "Generic (PLEG): container finished" podID="f20e6aa4-14c2-4931-bcd7-5bb26c849eca" containerID="0570bace6b768548b269fca391992e40387cd279376e260f835b14c3e3acf2f6" exitCode=0
Sep 18 20:02:58.055415 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:02:58.055054    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wzbg2" event={"ID":"f20e6aa4-14c2-4931-bcd7-5bb26c849eca","Type":"ContainerDied","Data":"0570bace6b768548b269fca391992e40387cd279376e260f835b14c3e3acf2f6"}
Sep 18 20:03:01.070505 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:01.070464    2549 generic.go:356] "Generic (PLEG): container finished" podID="f20e6aa4-14c2-4931-bcd7-5bb26c849eca" containerID="adc981fe5d7fe2831a75fd9452a25b1ed00eaa7e9a1b6089c54d7484c4a1f9e6" exitCode=0
Sep 18 20:03:01.070862 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:01.070516    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wzbg2" event={"ID":"f20e6aa4-14c2-4931-bcd7-5bb26c849eca","Type":"ContainerDied","Data":"adc981fe5d7fe2831a75fd9452a25b1ed00eaa7e9a1b6089c54d7484c4a1f9e6"}
Sep 18 20:03:08.102819 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:08.102786    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wzbg2" event={"ID":"f20e6aa4-14c2-4931-bcd7-5bb26c849eca","Type":"ContainerStarted","Data":"540d438929496b6eaf8ca82ff9e450e474e90091516b38fafa2612502b83c15f"}
Sep 18 20:03:08.124253 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:08.124204    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wzbg2" podStartSLOduration=1.9502210770000001 podStartE2EDuration="17.124188685s" totalImagesPullingTime="15.173967608s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:02:51 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:03:08.123104582 +0000 UTC m=+373.076845749" watchObservedRunningTime="2026-09-18 20:03:08.124188685 +0000 UTC m=+373.077929866"
Sep 18 20:03:09.108804 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:09.108769    2549 generic.go:356] "Generic (PLEG): container finished" podID="f20e6aa4-14c2-4931-bcd7-5bb26c849eca" containerID="540d438929496b6eaf8ca82ff9e450e474e90091516b38fafa2612502b83c15f" exitCode=0
Sep 18 20:03:09.109292 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:09.108820    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wzbg2" event={"ID":"f20e6aa4-14c2-4931-bcd7-5bb26c849eca","Type":"ContainerDied","Data":"540d438929496b6eaf8ca82ff9e450e474e90091516b38fafa2612502b83c15f"}
Sep 18 20:03:10.233805 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:10.233782    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wzbg2"
Sep 18 20:03:10.301467 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:10.301435    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f20e6aa4-14c2-4931-bcd7-5bb26c849eca-bundle\") pod \"f20e6aa4-14c2-4931-bcd7-5bb26c849eca\" (UID: \"f20e6aa4-14c2-4931-bcd7-5bb26c849eca\") "
Sep 18 20:03:10.301605 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:10.301493    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xhwlp\" (UniqueName: \"kubernetes.io/projected/f20e6aa4-14c2-4931-bcd7-5bb26c849eca-kube-api-access-xhwlp\") pod \"f20e6aa4-14c2-4931-bcd7-5bb26c849eca\" (UID: \"f20e6aa4-14c2-4931-bcd7-5bb26c849eca\") "
Sep 18 20:03:10.301605 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:10.301518    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f20e6aa4-14c2-4931-bcd7-5bb26c849eca-util\") pod \"f20e6aa4-14c2-4931-bcd7-5bb26c849eca\" (UID: \"f20e6aa4-14c2-4931-bcd7-5bb26c849eca\") "
Sep 18 20:03:10.302120 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:10.302095    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f20e6aa4-14c2-4931-bcd7-5bb26c849eca-bundle" (OuterVolumeSpecName: "bundle") pod "f20e6aa4-14c2-4931-bcd7-5bb26c849eca" (UID: "f20e6aa4-14c2-4931-bcd7-5bb26c849eca"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:03:10.303635 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:10.303608    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f20e6aa4-14c2-4931-bcd7-5bb26c849eca-kube-api-access-xhwlp" (OuterVolumeSpecName: "kube-api-access-xhwlp") pod "f20e6aa4-14c2-4931-bcd7-5bb26c849eca" (UID: "f20e6aa4-14c2-4931-bcd7-5bb26c849eca"). InnerVolumeSpecName "kube-api-access-xhwlp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:03:10.305703 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:10.305672    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f20e6aa4-14c2-4931-bcd7-5bb26c849eca-util" (OuterVolumeSpecName: "util") pod "f20e6aa4-14c2-4931-bcd7-5bb26c849eca" (UID: "f20e6aa4-14c2-4931-bcd7-5bb26c849eca"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:03:10.402694 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:10.402639    2549 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f20e6aa4-14c2-4931-bcd7-5bb26c849eca-bundle\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:03:10.402694 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:10.402662    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhwlp\" (UniqueName: \"kubernetes.io/projected/f20e6aa4-14c2-4931-bcd7-5bb26c849eca-kube-api-access-xhwlp\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:03:10.402694 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:10.402672    2549 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f20e6aa4-14c2-4931-bcd7-5bb26c849eca-util\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:03:11.118448 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:11.118416    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wzbg2" event={"ID":"f20e6aa4-14c2-4931-bcd7-5bb26c849eca","Type":"ContainerDied","Data":"9741ca322292dfd68820a99fc8afaf51ba40ba13227dc9ec18b7eac49bb30801"}
Sep 18 20:03:11.118448 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:11.118431    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wzbg2"
Sep 18 20:03:11.118646 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:11.118460    2549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9741ca322292dfd68820a99fc8afaf51ba40ba13227dc9ec18b7eac49bb30801"
Sep 18 20:03:14.510614 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:14.510584    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-76vkx"]
Sep 18 20:03:14.510981 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:14.510932    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f20e6aa4-14c2-4931-bcd7-5bb26c849eca" containerName="extract"
Sep 18 20:03:14.510981 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:14.510944    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="f20e6aa4-14c2-4931-bcd7-5bb26c849eca" containerName="extract"
Sep 18 20:03:14.510981 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:14.510964    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f20e6aa4-14c2-4931-bcd7-5bb26c849eca" containerName="util"
Sep 18 20:03:14.510981 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:14.510972    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="f20e6aa4-14c2-4931-bcd7-5bb26c849eca" containerName="util"
Sep 18 20:03:14.510981 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:14.510982    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f20e6aa4-14c2-4931-bcd7-5bb26c849eca" containerName="pull"
Sep 18 20:03:14.511136 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:14.510988    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="f20e6aa4-14c2-4931-bcd7-5bb26c849eca" containerName="pull"
Sep 18 20:03:14.511136 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:14.511037    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="f20e6aa4-14c2-4931-bcd7-5bb26c849eca" containerName="extract"
Sep 18 20:03:14.514827 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:14.514808    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-76vkx"
Sep 18 20:03:14.518740 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:14.518712    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 18 20:03:14.518859 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:14.518772    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-fq96m\""
Sep 18 20:03:14.518859 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:14.518833    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 18 20:03:14.531665 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:14.531642    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/83636bf0-4690-4e91-b328-3db9fe3748cf-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-76vkx\" (UID: \"83636bf0-4690-4e91-b328-3db9fe3748cf\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-76vkx"
Sep 18 20:03:14.531782 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:14.531671    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8rt8\" (UniqueName: \"kubernetes.io/projected/83636bf0-4690-4e91-b328-3db9fe3748cf-kube-api-access-r8rt8\") pod \"cert-manager-operator-controller-manager-55c444bf86-76vkx\" (UID: \"83636bf0-4690-4e91-b328-3db9fe3748cf\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-76vkx"
Sep 18 20:03:14.533406 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:14.533373    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-76vkx"]
Sep 18 20:03:14.632879 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:14.632851    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/83636bf0-4690-4e91-b328-3db9fe3748cf-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-76vkx\" (UID: \"83636bf0-4690-4e91-b328-3db9fe3748cf\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-76vkx"
Sep 18 20:03:14.633001 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:14.632882    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8rt8\" (UniqueName: \"kubernetes.io/projected/83636bf0-4690-4e91-b328-3db9fe3748cf-kube-api-access-r8rt8\") pod \"cert-manager-operator-controller-manager-55c444bf86-76vkx\" (UID: \"83636bf0-4690-4e91-b328-3db9fe3748cf\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-76vkx"
Sep 18 20:03:14.633268 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:14.633253    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/83636bf0-4690-4e91-b328-3db9fe3748cf-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-76vkx\" (UID: \"83636bf0-4690-4e91-b328-3db9fe3748cf\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-76vkx"
Sep 18 20:03:14.643125 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:14.643104    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8rt8\" (UniqueName: \"kubernetes.io/projected/83636bf0-4690-4e91-b328-3db9fe3748cf-kube-api-access-r8rt8\") pod \"cert-manager-operator-controller-manager-55c444bf86-76vkx\" (UID: \"83636bf0-4690-4e91-b328-3db9fe3748cf\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-76vkx"
Sep 18 20:03:14.824321 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:14.824294    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-76vkx"
Sep 18 20:03:15.108412 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:15.108365    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-76vkx"]
Sep 18 20:03:15.110314 ip-10-0-140-178 kubenswrapper[2549]: W0918 20:03:15.110288    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83636bf0_4690_4e91_b328_3db9fe3748cf.slice/crio-c0eea2b22aa3b529b2d22d821d3b32b646150119a189fdd62d89374c5855cb66 WatchSource:0}: Error finding container c0eea2b22aa3b529b2d22d821d3b32b646150119a189fdd62d89374c5855cb66: Status 404 returned error can't find the container with id c0eea2b22aa3b529b2d22d821d3b32b646150119a189fdd62d89374c5855cb66
Sep 18 20:03:15.135515 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:15.135491    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-76vkx" event={"ID":"83636bf0-4690-4e91-b328-3db9fe3748cf","Type":"ContainerStarted","Data":"c0eea2b22aa3b529b2d22d821d3b32b646150119a189fdd62d89374c5855cb66"}
Sep 18 20:03:25.180754 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:25.180712    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-76vkx" event={"ID":"83636bf0-4690-4e91-b328-3db9fe3748cf","Type":"ContainerStarted","Data":"eb2510d08d06a59fcf81daa376d673761340b989e7daf75a34ab2d1dc78116fd"}
Sep 18 20:03:25.207222 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:25.207174    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-76vkx" podStartSLOduration=1.656397074 podStartE2EDuration="11.207159924s" totalImagesPullingTime="9.55076285s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:03:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:03:25.206869085 +0000 UTC m=+390.160610254" watchObservedRunningTime="2026-09-18 20:03:25.207159924 +0000 UTC m=+390.160901093"
Sep 18 20:03:26.349258 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:26.349224    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-6sfdm"]
Sep 18 20:03:26.351276 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:26.351260    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-6sfdm"
Sep 18 20:03:26.354903 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:26.354880    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 18 20:03:26.355005 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:26.354891    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-gh5hp\""
Sep 18 20:03:26.355807 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:26.355789    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 18 20:03:26.365780 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:26.365758    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-6sfdm"]
Sep 18 20:03:26.419871 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:26.419848    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsg9m\" (UniqueName: \"kubernetes.io/projected/0ff50297-bf00-42a8-9cba-31eb5062f0d8-kube-api-access-zsg9m\") pod \"cert-manager-webhook-6486ffd997-6sfdm\" (UID: \"0ff50297-bf00-42a8-9cba-31eb5062f0d8\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6sfdm"
Sep 18 20:03:26.419967 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:26.419901    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0ff50297-bf00-42a8-9cba-31eb5062f0d8-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-6sfdm\" (UID: \"0ff50297-bf00-42a8-9cba-31eb5062f0d8\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6sfdm"
Sep 18 20:03:26.520272 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:26.520241    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsg9m\" (UniqueName: \"kubernetes.io/projected/0ff50297-bf00-42a8-9cba-31eb5062f0d8-kube-api-access-zsg9m\") pod \"cert-manager-webhook-6486ffd997-6sfdm\" (UID: \"0ff50297-bf00-42a8-9cba-31eb5062f0d8\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6sfdm"
Sep 18 20:03:26.520427 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:26.520314    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0ff50297-bf00-42a8-9cba-31eb5062f0d8-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-6sfdm\" (UID: \"0ff50297-bf00-42a8-9cba-31eb5062f0d8\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6sfdm"
Sep 18 20:03:26.529450 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:26.529425    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0ff50297-bf00-42a8-9cba-31eb5062f0d8-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-6sfdm\" (UID: \"0ff50297-bf00-42a8-9cba-31eb5062f0d8\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6sfdm"
Sep 18 20:03:26.529721 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:26.529700    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsg9m\" (UniqueName: \"kubernetes.io/projected/0ff50297-bf00-42a8-9cba-31eb5062f0d8-kube-api-access-zsg9m\") pod \"cert-manager-webhook-6486ffd997-6sfdm\" (UID: \"0ff50297-bf00-42a8-9cba-31eb5062f0d8\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6sfdm"
Sep 18 20:03:26.675879 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:26.675806    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-6sfdm"
Sep 18 20:03:26.949756 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:26.949734    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-6sfdm"]
Sep 18 20:03:26.952135 ip-10-0-140-178 kubenswrapper[2549]: W0918 20:03:26.952100    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0ff50297_bf00_42a8_9cba_31eb5062f0d8.slice/crio-38003c946d4133c753a5ed9b5c1c3ce560e924464951a6d9c0718c9e323b5e03 WatchSource:0}: Error finding container 38003c946d4133c753a5ed9b5c1c3ce560e924464951a6d9c0718c9e323b5e03: Status 404 returned error can't find the container with id 38003c946d4133c753a5ed9b5c1c3ce560e924464951a6d9c0718c9e323b5e03
Sep 18 20:03:27.189480 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:27.189445    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-6sfdm" event={"ID":"0ff50297-bf00-42a8-9cba-31eb5062f0d8","Type":"ContainerStarted","Data":"38003c946d4133c753a5ed9b5c1c3ce560e924464951a6d9c0718c9e323b5e03"}
Sep 18 20:03:30.205425 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:30.205364    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-6sfdm" event={"ID":"0ff50297-bf00-42a8-9cba-31eb5062f0d8","Type":"ContainerStarted","Data":"7a154c6de7fc527c7c20df236cdaa72b5fc3cc6f3bad23ba162e5da26fa78db2"}
Sep 18 20:03:30.205784 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:30.205558    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-6sfdm"
Sep 18 20:03:30.226514 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:30.226471    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-6sfdm" podStartSLOduration=1.57052887 podStartE2EDuration="4.226457265s" totalImagesPullingTime="2.655928395s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:03:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:03:30.226378797 +0000 UTC m=+395.180119967" watchObservedRunningTime="2026-09-18 20:03:30.226457265 +0000 UTC m=+395.180198433"
Sep 18 20:03:30.543698 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:30.543627    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-zgqwf"]
Sep 18 20:03:30.546185 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:30.546171    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-zgqwf"
Sep 18 20:03:30.549295 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:30.549274    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-2zb5x\""
Sep 18 20:03:30.556415 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:30.556394    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-zgqwf"]
Sep 18 20:03:30.653304 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:30.653266    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3a81be74-ef26-49d2-aea9-da6c322973e6-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-zgqwf\" (UID: \"3a81be74-ef26-49d2-aea9-da6c322973e6\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-zgqwf"
Sep 18 20:03:30.653491 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:30.653353    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhblx\" (UniqueName: \"kubernetes.io/projected/3a81be74-ef26-49d2-aea9-da6c322973e6-kube-api-access-hhblx\") pod \"cert-manager-cainjector-559cdd8c87-zgqwf\" (UID: \"3a81be74-ef26-49d2-aea9-da6c322973e6\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-zgqwf"
Sep 18 20:03:30.754779 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:30.754746    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3a81be74-ef26-49d2-aea9-da6c322973e6-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-zgqwf\" (UID: \"3a81be74-ef26-49d2-aea9-da6c322973e6\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-zgqwf"
Sep 18 20:03:30.754915 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:30.754801    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhblx\" (UniqueName: \"kubernetes.io/projected/3a81be74-ef26-49d2-aea9-da6c322973e6-kube-api-access-hhblx\") pod \"cert-manager-cainjector-559cdd8c87-zgqwf\" (UID: \"3a81be74-ef26-49d2-aea9-da6c322973e6\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-zgqwf"
Sep 18 20:03:30.764111 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:30.764086    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3a81be74-ef26-49d2-aea9-da6c322973e6-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-zgqwf\" (UID: \"3a81be74-ef26-49d2-aea9-da6c322973e6\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-zgqwf"
Sep 18 20:03:30.764298 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:30.764282    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhblx\" (UniqueName: \"kubernetes.io/projected/3a81be74-ef26-49d2-aea9-da6c322973e6-kube-api-access-hhblx\") pod \"cert-manager-cainjector-559cdd8c87-zgqwf\" (UID: \"3a81be74-ef26-49d2-aea9-da6c322973e6\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-zgqwf"
Sep 18 20:03:30.855476 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:30.855448    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-zgqwf"
Sep 18 20:03:31.124670 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:31.124627    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-zgqwf"]
Sep 18 20:03:31.126823 ip-10-0-140-178 kubenswrapper[2549]: W0918 20:03:31.126794    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3a81be74_ef26_49d2_aea9_da6c322973e6.slice/crio-56fb415240f02eda366f89ad6fe566c7a07d86c92760d96a85f7e9845ca2aa4d WatchSource:0}: Error finding container 56fb415240f02eda366f89ad6fe566c7a07d86c92760d96a85f7e9845ca2aa4d: Status 404 returned error can't find the container with id 56fb415240f02eda366f89ad6fe566c7a07d86c92760d96a85f7e9845ca2aa4d
Sep 18 20:03:31.211402 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:31.211352    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-zgqwf" event={"ID":"3a81be74-ef26-49d2-aea9-da6c322973e6","Type":"ContainerStarted","Data":"5695a1b5b328f8caa6758d065a8358f0679da9b695b5abfdf736cf6cd66ba774"}
Sep 18 20:03:31.211755 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:31.211417    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-zgqwf" event={"ID":"3a81be74-ef26-49d2-aea9-da6c322973e6","Type":"ContainerStarted","Data":"56fb415240f02eda366f89ad6fe566c7a07d86c92760d96a85f7e9845ca2aa4d"}
Sep 18 20:03:31.232166 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:31.232000    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-zgqwf" podStartSLOduration=1.231979691 podStartE2EDuration="1.231979691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:03:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:03:31.230360071 +0000 UTC m=+396.184101240" watchObservedRunningTime="2026-09-18 20:03:31.231979691 +0000 UTC m=+396.185720859"
Sep 18 20:03:36.214230 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:36.214182    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-6sfdm"
Sep 18 20:03:57.064265 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:57.064226    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxn4wg"]
Sep 18 20:03:57.066767 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:57.066743    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxn4wg"
Sep 18 20:03:57.069672 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:57.069653    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 18 20:03:57.070977 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:57.070957    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 18 20:03:57.071108 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:57.070989    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-zjfq9\""
Sep 18 20:03:57.076189 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:57.076166    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxn4wg"]
Sep 18 20:03:57.154550 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:57.154524    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5eaf8780-a8d2-49b0-baef-ad4ea5fa8f0f-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxn4wg\" (UID: \"5eaf8780-a8d2-49b0-baef-ad4ea5fa8f0f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxn4wg"
Sep 18 20:03:57.154695 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:57.154559    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5eaf8780-a8d2-49b0-baef-ad4ea5fa8f0f-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxn4wg\" (UID: \"5eaf8780-a8d2-49b0-baef-ad4ea5fa8f0f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxn4wg"
Sep 18 20:03:57.154695 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:57.154630    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcxcq\" (UniqueName: \"kubernetes.io/projected/5eaf8780-a8d2-49b0-baef-ad4ea5fa8f0f-kube-api-access-vcxcq\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxn4wg\" (UID: \"5eaf8780-a8d2-49b0-baef-ad4ea5fa8f0f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxn4wg"
Sep 18 20:03:57.255648 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:57.255617    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcxcq\" (UniqueName: \"kubernetes.io/projected/5eaf8780-a8d2-49b0-baef-ad4ea5fa8f0f-kube-api-access-vcxcq\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxn4wg\" (UID: \"5eaf8780-a8d2-49b0-baef-ad4ea5fa8f0f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxn4wg"
Sep 18 20:03:57.255816 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:57.255700    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5eaf8780-a8d2-49b0-baef-ad4ea5fa8f0f-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxn4wg\" (UID: \"5eaf8780-a8d2-49b0-baef-ad4ea5fa8f0f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxn4wg"
Sep 18 20:03:57.255816 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:57.255734    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5eaf8780-a8d2-49b0-baef-ad4ea5fa8f0f-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxn4wg\" (UID: \"5eaf8780-a8d2-49b0-baef-ad4ea5fa8f0f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxn4wg"
Sep 18 20:03:57.256208 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:57.256183    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5eaf8780-a8d2-49b0-baef-ad4ea5fa8f0f-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxn4wg\" (UID: \"5eaf8780-a8d2-49b0-baef-ad4ea5fa8f0f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxn4wg"
Sep 18 20:03:57.256208 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:57.256201    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5eaf8780-a8d2-49b0-baef-ad4ea5fa8f0f-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxn4wg\" (UID: \"5eaf8780-a8d2-49b0-baef-ad4ea5fa8f0f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxn4wg"
Sep 18 20:03:57.266016 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:57.265989    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcxcq\" (UniqueName: \"kubernetes.io/projected/5eaf8780-a8d2-49b0-baef-ad4ea5fa8f0f-kube-api-access-vcxcq\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxn4wg\" (UID: \"5eaf8780-a8d2-49b0-baef-ad4ea5fa8f0f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxn4wg"
Sep 18 20:03:57.376978 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:57.376935    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxn4wg"
Sep 18 20:03:57.648143 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:57.648119    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxn4wg"]
Sep 18 20:03:57.650047 ip-10-0-140-178 kubenswrapper[2549]: W0918 20:03:57.650012    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eaf8780_a8d2_49b0_baef_ad4ea5fa8f0f.slice/crio-8ca241cd5ad37f8a9d5e85fb322266ba4ab7e61ac9551d5cd6a2703ef8773dc8 WatchSource:0}: Error finding container 8ca241cd5ad37f8a9d5e85fb322266ba4ab7e61ac9551d5cd6a2703ef8773dc8: Status 404 returned error can't find the container with id 8ca241cd5ad37f8a9d5e85fb322266ba4ab7e61ac9551d5cd6a2703ef8773dc8
Sep 18 20:03:58.330477 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:58.330439    2549 generic.go:356] "Generic (PLEG): container finished" podID="5eaf8780-a8d2-49b0-baef-ad4ea5fa8f0f" containerID="2a6278e7e87c5c0fae27a4f067ea3a0fc230d8a13ddc03d82c85a512807da2d9" exitCode=0
Sep 18 20:03:58.330818 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:58.330494    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxn4wg" event={"ID":"5eaf8780-a8d2-49b0-baef-ad4ea5fa8f0f","Type":"ContainerDied","Data":"2a6278e7e87c5c0fae27a4f067ea3a0fc230d8a13ddc03d82c85a512807da2d9"}
Sep 18 20:03:58.330818 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:58.330537    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxn4wg" event={"ID":"5eaf8780-a8d2-49b0-baef-ad4ea5fa8f0f","Type":"ContainerStarted","Data":"8ca241cd5ad37f8a9d5e85fb322266ba4ab7e61ac9551d5cd6a2703ef8773dc8"}
Sep 18 20:03:59.337149 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:59.337114    2549 generic.go:356] "Generic (PLEG): container finished" podID="5eaf8780-a8d2-49b0-baef-ad4ea5fa8f0f" containerID="832ea7d053345f86bfe8b055ee9b21db49cd4963fb8d74d3047843bab615d2d3" exitCode=0
Sep 18 20:03:59.337523 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:03:59.337157    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxn4wg" event={"ID":"5eaf8780-a8d2-49b0-baef-ad4ea5fa8f0f","Type":"ContainerDied","Data":"832ea7d053345f86bfe8b055ee9b21db49cd4963fb8d74d3047843bab615d2d3"}
Sep 18 20:04:00.343916 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:00.343882    2549 generic.go:356] "Generic (PLEG): container finished" podID="5eaf8780-a8d2-49b0-baef-ad4ea5fa8f0f" containerID="c5629a46aec762eec480c11e4b9d5aa3e96605d6aa1d0273f708d9da47dab44c" exitCode=0
Sep 18 20:04:00.343916 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:00.343917    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxn4wg" event={"ID":"5eaf8780-a8d2-49b0-baef-ad4ea5fa8f0f","Type":"ContainerDied","Data":"c5629a46aec762eec480c11e4b9d5aa3e96605d6aa1d0273f708d9da47dab44c"}
Sep 18 20:04:01.472570 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:01.472542    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxn4wg"
Sep 18 20:04:01.588666 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:01.588638    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vcxcq\" (UniqueName: \"kubernetes.io/projected/5eaf8780-a8d2-49b0-baef-ad4ea5fa8f0f-kube-api-access-vcxcq\") pod \"5eaf8780-a8d2-49b0-baef-ad4ea5fa8f0f\" (UID: \"5eaf8780-a8d2-49b0-baef-ad4ea5fa8f0f\") "
Sep 18 20:04:01.588827 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:01.588737    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5eaf8780-a8d2-49b0-baef-ad4ea5fa8f0f-util\") pod \"5eaf8780-a8d2-49b0-baef-ad4ea5fa8f0f\" (UID: \"5eaf8780-a8d2-49b0-baef-ad4ea5fa8f0f\") "
Sep 18 20:04:01.588827 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:01.588773    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5eaf8780-a8d2-49b0-baef-ad4ea5fa8f0f-bundle\") pod \"5eaf8780-a8d2-49b0-baef-ad4ea5fa8f0f\" (UID: \"5eaf8780-a8d2-49b0-baef-ad4ea5fa8f0f\") "
Sep 18 20:04:01.589471 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:01.589447    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eaf8780-a8d2-49b0-baef-ad4ea5fa8f0f-bundle" (OuterVolumeSpecName: "bundle") pod "5eaf8780-a8d2-49b0-baef-ad4ea5fa8f0f" (UID: "5eaf8780-a8d2-49b0-baef-ad4ea5fa8f0f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:04:01.590835 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:01.590807    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eaf8780-a8d2-49b0-baef-ad4ea5fa8f0f-kube-api-access-vcxcq" (OuterVolumeSpecName: "kube-api-access-vcxcq") pod "5eaf8780-a8d2-49b0-baef-ad4ea5fa8f0f" (UID: "5eaf8780-a8d2-49b0-baef-ad4ea5fa8f0f"). InnerVolumeSpecName "kube-api-access-vcxcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:04:01.594118 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:01.594099    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eaf8780-a8d2-49b0-baef-ad4ea5fa8f0f-util" (OuterVolumeSpecName: "util") pod "5eaf8780-a8d2-49b0-baef-ad4ea5fa8f0f" (UID: "5eaf8780-a8d2-49b0-baef-ad4ea5fa8f0f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:04:01.631300 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:01.631245    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5848b5686c-597k2"]
Sep 18 20:04:01.631576 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:01.631562    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5eaf8780-a8d2-49b0-baef-ad4ea5fa8f0f" containerName="pull"
Sep 18 20:04:01.631628 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:01.631578    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="5eaf8780-a8d2-49b0-baef-ad4ea5fa8f0f" containerName="pull"
Sep 18 20:04:01.631628 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:01.631588    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5eaf8780-a8d2-49b0-baef-ad4ea5fa8f0f" containerName="extract"
Sep 18 20:04:01.631628 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:01.631593    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="5eaf8780-a8d2-49b0-baef-ad4ea5fa8f0f" containerName="extract"
Sep 18 20:04:01.631628 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:01.631605    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5eaf8780-a8d2-49b0-baef-ad4ea5fa8f0f" containerName="util"
Sep 18 20:04:01.631628 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:01.631610    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="5eaf8780-a8d2-49b0-baef-ad4ea5fa8f0f" containerName="util"
Sep 18 20:04:01.631773 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:01.631666    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="5eaf8780-a8d2-49b0-baef-ad4ea5fa8f0f" containerName="extract"
Sep 18 20:04:01.633492 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:01.633467    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5848b5686c-597k2"
Sep 18 20:04:01.639227 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:01.639204    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 18 20:04:01.639456 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:01.639433    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 18 20:04:01.639564 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:01.639554    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 18 20:04:01.639701 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:01.639680    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-jcn2q\""
Sep 18 20:04:01.639766 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:01.639737    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 18 20:04:01.639926 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:01.639912    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 18 20:04:01.652068 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:01.652047    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5848b5686c-597k2"]
Sep 18 20:04:01.689425 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:01.689404    2549 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5eaf8780-a8d2-49b0-baef-ad4ea5fa8f0f-bundle\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:04:01.689425 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:01.689425    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcxcq\" (UniqueName: \"kubernetes.io/projected/5eaf8780-a8d2-49b0-baef-ad4ea5fa8f0f-kube-api-access-vcxcq\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:04:01.689560 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:01.689435    2549 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5eaf8780-a8d2-49b0-baef-ad4ea5fa8f0f-util\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:04:01.790176 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:01.790153    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c541712c-e34c-4b54-90c6-0acfc9faa0bc-metrics-cert\") pod \"lws-controller-manager-5848b5686c-597k2\" (UID: \"c541712c-e34c-4b54-90c6-0acfc9faa0bc\") " pod="openshift-lws-operator/lws-controller-manager-5848b5686c-597k2"
Sep 18 20:04:01.790293 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:01.790187    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bls27\" (UniqueName: \"kubernetes.io/projected/c541712c-e34c-4b54-90c6-0acfc9faa0bc-kube-api-access-bls27\") pod \"lws-controller-manager-5848b5686c-597k2\" (UID: \"c541712c-e34c-4b54-90c6-0acfc9faa0bc\") " pod="openshift-lws-operator/lws-controller-manager-5848b5686c-597k2"
Sep 18 20:04:01.790293 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:01.790229    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c541712c-e34c-4b54-90c6-0acfc9faa0bc-manager-config\") pod \"lws-controller-manager-5848b5686c-597k2\" (UID: \"c541712c-e34c-4b54-90c6-0acfc9faa0bc\") " pod="openshift-lws-operator/lws-controller-manager-5848b5686c-597k2"
Sep 18 20:04:01.790293 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:01.790273    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c541712c-e34c-4b54-90c6-0acfc9faa0bc-cert\") pod \"lws-controller-manager-5848b5686c-597k2\" (UID: \"c541712c-e34c-4b54-90c6-0acfc9faa0bc\") " pod="openshift-lws-operator/lws-controller-manager-5848b5686c-597k2"
Sep 18 20:04:01.891118 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:01.891042    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bls27\" (UniqueName: \"kubernetes.io/projected/c541712c-e34c-4b54-90c6-0acfc9faa0bc-kube-api-access-bls27\") pod \"lws-controller-manager-5848b5686c-597k2\" (UID: \"c541712c-e34c-4b54-90c6-0acfc9faa0bc\") " pod="openshift-lws-operator/lws-controller-manager-5848b5686c-597k2"
Sep 18 20:04:01.891118 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:01.891092    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c541712c-e34c-4b54-90c6-0acfc9faa0bc-manager-config\") pod \"lws-controller-manager-5848b5686c-597k2\" (UID: \"c541712c-e34c-4b54-90c6-0acfc9faa0bc\") " pod="openshift-lws-operator/lws-controller-manager-5848b5686c-597k2"
Sep 18 20:04:01.891118 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:01.891118    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c541712c-e34c-4b54-90c6-0acfc9faa0bc-cert\") pod \"lws-controller-manager-5848b5686c-597k2\" (UID: \"c541712c-e34c-4b54-90c6-0acfc9faa0bc\") " pod="openshift-lws-operator/lws-controller-manager-5848b5686c-597k2"
Sep 18 20:04:01.891300 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:01.891148    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c541712c-e34c-4b54-90c6-0acfc9faa0bc-metrics-cert\") pod \"lws-controller-manager-5848b5686c-597k2\" (UID: \"c541712c-e34c-4b54-90c6-0acfc9faa0bc\") " pod="openshift-lws-operator/lws-controller-manager-5848b5686c-597k2"
Sep 18 20:04:01.891830 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:01.891804    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c541712c-e34c-4b54-90c6-0acfc9faa0bc-manager-config\") pod \"lws-controller-manager-5848b5686c-597k2\" (UID: \"c541712c-e34c-4b54-90c6-0acfc9faa0bc\") " pod="openshift-lws-operator/lws-controller-manager-5848b5686c-597k2"
Sep 18 20:04:01.893542 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:01.893514    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c541712c-e34c-4b54-90c6-0acfc9faa0bc-metrics-cert\") pod \"lws-controller-manager-5848b5686c-597k2\" (UID: \"c541712c-e34c-4b54-90c6-0acfc9faa0bc\") " pod="openshift-lws-operator/lws-controller-manager-5848b5686c-597k2"
Sep 18 20:04:01.893630 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:01.893517    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c541712c-e34c-4b54-90c6-0acfc9faa0bc-cert\") pod \"lws-controller-manager-5848b5686c-597k2\" (UID: \"c541712c-e34c-4b54-90c6-0acfc9faa0bc\") " pod="openshift-lws-operator/lws-controller-manager-5848b5686c-597k2"
Sep 18 20:04:01.903193 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:01.903174    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bls27\" (UniqueName: \"kubernetes.io/projected/c541712c-e34c-4b54-90c6-0acfc9faa0bc-kube-api-access-bls27\") pod \"lws-controller-manager-5848b5686c-597k2\" (UID: \"c541712c-e34c-4b54-90c6-0acfc9faa0bc\") " pod="openshift-lws-operator/lws-controller-manager-5848b5686c-597k2"
Sep 18 20:04:01.943152 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:01.943126    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5848b5686c-597k2"
Sep 18 20:04:02.215412 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:02.215360    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5848b5686c-597k2"]
Sep 18 20:04:02.217897 ip-10-0-140-178 kubenswrapper[2549]: W0918 20:04:02.217869    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc541712c_e34c_4b54_90c6_0acfc9faa0bc.slice/crio-f10834b392f31e6a22ccef914574d294210e7d445fab80d1c00176727ee06f49 WatchSource:0}: Error finding container f10834b392f31e6a22ccef914574d294210e7d445fab80d1c00176727ee06f49: Status 404 returned error can't find the container with id f10834b392f31e6a22ccef914574d294210e7d445fab80d1c00176727ee06f49
Sep 18 20:04:02.355458 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:02.355436    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxn4wg"
Sep 18 20:04:02.355639 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:02.355428    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxn4wg" event={"ID":"5eaf8780-a8d2-49b0-baef-ad4ea5fa8f0f","Type":"ContainerDied","Data":"8ca241cd5ad37f8a9d5e85fb322266ba4ab7e61ac9551d5cd6a2703ef8773dc8"}
Sep 18 20:04:02.355639 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:02.355547    2549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ca241cd5ad37f8a9d5e85fb322266ba4ab7e61ac9551d5cd6a2703ef8773dc8"
Sep 18 20:04:02.356865 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:02.356835    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5848b5686c-597k2" event={"ID":"c541712c-e34c-4b54-90c6-0acfc9faa0bc","Type":"ContainerStarted","Data":"f10834b392f31e6a22ccef914574d294210e7d445fab80d1c00176727ee06f49"}
Sep 18 20:04:12.405045 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:12.405010    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5848b5686c-597k2" event={"ID":"c541712c-e34c-4b54-90c6-0acfc9faa0bc","Type":"ContainerStarted","Data":"4d0b5a2ae7dd61d4d593cf73bed7f5d657ef9885fb46ba9d29fbd4500106d87c"}
Sep 18 20:04:12.405418 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:12.405102    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5848b5686c-597k2"
Sep 18 20:04:12.426736 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:12.426686    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5848b5686c-597k2" podStartSLOduration=1.791404787 podStartE2EDuration="11.426670341s" totalImagesPullingTime="9.635265554s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:04:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:04:12.424664368 +0000 UTC m=+437.378405538" watchObservedRunningTime="2026-09-18 20:04:12.426670341 +0000 UTC m=+437.380411508"
Sep 18 20:04:13.297746 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:13.297710    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zpgj5"]
Sep 18 20:04:13.300013 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:13.299997    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zpgj5"
Sep 18 20:04:13.303084 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:13.303069    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 18 20:04:13.303487 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:13.303460    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-zjfq9\""
Sep 18 20:04:13.304643 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:13.304630    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 18 20:04:13.313164 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:13.313144    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zpgj5"]
Sep 18 20:04:13.386672 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:13.386643    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c69f77e3-a5e5-446d-8d79-7dc3dc61f195-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zpgj5\" (UID: \"c69f77e3-a5e5-446d-8d79-7dc3dc61f195\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zpgj5"
Sep 18 20:04:13.386811 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:13.386681    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c69f77e3-a5e5-446d-8d79-7dc3dc61f195-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zpgj5\" (UID: \"c69f77e3-a5e5-446d-8d79-7dc3dc61f195\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zpgj5"
Sep 18 20:04:13.386811 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:13.386724    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ln5mv\" (UniqueName: \"kubernetes.io/projected/c69f77e3-a5e5-446d-8d79-7dc3dc61f195-kube-api-access-ln5mv\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zpgj5\" (UID: \"c69f77e3-a5e5-446d-8d79-7dc3dc61f195\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zpgj5"
Sep 18 20:04:13.487488 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:13.487458    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c69f77e3-a5e5-446d-8d79-7dc3dc61f195-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zpgj5\" (UID: \"c69f77e3-a5e5-446d-8d79-7dc3dc61f195\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zpgj5"
Sep 18 20:04:13.487912 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:13.487497    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c69f77e3-a5e5-446d-8d79-7dc3dc61f195-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zpgj5\" (UID: \"c69f77e3-a5e5-446d-8d79-7dc3dc61f195\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zpgj5"
Sep 18 20:04:13.487912 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:13.487535    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln5mv\" (UniqueName: \"kubernetes.io/projected/c69f77e3-a5e5-446d-8d79-7dc3dc61f195-kube-api-access-ln5mv\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zpgj5\" (UID: \"c69f77e3-a5e5-446d-8d79-7dc3dc61f195\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zpgj5"
Sep 18 20:04:13.487912 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:13.487874    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c69f77e3-a5e5-446d-8d79-7dc3dc61f195-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zpgj5\" (UID: \"c69f77e3-a5e5-446d-8d79-7dc3dc61f195\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zpgj5"
Sep 18 20:04:13.488020 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:13.487917    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c69f77e3-a5e5-446d-8d79-7dc3dc61f195-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zpgj5\" (UID: \"c69f77e3-a5e5-446d-8d79-7dc3dc61f195\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zpgj5"
Sep 18 20:04:13.500301 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:13.500272    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln5mv\" (UniqueName: \"kubernetes.io/projected/c69f77e3-a5e5-446d-8d79-7dc3dc61f195-kube-api-access-ln5mv\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zpgj5\" (UID: \"c69f77e3-a5e5-446d-8d79-7dc3dc61f195\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zpgj5"
Sep 18 20:04:13.609433 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:13.609405    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zpgj5"
Sep 18 20:04:13.885455 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:13.885429    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zpgj5"]
Sep 18 20:04:13.887325 ip-10-0-140-178 kubenswrapper[2549]: W0918 20:04:13.887298    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc69f77e3_a5e5_446d_8d79_7dc3dc61f195.slice/crio-7f73698fd609bcf40f7fce03d7f29d43c398fee60257915f297003333cb3e927 WatchSource:0}: Error finding container 7f73698fd609bcf40f7fce03d7f29d43c398fee60257915f297003333cb3e927: Status 404 returned error can't find the container with id 7f73698fd609bcf40f7fce03d7f29d43c398fee60257915f297003333cb3e927
Sep 18 20:04:14.071771 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:14.071736    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-f76f79b89-5bbqs"]
Sep 18 20:04:14.073898 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:14.073875    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-f76f79b89-5bbqs"
Sep 18 20:04:14.077986 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:14.077961    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 18 20:04:14.078083 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:14.078064    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 18 20:04:14.079730 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:14.079713    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 18 20:04:14.079817 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:14.079749    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-cphdw\""
Sep 18 20:04:14.079870 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:14.079823    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 18 20:04:14.099731 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:14.099708    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-f76f79b89-5bbqs"]
Sep 18 20:04:14.192233 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:14.192163    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3780e3fd-8628-4933-9654-3c7a08699756-manifests\") pod \"opendatahub-operator-controller-manager-f76f79b89-5bbqs\" (UID: \"3780e3fd-8628-4933-9654-3c7a08699756\") " pod="opendatahub/opendatahub-operator-controller-manager-f76f79b89-5bbqs"
Sep 18 20:04:14.192233 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:14.192225    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3780e3fd-8628-4933-9654-3c7a08699756-webhook-cert\") pod \"opendatahub-operator-controller-manager-f76f79b89-5bbqs\" (UID: \"3780e3fd-8628-4933-9654-3c7a08699756\") " pod="opendatahub/opendatahub-operator-controller-manager-f76f79b89-5bbqs"
Sep 18 20:04:14.192409 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:14.192246    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pxpq\" (UniqueName: \"kubernetes.io/projected/3780e3fd-8628-4933-9654-3c7a08699756-kube-api-access-8pxpq\") pod \"opendatahub-operator-controller-manager-f76f79b89-5bbqs\" (UID: \"3780e3fd-8628-4933-9654-3c7a08699756\") " pod="opendatahub/opendatahub-operator-controller-manager-f76f79b89-5bbqs"
Sep 18 20:04:14.192409 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:14.192271    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3780e3fd-8628-4933-9654-3c7a08699756-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f76f79b89-5bbqs\" (UID: \"3780e3fd-8628-4933-9654-3c7a08699756\") " pod="opendatahub/opendatahub-operator-controller-manager-f76f79b89-5bbqs"
Sep 18 20:04:14.192409 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:14.192355    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3780e3fd-8628-4933-9654-3c7a08699756-tmp\") pod \"opendatahub-operator-controller-manager-f76f79b89-5bbqs\" (UID: \"3780e3fd-8628-4933-9654-3c7a08699756\") " pod="opendatahub/opendatahub-operator-controller-manager-f76f79b89-5bbqs"
Sep 18 20:04:14.293594 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:14.293563    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pxpq\" (UniqueName: \"kubernetes.io/projected/3780e3fd-8628-4933-9654-3c7a08699756-kube-api-access-8pxpq\") pod \"opendatahub-operator-controller-manager-f76f79b89-5bbqs\" (UID: \"3780e3fd-8628-4933-9654-3c7a08699756\") " pod="opendatahub/opendatahub-operator-controller-manager-f76f79b89-5bbqs"
Sep 18 20:04:14.293594 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:14.293596    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3780e3fd-8628-4933-9654-3c7a08699756-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f76f79b89-5bbqs\" (UID: \"3780e3fd-8628-4933-9654-3c7a08699756\") " pod="opendatahub/opendatahub-operator-controller-manager-f76f79b89-5bbqs"
Sep 18 20:04:14.293775 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:14.293621    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3780e3fd-8628-4933-9654-3c7a08699756-tmp\") pod \"opendatahub-operator-controller-manager-f76f79b89-5bbqs\" (UID: \"3780e3fd-8628-4933-9654-3c7a08699756\") " pod="opendatahub/opendatahub-operator-controller-manager-f76f79b89-5bbqs"
Sep 18 20:04:14.293775 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:14.293667    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3780e3fd-8628-4933-9654-3c7a08699756-manifests\") pod \"opendatahub-operator-controller-manager-f76f79b89-5bbqs\" (UID: \"3780e3fd-8628-4933-9654-3c7a08699756\") " pod="opendatahub/opendatahub-operator-controller-manager-f76f79b89-5bbqs"
Sep 18 20:04:14.293775 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:14.293751    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3780e3fd-8628-4933-9654-3c7a08699756-webhook-cert\") pod \"opendatahub-operator-controller-manager-f76f79b89-5bbqs\" (UID: \"3780e3fd-8628-4933-9654-3c7a08699756\") " pod="opendatahub/opendatahub-operator-controller-manager-f76f79b89-5bbqs"
Sep 18 20:04:14.294179 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:14.294146    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3780e3fd-8628-4933-9654-3c7a08699756-manifests\") pod \"opendatahub-operator-controller-manager-f76f79b89-5bbqs\" (UID: \"3780e3fd-8628-4933-9654-3c7a08699756\") " pod="opendatahub/opendatahub-operator-controller-manager-f76f79b89-5bbqs"
Sep 18 20:04:14.295913 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:14.295890    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3780e3fd-8628-4933-9654-3c7a08699756-tmp\") pod \"opendatahub-operator-controller-manager-f76f79b89-5bbqs\" (UID: \"3780e3fd-8628-4933-9654-3c7a08699756\") " pod="opendatahub/opendatahub-operator-controller-manager-f76f79b89-5bbqs"
Sep 18 20:04:14.296147 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:14.296131    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3780e3fd-8628-4933-9654-3c7a08699756-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f76f79b89-5bbqs\" (UID: \"3780e3fd-8628-4933-9654-3c7a08699756\") " pod="opendatahub/opendatahub-operator-controller-manager-f76f79b89-5bbqs"
Sep 18 20:04:14.296147 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:14.296138    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3780e3fd-8628-4933-9654-3c7a08699756-webhook-cert\") pod \"opendatahub-operator-controller-manager-f76f79b89-5bbqs\" (UID: \"3780e3fd-8628-4933-9654-3c7a08699756\") " pod="opendatahub/opendatahub-operator-controller-manager-f76f79b89-5bbqs"
Sep 18 20:04:14.307928 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:14.307910    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pxpq\" (UniqueName: \"kubernetes.io/projected/3780e3fd-8628-4933-9654-3c7a08699756-kube-api-access-8pxpq\") pod \"opendatahub-operator-controller-manager-f76f79b89-5bbqs\" (UID: \"3780e3fd-8628-4933-9654-3c7a08699756\") " pod="opendatahub/opendatahub-operator-controller-manager-f76f79b89-5bbqs"
Sep 18 20:04:14.383780 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:14.383759    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-f76f79b89-5bbqs"
Sep 18 20:04:14.420078 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:14.420044    2549 generic.go:356] "Generic (PLEG): container finished" podID="c69f77e3-a5e5-446d-8d79-7dc3dc61f195" containerID="27d9bba9a770e6faa25837682bdec43556d4778586343440e40a21d2cf297ce6" exitCode=0
Sep 18 20:04:14.420509 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:14.420125    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zpgj5" event={"ID":"c69f77e3-a5e5-446d-8d79-7dc3dc61f195","Type":"ContainerDied","Data":"27d9bba9a770e6faa25837682bdec43556d4778586343440e40a21d2cf297ce6"}
Sep 18 20:04:14.420509 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:14.420190    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zpgj5" event={"ID":"c69f77e3-a5e5-446d-8d79-7dc3dc61f195","Type":"ContainerStarted","Data":"7f73698fd609bcf40f7fce03d7f29d43c398fee60257915f297003333cb3e927"}
Sep 18 20:04:14.665626 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:14.665601    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-f76f79b89-5bbqs"]
Sep 18 20:04:14.667163 ip-10-0-140-178 kubenswrapper[2549]: W0918 20:04:14.667137    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3780e3fd_8628_4933_9654_3c7a08699756.slice/crio-11a65c3d18d31b2ccbe333c6ef80aeb510d01ddede6739b8e274111b5c484809 WatchSource:0}: Error finding container 11a65c3d18d31b2ccbe333c6ef80aeb510d01ddede6739b8e274111b5c484809: Status 404 returned error can't find the container with id 11a65c3d18d31b2ccbe333c6ef80aeb510d01ddede6739b8e274111b5c484809
Sep 18 20:04:15.431054 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:15.430876    2549 generic.go:356] "Generic (PLEG): container finished" podID="c69f77e3-a5e5-446d-8d79-7dc3dc61f195" containerID="ea7ff78c9b0e1a9329c8415a4c9f187035c5e1e85756011eb28cdb1691c5553a" exitCode=0
Sep 18 20:04:15.431054 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:15.430995    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zpgj5" event={"ID":"c69f77e3-a5e5-446d-8d79-7dc3dc61f195","Type":"ContainerDied","Data":"ea7ff78c9b0e1a9329c8415a4c9f187035c5e1e85756011eb28cdb1691c5553a"}
Sep 18 20:04:15.436603 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:15.436573    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-f76f79b89-5bbqs" event={"ID":"3780e3fd-8628-4933-9654-3c7a08699756","Type":"ContainerStarted","Data":"11a65c3d18d31b2ccbe333c6ef80aeb510d01ddede6739b8e274111b5c484809"}
Sep 18 20:04:15.796337 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:15.796257    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c4b8dd4d8-mtlrh"]
Sep 18 20:04:16.444522 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:16.444378    2549 generic.go:356] "Generic (PLEG): container finished" podID="c69f77e3-a5e5-446d-8d79-7dc3dc61f195" containerID="fd92acfa9491c2d4527e62f606c6bdc1e6b71f9c3e0d42f47a8928800c2223e9" exitCode=0
Sep 18 20:04:16.444522 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:16.444444    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zpgj5" event={"ID":"c69f77e3-a5e5-446d-8d79-7dc3dc61f195","Type":"ContainerDied","Data":"fd92acfa9491c2d4527e62f606c6bdc1e6b71f9c3e0d42f47a8928800c2223e9"}
Sep 18 20:04:17.661136 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:17.661109    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zpgj5"
Sep 18 20:04:17.821351 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:17.821329    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c69f77e3-a5e5-446d-8d79-7dc3dc61f195-bundle\") pod \"c69f77e3-a5e5-446d-8d79-7dc3dc61f195\" (UID: \"c69f77e3-a5e5-446d-8d79-7dc3dc61f195\") "
Sep 18 20:04:17.821505 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:17.821401    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ln5mv\" (UniqueName: \"kubernetes.io/projected/c69f77e3-a5e5-446d-8d79-7dc3dc61f195-kube-api-access-ln5mv\") pod \"c69f77e3-a5e5-446d-8d79-7dc3dc61f195\" (UID: \"c69f77e3-a5e5-446d-8d79-7dc3dc61f195\") "
Sep 18 20:04:17.821505 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:17.821452    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c69f77e3-a5e5-446d-8d79-7dc3dc61f195-util\") pod \"c69f77e3-a5e5-446d-8d79-7dc3dc61f195\" (UID: \"c69f77e3-a5e5-446d-8d79-7dc3dc61f195\") "
Sep 18 20:04:17.822201 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:17.822174    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c69f77e3-a5e5-446d-8d79-7dc3dc61f195-bundle" (OuterVolumeSpecName: "bundle") pod "c69f77e3-a5e5-446d-8d79-7dc3dc61f195" (UID: "c69f77e3-a5e5-446d-8d79-7dc3dc61f195"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:04:17.823337 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:17.823307    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c69f77e3-a5e5-446d-8d79-7dc3dc61f195-kube-api-access-ln5mv" (OuterVolumeSpecName: "kube-api-access-ln5mv") pod "c69f77e3-a5e5-446d-8d79-7dc3dc61f195" (UID: "c69f77e3-a5e5-446d-8d79-7dc3dc61f195"). InnerVolumeSpecName "kube-api-access-ln5mv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:04:17.826813 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:17.826794    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c69f77e3-a5e5-446d-8d79-7dc3dc61f195-util" (OuterVolumeSpecName: "util") pod "c69f77e3-a5e5-446d-8d79-7dc3dc61f195" (UID: "c69f77e3-a5e5-446d-8d79-7dc3dc61f195"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:04:17.922962 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:17.922923    2549 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c69f77e3-a5e5-446d-8d79-7dc3dc61f195-bundle\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:04:17.922962 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:17.922963    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ln5mv\" (UniqueName: \"kubernetes.io/projected/c69f77e3-a5e5-446d-8d79-7dc3dc61f195-kube-api-access-ln5mv\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:04:17.922962 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:17.922973    2549 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c69f77e3-a5e5-446d-8d79-7dc3dc61f195-util\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:04:18.456832 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:18.456763    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zpgj5"
Sep 18 20:04:18.457008 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:18.456761    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zpgj5" event={"ID":"c69f77e3-a5e5-446d-8d79-7dc3dc61f195","Type":"ContainerDied","Data":"7f73698fd609bcf40f7fce03d7f29d43c398fee60257915f297003333cb3e927"}
Sep 18 20:04:18.457008 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:18.456891    2549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f73698fd609bcf40f7fce03d7f29d43c398fee60257915f297003333cb3e927"
Sep 18 20:04:18.458578 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:18.458551    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-f76f79b89-5bbqs" event={"ID":"3780e3fd-8628-4933-9654-3c7a08699756","Type":"ContainerStarted","Data":"5df02f8e60fc4890a2bafba52781d3162589b3fc01dd5546078ebced63c8ff31"}
Sep 18 20:04:19.469885 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:19.469852    2549 generic.go:356] "Generic (PLEG): container finished" podID="3780e3fd-8628-4933-9654-3c7a08699756" containerID="5df02f8e60fc4890a2bafba52781d3162589b3fc01dd5546078ebced63c8ff31" exitCode=0
Sep 18 20:04:19.470227 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:19.469935    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-f76f79b89-5bbqs" event={"ID":"3780e3fd-8628-4933-9654-3c7a08699756","Type":"ContainerDied","Data":"5df02f8e60fc4890a2bafba52781d3162589b3fc01dd5546078ebced63c8ff31"}
Sep 18 20:04:20.478799 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:20.478758    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-f76f79b89-5bbqs" event={"ID":"3780e3fd-8628-4933-9654-3c7a08699756","Type":"ContainerStarted","Data":"e5e2866b3b579acac54b28c2149520c46564e6e31da36a708c3d837a1c695d44"}
Sep 18 20:04:20.479279 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:20.478877    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-f76f79b89-5bbqs"
Sep 18 20:04:20.513523 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:20.513465    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-f76f79b89-5bbqs" podStartSLOduration=1.9539020520000001 podStartE2EDuration="6.513444301s" totalImagesPullingTime="3.549989065s" totalInitContainerRuntime="1.009553184s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:04:14 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:04:20.512889363 +0000 UTC m=+445.466630543" watchObservedRunningTime="2026-09-18 20:04:20.513444301 +0000 UTC m=+445.467185471"
Sep 18 20:04:23.414173 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:23.414146    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5848b5686c-597k2"
Sep 18 20:04:31.486071 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:31.486036    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-f76f79b89-5bbqs"
Sep 18 20:04:40.822637 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:40.822574    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6c4b8dd4d8-mtlrh" podUID="4e6a9e26-e76d-44c3-bb37-6258c7bf8907" containerName="console" containerID="cri-o://bd18a810f90dbd8d1729ed3e6850f236d4a4dbf6aa3126787e2411325a5e6a78" gracePeriod=15
Sep 18 20:04:41.065033 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:41.065010    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c4b8dd4d8-mtlrh"
Sep 18 20:04:41.115304 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:41.115228    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4e6a9e26-e76d-44c3-bb37-6258c7bf8907-oauth-serving-cert\") pod \"4e6a9e26-e76d-44c3-bb37-6258c7bf8907\" (UID: \"4e6a9e26-e76d-44c3-bb37-6258c7bf8907\") "
Sep 18 20:04:41.115304 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:41.115272    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4e6a9e26-e76d-44c3-bb37-6258c7bf8907-console-config\") pod \"4e6a9e26-e76d-44c3-bb37-6258c7bf8907\" (UID: \"4e6a9e26-e76d-44c3-bb37-6258c7bf8907\") "
Sep 18 20:04:41.115516 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:41.115310    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4e6a9e26-e76d-44c3-bb37-6258c7bf8907-console-serving-cert\") pod \"4e6a9e26-e76d-44c3-bb37-6258c7bf8907\" (UID: \"4e6a9e26-e76d-44c3-bb37-6258c7bf8907\") "
Sep 18 20:04:41.115516 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:41.115335    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4e6a9e26-e76d-44c3-bb37-6258c7bf8907-console-oauth-config\") pod \"4e6a9e26-e76d-44c3-bb37-6258c7bf8907\" (UID: \"4e6a9e26-e76d-44c3-bb37-6258c7bf8907\") "
Sep 18 20:04:41.115516 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:41.115355    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e6a9e26-e76d-44c3-bb37-6258c7bf8907-trusted-ca-bundle\") pod \"4e6a9e26-e76d-44c3-bb37-6258c7bf8907\" (UID: \"4e6a9e26-e76d-44c3-bb37-6258c7bf8907\") "
Sep 18 20:04:41.115516 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:41.115375    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xfjfq\" (UniqueName: \"kubernetes.io/projected/4e6a9e26-e76d-44c3-bb37-6258c7bf8907-kube-api-access-xfjfq\") pod \"4e6a9e26-e76d-44c3-bb37-6258c7bf8907\" (UID: \"4e6a9e26-e76d-44c3-bb37-6258c7bf8907\") "
Sep 18 20:04:41.115516 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:41.115458    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4e6a9e26-e76d-44c3-bb37-6258c7bf8907-service-ca\") pod \"4e6a9e26-e76d-44c3-bb37-6258c7bf8907\" (UID: \"4e6a9e26-e76d-44c3-bb37-6258c7bf8907\") "
Sep 18 20:04:41.115774 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:41.115664    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e6a9e26-e76d-44c3-bb37-6258c7bf8907-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "4e6a9e26-e76d-44c3-bb37-6258c7bf8907" (UID: "4e6a9e26-e76d-44c3-bb37-6258c7bf8907"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:04:41.115931 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:41.115899    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e6a9e26-e76d-44c3-bb37-6258c7bf8907-console-config" (OuterVolumeSpecName: "console-config") pod "4e6a9e26-e76d-44c3-bb37-6258c7bf8907" (UID: "4e6a9e26-e76d-44c3-bb37-6258c7bf8907"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:04:41.116001 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:41.115976    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e6a9e26-e76d-44c3-bb37-6258c7bf8907-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "4e6a9e26-e76d-44c3-bb37-6258c7bf8907" (UID: "4e6a9e26-e76d-44c3-bb37-6258c7bf8907"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:04:41.116074 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:41.116053    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e6a9e26-e76d-44c3-bb37-6258c7bf8907-service-ca" (OuterVolumeSpecName: "service-ca") pod "4e6a9e26-e76d-44c3-bb37-6258c7bf8907" (UID: "4e6a9e26-e76d-44c3-bb37-6258c7bf8907"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:04:41.117528 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:41.117506    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e6a9e26-e76d-44c3-bb37-6258c7bf8907-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "4e6a9e26-e76d-44c3-bb37-6258c7bf8907" (UID: "4e6a9e26-e76d-44c3-bb37-6258c7bf8907"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:04:41.118038 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:41.118011    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e6a9e26-e76d-44c3-bb37-6258c7bf8907-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "4e6a9e26-e76d-44c3-bb37-6258c7bf8907" (UID: "4e6a9e26-e76d-44c3-bb37-6258c7bf8907"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:04:41.118121 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:41.118017    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e6a9e26-e76d-44c3-bb37-6258c7bf8907-kube-api-access-xfjfq" (OuterVolumeSpecName: "kube-api-access-xfjfq") pod "4e6a9e26-e76d-44c3-bb37-6258c7bf8907" (UID: "4e6a9e26-e76d-44c3-bb37-6258c7bf8907"). InnerVolumeSpecName "kube-api-access-xfjfq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:04:41.216997 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:41.216969    2549 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4e6a9e26-e76d-44c3-bb37-6258c7bf8907-console-serving-cert\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:04:41.216997 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:41.216994    2549 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4e6a9e26-e76d-44c3-bb37-6258c7bf8907-console-oauth-config\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:04:41.216997 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:41.217004    2549 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e6a9e26-e76d-44c3-bb37-6258c7bf8907-trusted-ca-bundle\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:04:41.217197 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:41.217013    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfjfq\" (UniqueName: \"kubernetes.io/projected/4e6a9e26-e76d-44c3-bb37-6258c7bf8907-kube-api-access-xfjfq\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:04:41.217197 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:41.217023    2549 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4e6a9e26-e76d-44c3-bb37-6258c7bf8907-service-ca\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:04:41.217197 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:41.217030    2549 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4e6a9e26-e76d-44c3-bb37-6258c7bf8907-oauth-serving-cert\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:04:41.217197 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:41.217039    2549 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4e6a9e26-e76d-44c3-bb37-6258c7bf8907-console-config\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:04:41.569216 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:41.569182    2549 generic.go:356] "Generic (PLEG): container finished" podID="4e6a9e26-e76d-44c3-bb37-6258c7bf8907" containerID="bd18a810f90dbd8d1729ed3e6850f236d4a4dbf6aa3126787e2411325a5e6a78" exitCode=0
Sep 18 20:04:41.569340 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:41.569229    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c4b8dd4d8-mtlrh"
Sep 18 20:04:41.569340 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:41.569273    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c4b8dd4d8-mtlrh" event={"ID":"4e6a9e26-e76d-44c3-bb37-6258c7bf8907","Type":"ContainerDied","Data":"bd18a810f90dbd8d1729ed3e6850f236d4a4dbf6aa3126787e2411325a5e6a78"}
Sep 18 20:04:41.569340 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:41.569320    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c4b8dd4d8-mtlrh" event={"ID":"4e6a9e26-e76d-44c3-bb37-6258c7bf8907","Type":"ContainerDied","Data":"d1a054dbe430e8280c015a710b186bd2adf8acb7fb0a25843f9c9a63fca07276"}
Sep 18 20:04:41.569489 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:41.569341    2549 scope.go:117] "RemoveContainer" containerID="bd18a810f90dbd8d1729ed3e6850f236d4a4dbf6aa3126787e2411325a5e6a78"
Sep 18 20:04:41.577622 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:41.577606    2549 scope.go:117] "RemoveContainer" containerID="bd18a810f90dbd8d1729ed3e6850f236d4a4dbf6aa3126787e2411325a5e6a78"
Sep 18 20:04:41.577906 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:04:41.577886    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd18a810f90dbd8d1729ed3e6850f236d4a4dbf6aa3126787e2411325a5e6a78\": container with ID starting with bd18a810f90dbd8d1729ed3e6850f236d4a4dbf6aa3126787e2411325a5e6a78 not found: ID does not exist" containerID="bd18a810f90dbd8d1729ed3e6850f236d4a4dbf6aa3126787e2411325a5e6a78"
Sep 18 20:04:41.577987 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:41.577917    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd18a810f90dbd8d1729ed3e6850f236d4a4dbf6aa3126787e2411325a5e6a78"} err="failed to get container status \"bd18a810f90dbd8d1729ed3e6850f236d4a4dbf6aa3126787e2411325a5e6a78\": rpc error: code = NotFound desc = could not find container \"bd18a810f90dbd8d1729ed3e6850f236d4a4dbf6aa3126787e2411325a5e6a78\": container with ID starting with bd18a810f90dbd8d1729ed3e6850f236d4a4dbf6aa3126787e2411325a5e6a78 not found: ID does not exist"
Sep 18 20:04:41.592964 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:41.592935    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c4b8dd4d8-mtlrh"]
Sep 18 20:04:41.600156 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:41.600134    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6c4b8dd4d8-mtlrh"]
Sep 18 20:04:42.115368 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:42.115337    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5zktm"]
Sep 18 20:04:42.115819 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:42.115803    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c69f77e3-a5e5-446d-8d79-7dc3dc61f195" containerName="extract"
Sep 18 20:04:42.115861 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:42.115822    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="c69f77e3-a5e5-446d-8d79-7dc3dc61f195" containerName="extract"
Sep 18 20:04:42.115861 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:42.115835    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e6a9e26-e76d-44c3-bb37-6258c7bf8907" containerName="console"
Sep 18 20:04:42.115861 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:42.115845    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e6a9e26-e76d-44c3-bb37-6258c7bf8907" containerName="console"
Sep 18 20:04:42.115953 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:42.115868    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c69f77e3-a5e5-446d-8d79-7dc3dc61f195" containerName="pull"
Sep 18 20:04:42.115953 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:42.115877    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="c69f77e3-a5e5-446d-8d79-7dc3dc61f195" containerName="pull"
Sep 18 20:04:42.115953 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:42.115889    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c69f77e3-a5e5-446d-8d79-7dc3dc61f195" containerName="util"
Sep 18 20:04:42.115953 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:42.115898    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="c69f77e3-a5e5-446d-8d79-7dc3dc61f195" containerName="util"
Sep 18 20:04:42.116067 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:42.115976    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e6a9e26-e76d-44c3-bb37-6258c7bf8907" containerName="console"
Sep 18 20:04:42.116067 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:42.115992    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="c69f77e3-a5e5-446d-8d79-7dc3dc61f195" containerName="extract"
Sep 18 20:04:42.118880 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:42.118852    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5zktm"
Sep 18 20:04:42.125512 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:42.125492    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-qpnkq\""
Sep 18 20:04:42.125812 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:42.125791    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 18 20:04:42.135409 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:42.135371    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5zktm"]
Sep 18 20:04:42.225203 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:42.225177    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4gbv\" (UniqueName: \"kubernetes.io/projected/61625208-077f-481b-8160-1699ab7e11fd-kube-api-access-w4gbv\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5zktm\" (UID: \"61625208-077f-481b-8160-1699ab7e11fd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5zktm"
Sep 18 20:04:42.225203 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:42.225207    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61625208-077f-481b-8160-1699ab7e11fd-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5zktm\" (UID: \"61625208-077f-481b-8160-1699ab7e11fd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5zktm"
Sep 18 20:04:42.225371 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:42.225224    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61625208-077f-481b-8160-1699ab7e11fd-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5zktm\" (UID: \"61625208-077f-481b-8160-1699ab7e11fd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5zktm"
Sep 18 20:04:42.225371 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:42.225245    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61625208-077f-481b-8160-1699ab7e11fd-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5zktm\" (UID: \"61625208-077f-481b-8160-1699ab7e11fd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5zktm"
Sep 18 20:04:42.225371 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:42.225327    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61625208-077f-481b-8160-1699ab7e11fd-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5zktm\" (UID: \"61625208-077f-481b-8160-1699ab7e11fd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5zktm"
Sep 18 20:04:42.225495 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:42.225379    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61625208-077f-481b-8160-1699ab7e11fd-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5zktm\" (UID: \"61625208-077f-481b-8160-1699ab7e11fd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5zktm"
Sep 18 20:04:42.225495 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:42.225434    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61625208-077f-481b-8160-1699ab7e11fd-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5zktm\" (UID: \"61625208-077f-481b-8160-1699ab7e11fd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5zktm"
Sep 18 20:04:42.225495 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:42.225452    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61625208-077f-481b-8160-1699ab7e11fd-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5zktm\" (UID: \"61625208-077f-481b-8160-1699ab7e11fd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5zktm"
Sep 18 20:04:42.225589 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:42.225496    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61625208-077f-481b-8160-1699ab7e11fd-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5zktm\" (UID: \"61625208-077f-481b-8160-1699ab7e11fd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5zktm"
Sep 18 20:04:42.326295 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:42.326262    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61625208-077f-481b-8160-1699ab7e11fd-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5zktm\" (UID: \"61625208-077f-481b-8160-1699ab7e11fd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5zktm"
Sep 18 20:04:42.326456 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:42.326309    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61625208-077f-481b-8160-1699ab7e11fd-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5zktm\" (UID: \"61625208-077f-481b-8160-1699ab7e11fd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5zktm"
Sep 18 20:04:42.326456 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:42.326336    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61625208-077f-481b-8160-1699ab7e11fd-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5zktm\" (UID: \"61625208-077f-481b-8160-1699ab7e11fd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5zktm"
Sep 18 20:04:42.326456 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:42.326363    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61625208-077f-481b-8160-1699ab7e11fd-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5zktm\" (UID: \"61625208-077f-481b-8160-1699ab7e11fd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5zktm"
Sep 18 20:04:42.326456 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:42.326419    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61625208-077f-481b-8160-1699ab7e11fd-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5zktm\" (UID: \"61625208-077f-481b-8160-1699ab7e11fd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5zktm"
Sep 18 20:04:42.326456 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:42.326452    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4gbv\" (UniqueName: \"kubernetes.io/projected/61625208-077f-481b-8160-1699ab7e11fd-kube-api-access-w4gbv\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5zktm\" (UID: \"61625208-077f-481b-8160-1699ab7e11fd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5zktm"
Sep 18 20:04:42.326696 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:42.326476    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61625208-077f-481b-8160-1699ab7e11fd-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5zktm\" (UID: \"61625208-077f-481b-8160-1699ab7e11fd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5zktm"
Sep 18 20:04:42.326696 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:42.326498    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61625208-077f-481b-8160-1699ab7e11fd-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5zktm\" (UID: \"61625208-077f-481b-8160-1699ab7e11fd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5zktm"
Sep 18 20:04:42.326696 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:42.326529    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61625208-077f-481b-8160-1699ab7e11fd-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5zktm\" (UID: \"61625208-077f-481b-8160-1699ab7e11fd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5zktm"
Sep 18 20:04:42.326845 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:42.326802    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61625208-077f-481b-8160-1699ab7e11fd-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5zktm\" (UID: \"61625208-077f-481b-8160-1699ab7e11fd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5zktm"
Sep 18 20:04:42.326845 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:42.326825    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61625208-077f-481b-8160-1699ab7e11fd-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5zktm\" (UID: \"61625208-077f-481b-8160-1699ab7e11fd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5zktm"
Sep 18 20:04:42.327000 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:42.326974    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61625208-077f-481b-8160-1699ab7e11fd-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5zktm\" (UID: \"61625208-077f-481b-8160-1699ab7e11fd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5zktm"
Sep 18 20:04:42.327000 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:42.326992    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61625208-077f-481b-8160-1699ab7e11fd-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5zktm\" (UID: \"61625208-077f-481b-8160-1699ab7e11fd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5zktm"
Sep 18 20:04:42.327107 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:42.327042    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61625208-077f-481b-8160-1699ab7e11fd-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5zktm\" (UID: \"61625208-077f-481b-8160-1699ab7e11fd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5zktm"
Sep 18 20:04:42.328727 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:42.328707    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61625208-077f-481b-8160-1699ab7e11fd-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5zktm\" (UID: \"61625208-077f-481b-8160-1699ab7e11fd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5zktm"
Sep 18 20:04:42.328926 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:42.328911    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61625208-077f-481b-8160-1699ab7e11fd-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5zktm\" (UID: \"61625208-077f-481b-8160-1699ab7e11fd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5zktm"
Sep 18 20:04:42.337148 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:42.337128    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61625208-077f-481b-8160-1699ab7e11fd-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5zktm\" (UID: \"61625208-077f-481b-8160-1699ab7e11fd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5zktm"
Sep 18 20:04:42.337493 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:42.337450    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4gbv\" (UniqueName: \"kubernetes.io/projected/61625208-077f-481b-8160-1699ab7e11fd-kube-api-access-w4gbv\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5zktm\" (UID: \"61625208-077f-481b-8160-1699ab7e11fd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5zktm"
Sep 18 20:04:42.429662 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:42.429598    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5zktm"
Sep 18 20:04:42.703263 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:42.703238    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5zktm"]
Sep 18 20:04:42.705732 ip-10-0-140-178 kubenswrapper[2549]: W0918 20:04:42.705705    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61625208_077f_481b_8160_1699ab7e11fd.slice/crio-5b77eff8509ea3c8a0989d9e65e586a710612c06f6f415f2e54a805e2bfca3f8 WatchSource:0}: Error finding container 5b77eff8509ea3c8a0989d9e65e586a710612c06f6f415f2e54a805e2bfca3f8: Status 404 returned error can't find the container with id 5b77eff8509ea3c8a0989d9e65e586a710612c06f6f415f2e54a805e2bfca3f8
Sep 18 20:04:43.520288 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:43.520253    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e6a9e26-e76d-44c3-bb37-6258c7bf8907" path="/var/lib/kubelet/pods/4e6a9e26-e76d-44c3-bb37-6258c7bf8907/volumes"
Sep 18 20:04:43.579612 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:43.579580    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5zktm" event={"ID":"61625208-077f-481b-8160-1699ab7e11fd","Type":"ContainerStarted","Data":"5b77eff8509ea3c8a0989d9e65e586a710612c06f6f415f2e54a805e2bfca3f8"}
Sep 18 20:04:45.235077 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:45.235037    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-z2rn8"]
Sep 18 20:04:45.240494 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:45.240468    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-z2rn8"
Sep 18 20:04:45.244548 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:45.244510    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-z7r4x\""
Sep 18 20:04:45.244977 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:45.244954    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Sep 18 20:04:45.260727 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:45.260692    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-z2rn8"]
Sep 18 20:04:45.356694 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:45.356654    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xzj8\" (UniqueName: \"kubernetes.io/projected/49228637-596f-49f2-a97a-646c9272793f-kube-api-access-4xzj8\") pod \"odh-model-controller-7998c67dc4-z2rn8\" (UID: \"49228637-596f-49f2-a97a-646c9272793f\") " pod="opendatahub/odh-model-controller-7998c67dc4-z2rn8"
Sep 18 20:04:45.356867 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:45.356730    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/49228637-596f-49f2-a97a-646c9272793f-cert\") pod \"odh-model-controller-7998c67dc4-z2rn8\" (UID: \"49228637-596f-49f2-a97a-646c9272793f\") " pod="opendatahub/odh-model-controller-7998c67dc4-z2rn8"
Sep 18 20:04:45.457684 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:45.457646    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xzj8\" (UniqueName: \"kubernetes.io/projected/49228637-596f-49f2-a97a-646c9272793f-kube-api-access-4xzj8\") pod \"odh-model-controller-7998c67dc4-z2rn8\" (UID: \"49228637-596f-49f2-a97a-646c9272793f\") " pod="opendatahub/odh-model-controller-7998c67dc4-z2rn8"
Sep 18 20:04:45.457856 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:45.457743    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/49228637-596f-49f2-a97a-646c9272793f-cert\") pod \"odh-model-controller-7998c67dc4-z2rn8\" (UID: \"49228637-596f-49f2-a97a-646c9272793f\") " pod="opendatahub/odh-model-controller-7998c67dc4-z2rn8"
Sep 18 20:04:45.460872 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:45.460833    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/49228637-596f-49f2-a97a-646c9272793f-cert\") pod \"odh-model-controller-7998c67dc4-z2rn8\" (UID: \"49228637-596f-49f2-a97a-646c9272793f\") " pod="opendatahub/odh-model-controller-7998c67dc4-z2rn8"
Sep 18 20:04:45.469530 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:45.469500    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xzj8\" (UniqueName: \"kubernetes.io/projected/49228637-596f-49f2-a97a-646c9272793f-kube-api-access-4xzj8\") pod \"odh-model-controller-7998c67dc4-z2rn8\" (UID: \"49228637-596f-49f2-a97a-646c9272793f\") " pod="opendatahub/odh-model-controller-7998c67dc4-z2rn8"
Sep 18 20:04:45.556615 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:45.556530    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-z2rn8"
Sep 18 20:04:45.622353 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:45.622316    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:04:45.622470 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:45.622411    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:04:45.622470 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:45.622443    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:04:45.852912 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:45.852878    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-nrr8r"]
Sep 18 20:04:45.855303 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:45.855279    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-nrr8r"
Sep 18 20:04:45.860399 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:45.860354    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Sep 18 20:04:45.870987 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:45.870902    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-nrr8r"]
Sep 18 20:04:45.873937 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:45.873915    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-z2rn8"]
Sep 18 20:04:45.881589 ip-10-0-140-178 kubenswrapper[2549]: W0918 20:04:45.881564    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49228637_596f_49f2_a97a_646c9272793f.slice/crio-cc1995c18b618d7bb547b3eb9a5bd525e1346e4477680059f8d9f7b232919b6a WatchSource:0}: Error finding container cc1995c18b618d7bb547b3eb9a5bd525e1346e4477680059f8d9f7b232919b6a: Status 404 returned error can't find the container with id cc1995c18b618d7bb547b3eb9a5bd525e1346e4477680059f8d9f7b232919b6a
Sep 18 20:04:45.960763 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:45.960729    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d199cb6d-b458-4475-86ae-5aa634c8f066-tls-certs\") pod \"model-serving-api-54d89bd5fc-nrr8r\" (UID: \"d199cb6d-b458-4475-86ae-5aa634c8f066\") " pod="opendatahub/model-serving-api-54d89bd5fc-nrr8r"
Sep 18 20:04:45.960924 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:45.960773    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtmlg\" (UniqueName: \"kubernetes.io/projected/d199cb6d-b458-4475-86ae-5aa634c8f066-kube-api-access-jtmlg\") pod \"model-serving-api-54d89bd5fc-nrr8r\" (UID: \"d199cb6d-b458-4475-86ae-5aa634c8f066\") " pod="opendatahub/model-serving-api-54d89bd5fc-nrr8r"
Sep 18 20:04:46.061798 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:46.061760    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtmlg\" (UniqueName: \"kubernetes.io/projected/d199cb6d-b458-4475-86ae-5aa634c8f066-kube-api-access-jtmlg\") pod \"model-serving-api-54d89bd5fc-nrr8r\" (UID: \"d199cb6d-b458-4475-86ae-5aa634c8f066\") " pod="opendatahub/model-serving-api-54d89bd5fc-nrr8r"
Sep 18 20:04:46.061974 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:46.061863    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d199cb6d-b458-4475-86ae-5aa634c8f066-tls-certs\") pod \"model-serving-api-54d89bd5fc-nrr8r\" (UID: \"d199cb6d-b458-4475-86ae-5aa634c8f066\") " pod="opendatahub/model-serving-api-54d89bd5fc-nrr8r"
Sep 18 20:04:46.064279 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:46.064256    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d199cb6d-b458-4475-86ae-5aa634c8f066-tls-certs\") pod \"model-serving-api-54d89bd5fc-nrr8r\" (UID: \"d199cb6d-b458-4475-86ae-5aa634c8f066\") " pod="opendatahub/model-serving-api-54d89bd5fc-nrr8r"
Sep 18 20:04:46.075433 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:46.075376    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtmlg\" (UniqueName: \"kubernetes.io/projected/d199cb6d-b458-4475-86ae-5aa634c8f066-kube-api-access-jtmlg\") pod \"model-serving-api-54d89bd5fc-nrr8r\" (UID: \"d199cb6d-b458-4475-86ae-5aa634c8f066\") " pod="opendatahub/model-serving-api-54d89bd5fc-nrr8r"
Sep 18 20:04:46.166338 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:46.166242    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-nrr8r"
Sep 18 20:04:46.438466 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:46.438442    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-nrr8r"]
Sep 18 20:04:46.440193 ip-10-0-140-178 kubenswrapper[2549]: W0918 20:04:46.440167    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd199cb6d_b458_4475_86ae_5aa634c8f066.slice/crio-5351ab7af4b39ad0403fe9a998ac6fc33dd7e85e34279b52db9f930b5a0e7811 WatchSource:0}: Error finding container 5351ab7af4b39ad0403fe9a998ac6fc33dd7e85e34279b52db9f930b5a0e7811: Status 404 returned error can't find the container with id 5351ab7af4b39ad0403fe9a998ac6fc33dd7e85e34279b52db9f930b5a0e7811
Sep 18 20:04:46.597767 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:46.597724    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-z2rn8" event={"ID":"49228637-596f-49f2-a97a-646c9272793f","Type":"ContainerStarted","Data":"cc1995c18b618d7bb547b3eb9a5bd525e1346e4477680059f8d9f7b232919b6a"}
Sep 18 20:04:46.599744 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:46.599712    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5zktm" event={"ID":"61625208-077f-481b-8160-1699ab7e11fd","Type":"ContainerStarted","Data":"e7a5aeeebd06103692b7870aedf754ae14fa7c7b0b3236b02377a4f35aed0a2d"}
Sep 18 20:04:46.602437 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:46.602412    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-nrr8r" event={"ID":"d199cb6d-b458-4475-86ae-5aa634c8f066","Type":"ContainerStarted","Data":"5351ab7af4b39ad0403fe9a998ac6fc33dd7e85e34279b52db9f930b5a0e7811"}
Sep 18 20:04:46.622646 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:46.622594    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5zktm" podStartSLOduration=1.708182332 podStartE2EDuration="4.622575984s" totalImagesPullingTime="2.914393652s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:04:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:04:46.621251886 +0000 UTC m=+471.574993057" watchObservedRunningTime="2026-09-18 20:04:46.622575984 +0000 UTC m=+471.576317155"
Sep 18 20:04:47.430803 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:47.430720    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5zktm"
Sep 18 20:04:47.437529 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:47.437160    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5zktm"
Sep 18 20:04:47.610597 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:47.610376    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5zktm"
Sep 18 20:04:47.612712 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:47.612690    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5zktm"
Sep 18 20:04:50.627992 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:50.627949    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-nrr8r" event={"ID":"d199cb6d-b458-4475-86ae-5aa634c8f066","Type":"ContainerStarted","Data":"d1566f82d7b4135d069a891852588e641a053b5a6b3dd88536d010adfa9b6e58"}
Sep 18 20:04:50.628465 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:50.628026    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-nrr8r"
Sep 18 20:04:50.629610 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:50.629582    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-z2rn8" event={"ID":"49228637-596f-49f2-a97a-646c9272793f","Type":"ContainerStarted","Data":"4b613719132a59abf08cead468e009f76e82c340318fcd581bf180199d23cfe0"}
Sep 18 20:04:50.629689 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:50.629636    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-z2rn8"
Sep 18 20:04:50.650234 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:50.650183    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-nrr8r" podStartSLOduration=2.174537221 podStartE2EDuration="5.650164661s" totalImagesPullingTime="3.47562744s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:04:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:04:50.646189937 +0000 UTC m=+475.599931107" watchObservedRunningTime="2026-09-18 20:04:50.650164661 +0000 UTC m=+475.603905830"
Sep 18 20:04:50.664784 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:50.664742    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-z2rn8" podStartSLOduration=1.630445 podStartE2EDuration="5.664726304s" totalImagesPullingTime="4.034281304s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:04:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:04:50.663129725 +0000 UTC m=+475.616870897" watchObservedRunningTime="2026-09-18 20:04:50.664726304 +0000 UTC m=+475.618467472"
Sep 18 20:04:56.895338 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:56.895307    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-rmlx2"]
Sep 18 20:04:56.898963 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:56.898938    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rmlx2"
Sep 18 20:04:56.920819 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:56.920788    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-qdnm4\""
Sep 18 20:04:56.920964 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:56.920893    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Sep 18 20:04:56.941395 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:56.941352    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-rmlx2"]
Sep 18 20:04:57.055505 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:57.055476    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/10c00388-66f0-49f6-aa95-30a676551fa3-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-rmlx2\" (UID: \"10c00388-66f0-49f6-aa95-30a676551fa3\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rmlx2"
Sep 18 20:04:57.055651 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:57.055513    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7spd7\" (UniqueName: \"kubernetes.io/projected/10c00388-66f0-49f6-aa95-30a676551fa3-kube-api-access-7spd7\") pod \"llmisvc-controller-manager-777c6bd7d5-rmlx2\" (UID: \"10c00388-66f0-49f6-aa95-30a676551fa3\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rmlx2"
Sep 18 20:04:57.156826 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:57.156739    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/10c00388-66f0-49f6-aa95-30a676551fa3-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-rmlx2\" (UID: \"10c00388-66f0-49f6-aa95-30a676551fa3\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rmlx2"
Sep 18 20:04:57.156826 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:57.156777    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7spd7\" (UniqueName: \"kubernetes.io/projected/10c00388-66f0-49f6-aa95-30a676551fa3-kube-api-access-7spd7\") pod \"llmisvc-controller-manager-777c6bd7d5-rmlx2\" (UID: \"10c00388-66f0-49f6-aa95-30a676551fa3\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rmlx2"
Sep 18 20:04:57.159232 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:57.159205    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/10c00388-66f0-49f6-aa95-30a676551fa3-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-rmlx2\" (UID: \"10c00388-66f0-49f6-aa95-30a676551fa3\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rmlx2"
Sep 18 20:04:57.195327 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:57.195294    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7spd7\" (UniqueName: \"kubernetes.io/projected/10c00388-66f0-49f6-aa95-30a676551fa3-kube-api-access-7spd7\") pod \"llmisvc-controller-manager-777c6bd7d5-rmlx2\" (UID: \"10c00388-66f0-49f6-aa95-30a676551fa3\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rmlx2"
Sep 18 20:04:57.210112 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:57.210085    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rmlx2"
Sep 18 20:04:57.483602 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:57.483575    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-rmlx2"]
Sep 18 20:04:57.485278 ip-10-0-140-178 kubenswrapper[2549]: W0918 20:04:57.485250    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod10c00388_66f0_49f6_aa95_30a676551fa3.slice/crio-8c999e726c5f8e9838c58a7abf722ddd7a8f62143b5cb633097112e42aabcb05 WatchSource:0}: Error finding container 8c999e726c5f8e9838c58a7abf722ddd7a8f62143b5cb633097112e42aabcb05: Status 404 returned error can't find the container with id 8c999e726c5f8e9838c58a7abf722ddd7a8f62143b5cb633097112e42aabcb05
Sep 18 20:04:57.663666 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:04:57.663633    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rmlx2" event={"ID":"10c00388-66f0-49f6-aa95-30a676551fa3","Type":"ContainerStarted","Data":"8c999e726c5f8e9838c58a7abf722ddd7a8f62143b5cb633097112e42aabcb05"}
Sep 18 20:05:00.682041 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:05:00.682006    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rmlx2" event={"ID":"10c00388-66f0-49f6-aa95-30a676551fa3","Type":"ContainerStarted","Data":"6145309618b46a353635a3d82fda046a737b4078f6f469f0fb025b5b02e04c36"}
Sep 18 20:05:00.682551 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:05:00.682071    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rmlx2"
Sep 18 20:05:01.636639 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:05:01.636612    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-z2rn8"
Sep 18 20:05:01.638506 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:05:01.638483    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-nrr8r"
Sep 18 20:05:01.658786 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:05:01.658738    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rmlx2" podStartSLOduration=3.216484581 podStartE2EDuration="5.658723173s" totalImagesPullingTime="2.442238592s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:04:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:05:00.745787335 +0000 UTC m=+485.699528502" watchObservedRunningTime="2026-09-18 20:05:01.658723173 +0000 UTC m=+486.612464341"
Sep 18 20:05:31.690474 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:05:31.690443    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rmlx2"
Sep 18 20:05:52.195891 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:05:52.195859    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4gqw5"]
Sep 18 20:05:52.198277 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:05:52.198259    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4gqw5"
Sep 18 20:05:52.202787 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:05:52.202766    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-zjfq9\""
Sep 18 20:05:52.202888 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:05:52.202841    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 18 20:05:52.204136 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:05:52.204115    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 18 20:05:52.214832 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:05:52.214811    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4gqw5"]
Sep 18 20:05:52.311354 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:05:52.311323    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/070e6179-6f17-414d-a036-7d5d42a1a55e-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4gqw5\" (UID: \"070e6179-6f17-414d-a036-7d5d42a1a55e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4gqw5"
Sep 18 20:05:52.311525 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:05:52.311423    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/070e6179-6f17-414d-a036-7d5d42a1a55e-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4gqw5\" (UID: \"070e6179-6f17-414d-a036-7d5d42a1a55e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4gqw5"
Sep 18 20:05:52.311525 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:05:52.311452    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldb9h\" (UniqueName: \"kubernetes.io/projected/070e6179-6f17-414d-a036-7d5d42a1a55e-kube-api-access-ldb9h\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4gqw5\" (UID: \"070e6179-6f17-414d-a036-7d5d42a1a55e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4gqw5"
Sep 18 20:05:52.412750 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:05:52.412713    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/070e6179-6f17-414d-a036-7d5d42a1a55e-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4gqw5\" (UID: \"070e6179-6f17-414d-a036-7d5d42a1a55e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4gqw5"
Sep 18 20:05:52.412925 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:05:52.412802    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/070e6179-6f17-414d-a036-7d5d42a1a55e-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4gqw5\" (UID: \"070e6179-6f17-414d-a036-7d5d42a1a55e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4gqw5"
Sep 18 20:05:52.412925 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:05:52.412830    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldb9h\" (UniqueName: \"kubernetes.io/projected/070e6179-6f17-414d-a036-7d5d42a1a55e-kube-api-access-ldb9h\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4gqw5\" (UID: \"070e6179-6f17-414d-a036-7d5d42a1a55e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4gqw5"
Sep 18 20:05:52.413148 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:05:52.413127    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/070e6179-6f17-414d-a036-7d5d42a1a55e-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4gqw5\" (UID: \"070e6179-6f17-414d-a036-7d5d42a1a55e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4gqw5"
Sep 18 20:05:52.413211 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:05:52.413144    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/070e6179-6f17-414d-a036-7d5d42a1a55e-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4gqw5\" (UID: \"070e6179-6f17-414d-a036-7d5d42a1a55e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4gqw5"
Sep 18 20:05:52.424200 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:05:52.424179    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldb9h\" (UniqueName: \"kubernetes.io/projected/070e6179-6f17-414d-a036-7d5d42a1a55e-kube-api-access-ldb9h\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4gqw5\" (UID: \"070e6179-6f17-414d-a036-7d5d42a1a55e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4gqw5"
Sep 18 20:05:52.507412 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:05:52.507317    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4gqw5"
Sep 18 20:05:52.793754 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:05:52.793731    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4gqw5"]
Sep 18 20:05:52.795620 ip-10-0-140-178 kubenswrapper[2549]: W0918 20:05:52.795592    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod070e6179_6f17_414d_a036_7d5d42a1a55e.slice/crio-6377e129a6d6136d90d0743b455b66541d8dac61e1837e5805b6f50b1332c732 WatchSource:0}: Error finding container 6377e129a6d6136d90d0743b455b66541d8dac61e1837e5805b6f50b1332c732: Status 404 returned error can't find the container with id 6377e129a6d6136d90d0743b455b66541d8dac61e1837e5805b6f50b1332c732
Sep 18 20:05:52.925277 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:05:52.925240    2549 generic.go:356] "Generic (PLEG): container finished" podID="070e6179-6f17-414d-a036-7d5d42a1a55e" containerID="b12ac9b9dde62db3c9e3fc6a8cf76f0449f50a24c60d900408d1b59c0a4a7aa1" exitCode=0
Sep 18 20:05:52.925420 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:05:52.925331    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4gqw5" event={"ID":"070e6179-6f17-414d-a036-7d5d42a1a55e","Type":"ContainerDied","Data":"b12ac9b9dde62db3c9e3fc6a8cf76f0449f50a24c60d900408d1b59c0a4a7aa1"}
Sep 18 20:05:52.925493 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:05:52.925422    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4gqw5" event={"ID":"070e6179-6f17-414d-a036-7d5d42a1a55e","Type":"ContainerStarted","Data":"6377e129a6d6136d90d0743b455b66541d8dac61e1837e5805b6f50b1332c732"}
Sep 18 20:05:53.932208 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:05:53.932175    2549 generic.go:356] "Generic (PLEG): container finished" podID="070e6179-6f17-414d-a036-7d5d42a1a55e" containerID="5a88b778c11b074d15a3454bfc88aee71fa0f63260b1704b9ca7f9b1232bf6d2" exitCode=0
Sep 18 20:05:53.932607 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:05:53.932231    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4gqw5" event={"ID":"070e6179-6f17-414d-a036-7d5d42a1a55e","Type":"ContainerDied","Data":"5a88b778c11b074d15a3454bfc88aee71fa0f63260b1704b9ca7f9b1232bf6d2"}
Sep 18 20:05:54.939503 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:05:54.939473    2549 generic.go:356] "Generic (PLEG): container finished" podID="070e6179-6f17-414d-a036-7d5d42a1a55e" containerID="00fca0a71cad0a9469f118f86e5c8f420f0e09bed01d83fc0879e2fc3aace01d" exitCode=0
Sep 18 20:05:54.939872 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:05:54.939531    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4gqw5" event={"ID":"070e6179-6f17-414d-a036-7d5d42a1a55e","Type":"ContainerDied","Data":"00fca0a71cad0a9469f118f86e5c8f420f0e09bed01d83fc0879e2fc3aace01d"}
Sep 18 20:05:56.072062 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:05:56.072038    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4gqw5"
Sep 18 20:05:56.147377 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:05:56.147345    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ldb9h\" (UniqueName: \"kubernetes.io/projected/070e6179-6f17-414d-a036-7d5d42a1a55e-kube-api-access-ldb9h\") pod \"070e6179-6f17-414d-a036-7d5d42a1a55e\" (UID: \"070e6179-6f17-414d-a036-7d5d42a1a55e\") "
Sep 18 20:05:56.147534 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:05:56.147412    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/070e6179-6f17-414d-a036-7d5d42a1a55e-bundle\") pod \"070e6179-6f17-414d-a036-7d5d42a1a55e\" (UID: \"070e6179-6f17-414d-a036-7d5d42a1a55e\") "
Sep 18 20:05:56.147534 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:05:56.147480    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/070e6179-6f17-414d-a036-7d5d42a1a55e-util\") pod \"070e6179-6f17-414d-a036-7d5d42a1a55e\" (UID: \"070e6179-6f17-414d-a036-7d5d42a1a55e\") "
Sep 18 20:05:56.147922 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:05:56.147897    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/070e6179-6f17-414d-a036-7d5d42a1a55e-bundle" (OuterVolumeSpecName: "bundle") pod "070e6179-6f17-414d-a036-7d5d42a1a55e" (UID: "070e6179-6f17-414d-a036-7d5d42a1a55e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:05:56.149461 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:05:56.149433    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/070e6179-6f17-414d-a036-7d5d42a1a55e-kube-api-access-ldb9h" (OuterVolumeSpecName: "kube-api-access-ldb9h") pod "070e6179-6f17-414d-a036-7d5d42a1a55e" (UID: "070e6179-6f17-414d-a036-7d5d42a1a55e"). InnerVolumeSpecName "kube-api-access-ldb9h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:05:56.153027 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:05:56.153004    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/070e6179-6f17-414d-a036-7d5d42a1a55e-util" (OuterVolumeSpecName: "util") pod "070e6179-6f17-414d-a036-7d5d42a1a55e" (UID: "070e6179-6f17-414d-a036-7d5d42a1a55e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:05:56.248872 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:05:56.248784    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldb9h\" (UniqueName: \"kubernetes.io/projected/070e6179-6f17-414d-a036-7d5d42a1a55e-kube-api-access-ldb9h\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:05:56.248872 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:05:56.248818    2549 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/070e6179-6f17-414d-a036-7d5d42a1a55e-bundle\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:05:56.248872 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:05:56.248831    2549 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/070e6179-6f17-414d-a036-7d5d42a1a55e-util\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:05:56.950098 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:05:56.950056    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4gqw5" event={"ID":"070e6179-6f17-414d-a036-7d5d42a1a55e","Type":"ContainerDied","Data":"6377e129a6d6136d90d0743b455b66541d8dac61e1837e5805b6f50b1332c732"}
Sep 18 20:05:56.950098 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:05:56.950092    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4gqw5"
Sep 18 20:05:56.950305 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:05:56.950113    2549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6377e129a6d6136d90d0743b455b66541d8dac61e1837e5805b6f50b1332c732"
Sep 18 20:06:08.313704 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:08.313670    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-v7sqd"]
Sep 18 20:06:08.314276 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:08.314235    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="070e6179-6f17-414d-a036-7d5d42a1a55e" containerName="util"
Sep 18 20:06:08.314276 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:08.314255    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="070e6179-6f17-414d-a036-7d5d42a1a55e" containerName="util"
Sep 18 20:06:08.314276 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:08.314271    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="070e6179-6f17-414d-a036-7d5d42a1a55e" containerName="pull"
Sep 18 20:06:08.314471 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:08.314280    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="070e6179-6f17-414d-a036-7d5d42a1a55e" containerName="pull"
Sep 18 20:06:08.314471 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:08.314301    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="070e6179-6f17-414d-a036-7d5d42a1a55e" containerName="extract"
Sep 18 20:06:08.314471 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:08.314309    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="070e6179-6f17-414d-a036-7d5d42a1a55e" containerName="extract"
Sep 18 20:06:08.314471 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:08.314421    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="070e6179-6f17-414d-a036-7d5d42a1a55e" containerName="extract"
Sep 18 20:06:08.316630 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:08.316612    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-v7sqd"
Sep 18 20:06:08.322800 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:08.322778    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-7hrjh\""
Sep 18 20:06:08.323169 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:08.323146    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 18 20:06:08.323277 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:08.323220    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 18 20:06:08.341272 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:08.341229    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-v7sqd"]
Sep 18 20:06:08.341477 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:08.341443    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 18 20:06:08.446961 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:08.446929    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6x48\" (UniqueName: \"kubernetes.io/projected/a89f83d9-2e6d-4f5f-ae28-f8e093121094-kube-api-access-k6x48\") pod \"dns-operator-controller-manager-7b84cd7c9c-v7sqd\" (UID: \"a89f83d9-2e6d-4f5f-ae28-f8e093121094\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-v7sqd"
Sep 18 20:06:08.548044 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:08.548011    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6x48\" (UniqueName: \"kubernetes.io/projected/a89f83d9-2e6d-4f5f-ae28-f8e093121094-kube-api-access-k6x48\") pod \"dns-operator-controller-manager-7b84cd7c9c-v7sqd\" (UID: \"a89f83d9-2e6d-4f5f-ae28-f8e093121094\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-v7sqd"
Sep 18 20:06:08.584480 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:08.584452    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6x48\" (UniqueName: \"kubernetes.io/projected/a89f83d9-2e6d-4f5f-ae28-f8e093121094-kube-api-access-k6x48\") pod \"dns-operator-controller-manager-7b84cd7c9c-v7sqd\" (UID: \"a89f83d9-2e6d-4f5f-ae28-f8e093121094\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-v7sqd"
Sep 18 20:06:08.629263 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:08.629236    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-v7sqd"
Sep 18 20:06:09.114902 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:09.114880    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-v7sqd"]
Sep 18 20:06:09.117152 ip-10-0-140-178 kubenswrapper[2549]: W0918 20:06:09.117122    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda89f83d9_2e6d_4f5f_ae28_f8e093121094.slice/crio-6ffb2e59ae9f32f49e2e69de3f1c552343022b4d70868408df39f67f7d111a95 WatchSource:0}: Error finding container 6ffb2e59ae9f32f49e2e69de3f1c552343022b4d70868408df39f67f7d111a95: Status 404 returned error can't find the container with id 6ffb2e59ae9f32f49e2e69de3f1c552343022b4d70868408df39f67f7d111a95
Sep 18 20:06:10.015203 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:10.015168    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-v7sqd" event={"ID":"a89f83d9-2e6d-4f5f-ae28-f8e093121094","Type":"ContainerStarted","Data":"6ffb2e59ae9f32f49e2e69de3f1c552343022b4d70868408df39f67f7d111a95"}
Sep 18 20:06:13.279212 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:13.279177    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-r4b8p"]
Sep 18 20:06:13.282326 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:13.282309    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-r4b8p"
Sep 18 20:06:13.285566 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:13.285547    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-x7vvt\""
Sep 18 20:06:13.299601 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:13.299576    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-r4b8p"]
Sep 18 20:06:13.387612 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:13.387579    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sv7mb\" (UniqueName: \"kubernetes.io/projected/c2d46052-b16a-4346-ba8c-d8601bbb5cc5-kube-api-access-sv7mb\") pod \"kuadrant-operator-controller-manager-76f8f686bf-r4b8p\" (UID: \"c2d46052-b16a-4346-ba8c-d8601bbb5cc5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-r4b8p"
Sep 18 20:06:13.387768 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:13.387625    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c2d46052-b16a-4346-ba8c-d8601bbb5cc5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-r4b8p\" (UID: \"c2d46052-b16a-4346-ba8c-d8601bbb5cc5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-r4b8p"
Sep 18 20:06:13.489598 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:13.488699    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c2d46052-b16a-4346-ba8c-d8601bbb5cc5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-r4b8p\" (UID: \"c2d46052-b16a-4346-ba8c-d8601bbb5cc5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-r4b8p"
Sep 18 20:06:13.489598 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:13.488861    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv7mb\" (UniqueName: \"kubernetes.io/projected/c2d46052-b16a-4346-ba8c-d8601bbb5cc5-kube-api-access-sv7mb\") pod \"kuadrant-operator-controller-manager-76f8f686bf-r4b8p\" (UID: \"c2d46052-b16a-4346-ba8c-d8601bbb5cc5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-r4b8p"
Sep 18 20:06:13.490409 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:13.489879    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c2d46052-b16a-4346-ba8c-d8601bbb5cc5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-r4b8p\" (UID: \"c2d46052-b16a-4346-ba8c-d8601bbb5cc5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-r4b8p"
Sep 18 20:06:13.500698 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:13.500671    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv7mb\" (UniqueName: \"kubernetes.io/projected/c2d46052-b16a-4346-ba8c-d8601bbb5cc5-kube-api-access-sv7mb\") pod \"kuadrant-operator-controller-manager-76f8f686bf-r4b8p\" (UID: \"c2d46052-b16a-4346-ba8c-d8601bbb5cc5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-r4b8p"
Sep 18 20:06:13.593244 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:13.593221    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-r4b8p"
Sep 18 20:06:13.879605 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:13.879581    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-r4b8p"]
Sep 18 20:06:13.882202 ip-10-0-140-178 kubenswrapper[2549]: W0918 20:06:13.882174    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2d46052_b16a_4346_ba8c_d8601bbb5cc5.slice/crio-9921bc8ad881f062f8cab398b67f02b14634d086cea8d21c4832ff208afa3b9f WatchSource:0}: Error finding container 9921bc8ad881f062f8cab398b67f02b14634d086cea8d21c4832ff208afa3b9f: Status 404 returned error can't find the container with id 9921bc8ad881f062f8cab398b67f02b14634d086cea8d21c4832ff208afa3b9f
Sep 18 20:06:14.036002 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:14.035964    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-r4b8p" event={"ID":"c2d46052-b16a-4346-ba8c-d8601bbb5cc5","Type":"ContainerStarted","Data":"9921bc8ad881f062f8cab398b67f02b14634d086cea8d21c4832ff208afa3b9f"}
Sep 18 20:06:18.063343 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:18.063297    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-r4b8p" event={"ID":"c2d46052-b16a-4346-ba8c-d8601bbb5cc5","Type":"ContainerStarted","Data":"595462667ca8956e2e3e0faadb8cf7bfdafa196fd59c1b340bdb48a7d1660cea"}
Sep 18 20:06:18.063797 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:18.063475    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-r4b8p"
Sep 18 20:06:18.105772 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:18.105724    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-r4b8p" podStartSLOduration=1.444028053 podStartE2EDuration="5.105707406s" totalImagesPullingTime="3.661679353s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:06:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:06:18.103879062 +0000 UTC m=+563.057620232" watchObservedRunningTime="2026-09-18 20:06:18.105707406 +0000 UTC m=+563.059448574"
Sep 18 20:06:19.069578 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:19.069535    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-v7sqd" event={"ID":"a89f83d9-2e6d-4f5f-ae28-f8e093121094","Type":"ContainerStarted","Data":"cd496dbc4fd20436bfbd9a7a767bde34ec996a4d9b170be5fd4145ba40051da3"}
Sep 18 20:06:19.094059 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:19.094010    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-v7sqd" podStartSLOduration=1.8575378310000001 podStartE2EDuration="11.09399598s" totalImagesPullingTime="9.236458149s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:06:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:06:19.090961167 +0000 UTC m=+564.044702335" watchObservedRunningTime="2026-09-18 20:06:19.09399598 +0000 UTC m=+564.047737148"
Sep 18 20:06:20.074652 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:20.074621    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-v7sqd"
Sep 18 20:06:20.486480 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:20.486374    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-bbxwx"]
Sep 18 20:06:20.489637 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:20.489609    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bbxwx"
Sep 18 20:06:20.492823 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:20.492802    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 18 20:06:20.492823 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:20.492812    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 18 20:06:20.493479 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:20.493457    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-4qclr\""
Sep 18 20:06:20.499348 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:20.499325    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-bbxwx"]
Sep 18 20:06:20.553178 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:20.553146    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7904c124-a678-4cf1-809a-fd67593611b0-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-bbxwx\" (UID: \"7904c124-a678-4cf1-809a-fd67593611b0\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bbxwx"
Sep 18 20:06:20.553348 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:20.553202    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7904c124-a678-4cf1-809a-fd67593611b0-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-bbxwx\" (UID: \"7904c124-a678-4cf1-809a-fd67593611b0\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bbxwx"
Sep 18 20:06:20.553348 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:20.553271    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65mwg\" (UniqueName: \"kubernetes.io/projected/7904c124-a678-4cf1-809a-fd67593611b0-kube-api-access-65mwg\") pod \"kuadrant-console-plugin-fbdd4f445-bbxwx\" (UID: \"7904c124-a678-4cf1-809a-fd67593611b0\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bbxwx"
Sep 18 20:06:20.653913 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:20.653881    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7904c124-a678-4cf1-809a-fd67593611b0-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-bbxwx\" (UID: \"7904c124-a678-4cf1-809a-fd67593611b0\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bbxwx"
Sep 18 20:06:20.654102 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:20.653941    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65mwg\" (UniqueName: \"kubernetes.io/projected/7904c124-a678-4cf1-809a-fd67593611b0-kube-api-access-65mwg\") pod \"kuadrant-console-plugin-fbdd4f445-bbxwx\" (UID: \"7904c124-a678-4cf1-809a-fd67593611b0\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bbxwx"
Sep 18 20:06:20.654102 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:20.653997    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7904c124-a678-4cf1-809a-fd67593611b0-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-bbxwx\" (UID: \"7904c124-a678-4cf1-809a-fd67593611b0\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bbxwx"
Sep 18 20:06:20.654720 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:20.654689    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7904c124-a678-4cf1-809a-fd67593611b0-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-bbxwx\" (UID: \"7904c124-a678-4cf1-809a-fd67593611b0\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bbxwx"
Sep 18 20:06:20.656590 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:20.656566    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7904c124-a678-4cf1-809a-fd67593611b0-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-bbxwx\" (UID: \"7904c124-a678-4cf1-809a-fd67593611b0\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bbxwx"
Sep 18 20:06:20.665156 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:20.665133    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65mwg\" (UniqueName: \"kubernetes.io/projected/7904c124-a678-4cf1-809a-fd67593611b0-kube-api-access-65mwg\") pod \"kuadrant-console-plugin-fbdd4f445-bbxwx\" (UID: \"7904c124-a678-4cf1-809a-fd67593611b0\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bbxwx"
Sep 18 20:06:20.800161 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:20.800085    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bbxwx"
Sep 18 20:06:21.077409 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:21.077369    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-bbxwx"]
Sep 18 20:06:21.078918 ip-10-0-140-178 kubenswrapper[2549]: W0918 20:06:21.078894    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7904c124_a678_4cf1_809a_fd67593611b0.slice/crio-fa31bbc9a7b3bc07c0ec785a5404d3ad473cdeea04937f51ae1972656b36d66c WatchSource:0}: Error finding container fa31bbc9a7b3bc07c0ec785a5404d3ad473cdeea04937f51ae1972656b36d66c: Status 404 returned error can't find the container with id fa31bbc9a7b3bc07c0ec785a5404d3ad473cdeea04937f51ae1972656b36d66c
Sep 18 20:06:22.089298 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:22.089236    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bbxwx" event={"ID":"7904c124-a678-4cf1-809a-fd67593611b0","Type":"ContainerStarted","Data":"fa31bbc9a7b3bc07c0ec785a5404d3ad473cdeea04937f51ae1972656b36d66c"}
Sep 18 20:06:26.113733 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:26.113700    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bbxwx" event={"ID":"7904c124-a678-4cf1-809a-fd67593611b0","Type":"ContainerStarted","Data":"20b0c96e94cc5ee73b254f66271b22e54611092aac64b90797da843ed54947a9"}
Sep 18 20:06:26.139313 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:26.139266    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bbxwx" podStartSLOduration=1.6818133309999999 podStartE2EDuration="6.139250993s" totalImagesPullingTime="4.457437662s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:06:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:06:26.136026909 +0000 UTC m=+571.089768077" watchObservedRunningTime="2026-09-18 20:06:26.139250993 +0000 UTC m=+571.092992160"
Sep 18 20:06:29.072244 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:29.072206    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-r4b8p"
Sep 18 20:06:31.082484 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:31.082454    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-v7sqd"
Sep 18 20:06:31.406565 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:31.406480    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-r4b8p"]
Sep 18 20:06:31.406711 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:31.406688    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-r4b8p" podUID="c2d46052-b16a-4346-ba8c-d8601bbb5cc5" containerName="manager" containerID="cri-o://595462667ca8956e2e3e0faadb8cf7bfdafa196fd59c1b340bdb48a7d1660cea" gracePeriod=2
Sep 18 20:06:31.416728 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:31.416700    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-r4b8p"]
Sep 18 20:06:31.442675 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:31.442647    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gwfpk"]
Sep 18 20:06:31.443006 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:31.442994    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2d46052-b16a-4346-ba8c-d8601bbb5cc5" containerName="manager"
Sep 18 20:06:31.443006 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:31.443007    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2d46052-b16a-4346-ba8c-d8601bbb5cc5" containerName="manager"
Sep 18 20:06:31.443090 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:31.443066    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2d46052-b16a-4346-ba8c-d8601bbb5cc5" containerName="manager"
Sep 18 20:06:31.446436 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:31.446413    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gwfpk"
Sep 18 20:06:31.451358 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:06:31.451331    2549 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-r4b8p\" is forbidden: User \"system:node:ip-10-0-140-178.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-178.ec2.internal' and this object" podUID="c2d46052-b16a-4346-ba8c-d8601bbb5cc5" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-r4b8p"
Sep 18 20:06:31.468164 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:31.468135    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gwfpk"]
Sep 18 20:06:31.550033 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:31.550003    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5lgg\" (UniqueName: \"kubernetes.io/projected/2e848b7b-03e2-423b-ab89-92c0a5b22782-kube-api-access-q5lgg\") pod \"kuadrant-operator-controller-manager-76f8f686bf-gwfpk\" (UID: \"2e848b7b-03e2-423b-ab89-92c0a5b22782\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gwfpk"
Sep 18 20:06:31.550194 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:31.550064    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2e848b7b-03e2-423b-ab89-92c0a5b22782-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-gwfpk\" (UID: \"2e848b7b-03e2-423b-ab89-92c0a5b22782\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gwfpk"
Sep 18 20:06:31.651005 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:31.650980    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5lgg\" (UniqueName: \"kubernetes.io/projected/2e848b7b-03e2-423b-ab89-92c0a5b22782-kube-api-access-q5lgg\") pod \"kuadrant-operator-controller-manager-76f8f686bf-gwfpk\" (UID: \"2e848b7b-03e2-423b-ab89-92c0a5b22782\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gwfpk"
Sep 18 20:06:31.651127 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:31.651040    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2e848b7b-03e2-423b-ab89-92c0a5b22782-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-gwfpk\" (UID: \"2e848b7b-03e2-423b-ab89-92c0a5b22782\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gwfpk"
Sep 18 20:06:31.651368 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:31.651349    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2e848b7b-03e2-423b-ab89-92c0a5b22782-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-gwfpk\" (UID: \"2e848b7b-03e2-423b-ab89-92c0a5b22782\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gwfpk"
Sep 18 20:06:31.663511 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:31.663459    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-r4b8p"
Sep 18 20:06:31.677371 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:31.677343    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5lgg\" (UniqueName: \"kubernetes.io/projected/2e848b7b-03e2-423b-ab89-92c0a5b22782-kube-api-access-q5lgg\") pod \"kuadrant-operator-controller-manager-76f8f686bf-gwfpk\" (UID: \"2e848b7b-03e2-423b-ab89-92c0a5b22782\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gwfpk"
Sep 18 20:06:31.752188 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:31.752161    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sv7mb\" (UniqueName: \"kubernetes.io/projected/c2d46052-b16a-4346-ba8c-d8601bbb5cc5-kube-api-access-sv7mb\") pod \"c2d46052-b16a-4346-ba8c-d8601bbb5cc5\" (UID: \"c2d46052-b16a-4346-ba8c-d8601bbb5cc5\") "
Sep 18 20:06:31.752307 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:31.752208    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c2d46052-b16a-4346-ba8c-d8601bbb5cc5-extensions-socket-volume\") pod \"c2d46052-b16a-4346-ba8c-d8601bbb5cc5\" (UID: \"c2d46052-b16a-4346-ba8c-d8601bbb5cc5\") "
Sep 18 20:06:31.752701 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:31.752678    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2d46052-b16a-4346-ba8c-d8601bbb5cc5-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "c2d46052-b16a-4346-ba8c-d8601bbb5cc5" (UID: "c2d46052-b16a-4346-ba8c-d8601bbb5cc5"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:06:31.754275 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:31.754254    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2d46052-b16a-4346-ba8c-d8601bbb5cc5-kube-api-access-sv7mb" (OuterVolumeSpecName: "kube-api-access-sv7mb") pod "c2d46052-b16a-4346-ba8c-d8601bbb5cc5" (UID: "c2d46052-b16a-4346-ba8c-d8601bbb5cc5"). InnerVolumeSpecName "kube-api-access-sv7mb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:06:31.782563 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:31.782539    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gwfpk"
Sep 18 20:06:31.852932 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:31.852905    2549 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c2d46052-b16a-4346-ba8c-d8601bbb5cc5-extensions-socket-volume\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:06:31.852932 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:31.852928    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sv7mb\" (UniqueName: \"kubernetes.io/projected/c2d46052-b16a-4346-ba8c-d8601bbb5cc5-kube-api-access-sv7mb\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:06:32.103325 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:32.103299    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gwfpk"]
Sep 18 20:06:32.105200 ip-10-0-140-178 kubenswrapper[2549]: W0918 20:06:32.105171    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e848b7b_03e2_423b_ab89_92c0a5b22782.slice/crio-460ce989d60fcb8e838265572c679fde9f0c24413c289ddd2de5ed5e4dee4cc7 WatchSource:0}: Error finding container 460ce989d60fcb8e838265572c679fde9f0c24413c289ddd2de5ed5e4dee4cc7: Status 404 returned error can't find the container with id 460ce989d60fcb8e838265572c679fde9f0c24413c289ddd2de5ed5e4dee4cc7
Sep 18 20:06:32.146327 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:32.146301    2549 generic.go:356] "Generic (PLEG): container finished" podID="c2d46052-b16a-4346-ba8c-d8601bbb5cc5" containerID="595462667ca8956e2e3e0faadb8cf7bfdafa196fd59c1b340bdb48a7d1660cea" exitCode=0
Sep 18 20:06:32.146457 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:32.146355    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-r4b8p"
Sep 18 20:06:32.146457 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:32.146361    2549 scope.go:117] "RemoveContainer" containerID="595462667ca8956e2e3e0faadb8cf7bfdafa196fd59c1b340bdb48a7d1660cea"
Sep 18 20:06:32.147984 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:32.147961    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gwfpk" event={"ID":"2e848b7b-03e2-423b-ab89-92c0a5b22782","Type":"ContainerStarted","Data":"460ce989d60fcb8e838265572c679fde9f0c24413c289ddd2de5ed5e4dee4cc7"}
Sep 18 20:06:32.164621 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:32.164599    2549 scope.go:117] "RemoveContainer" containerID="595462667ca8956e2e3e0faadb8cf7bfdafa196fd59c1b340bdb48a7d1660cea"
Sep 18 20:06:32.164893 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:06:32.164875    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"595462667ca8956e2e3e0faadb8cf7bfdafa196fd59c1b340bdb48a7d1660cea\": container with ID starting with 595462667ca8956e2e3e0faadb8cf7bfdafa196fd59c1b340bdb48a7d1660cea not found: ID does not exist" containerID="595462667ca8956e2e3e0faadb8cf7bfdafa196fd59c1b340bdb48a7d1660cea"
Sep 18 20:06:32.164975 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:32.164901    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"595462667ca8956e2e3e0faadb8cf7bfdafa196fd59c1b340bdb48a7d1660cea"} err="failed to get container status \"595462667ca8956e2e3e0faadb8cf7bfdafa196fd59c1b340bdb48a7d1660cea\": rpc error: code = NotFound desc = could not find container \"595462667ca8956e2e3e0faadb8cf7bfdafa196fd59c1b340bdb48a7d1660cea\": container with ID starting with 595462667ca8956e2e3e0faadb8cf7bfdafa196fd59c1b340bdb48a7d1660cea not found: ID does not exist"
Sep 18 20:06:33.156748 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:33.156706    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gwfpk" event={"ID":"2e848b7b-03e2-423b-ab89-92c0a5b22782","Type":"ContainerStarted","Data":"6797808ab43f940aef816bbc8a588b919a1e004393b86df533f324442b93fefd"}
Sep 18 20:06:33.157189 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:33.156775    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gwfpk"
Sep 18 20:06:33.184952 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:33.184908    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gwfpk" podStartSLOduration=2.1848912 podStartE2EDuration="2.1848912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:06:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:06:33.182424445 +0000 UTC m=+578.136165613" watchObservedRunningTime="2026-09-18 20:06:33.1848912 +0000 UTC m=+578.138632367"
Sep 18 20:06:33.293919 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:33.293885    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9kdnl"]
Sep 18 20:06:33.297411 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:33.297378    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9kdnl"
Sep 18 20:06:33.314489 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:33.314465    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9kdnl"]
Sep 18 20:06:33.366128 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:33.366099    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2nkc\" (UniqueName: \"kubernetes.io/projected/4817b2de-b55e-4ddd-8c8a-4eb953490a17-kube-api-access-h2nkc\") pod \"kuadrant-operator-controller-manager-5769f99d9f-9kdnl\" (UID: \"4817b2de-b55e-4ddd-8c8a-4eb953490a17\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9kdnl"
Sep 18 20:06:33.366247 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:33.366152    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4817b2de-b55e-4ddd-8c8a-4eb953490a17-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-9kdnl\" (UID: \"4817b2de-b55e-4ddd-8c8a-4eb953490a17\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9kdnl"
Sep 18 20:06:33.467799 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:33.467725    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2nkc\" (UniqueName: \"kubernetes.io/projected/4817b2de-b55e-4ddd-8c8a-4eb953490a17-kube-api-access-h2nkc\") pod \"kuadrant-operator-controller-manager-5769f99d9f-9kdnl\" (UID: \"4817b2de-b55e-4ddd-8c8a-4eb953490a17\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9kdnl"
Sep 18 20:06:33.467921 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:33.467803    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4817b2de-b55e-4ddd-8c8a-4eb953490a17-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-9kdnl\" (UID: \"4817b2de-b55e-4ddd-8c8a-4eb953490a17\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9kdnl"
Sep 18 20:06:33.468151 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:33.468134    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4817b2de-b55e-4ddd-8c8a-4eb953490a17-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-9kdnl\" (UID: \"4817b2de-b55e-4ddd-8c8a-4eb953490a17\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9kdnl"
Sep 18 20:06:33.485102 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:33.485078    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2nkc\" (UniqueName: \"kubernetes.io/projected/4817b2de-b55e-4ddd-8c8a-4eb953490a17-kube-api-access-h2nkc\") pod \"kuadrant-operator-controller-manager-5769f99d9f-9kdnl\" (UID: \"4817b2de-b55e-4ddd-8c8a-4eb953490a17\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9kdnl"
Sep 18 20:06:33.526463 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:33.526434    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2d46052-b16a-4346-ba8c-d8601bbb5cc5" path="/var/lib/kubelet/pods/c2d46052-b16a-4346-ba8c-d8601bbb5cc5/volumes"
Sep 18 20:06:33.607666 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:33.607643    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9kdnl"
Sep 18 20:06:33.888411 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:33.888369    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9kdnl"]
Sep 18 20:06:33.889055 ip-10-0-140-178 kubenswrapper[2549]: W0918 20:06:33.889034    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4817b2de_b55e_4ddd_8c8a_4eb953490a17.slice/crio-fb918bd342d657d1912b9020ec08dd25df3a8c304fe5972c9d7a097c076a9a7f WatchSource:0}: Error finding container fb918bd342d657d1912b9020ec08dd25df3a8c304fe5972c9d7a097c076a9a7f: Status 404 returned error can't find the container with id fb918bd342d657d1912b9020ec08dd25df3a8c304fe5972c9d7a097c076a9a7f
Sep 18 20:06:34.164261 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:34.164169    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9kdnl" event={"ID":"4817b2de-b55e-4ddd-8c8a-4eb953490a17","Type":"ContainerStarted","Data":"9d03c612748e55076c53d7da80b95a977bed8b788839cdf0244a6576a30d63e4"}
Sep 18 20:06:34.164261 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:34.164227    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9kdnl" event={"ID":"4817b2de-b55e-4ddd-8c8a-4eb953490a17","Type":"ContainerStarted","Data":"fb918bd342d657d1912b9020ec08dd25df3a8c304fe5972c9d7a097c076a9a7f"}
Sep 18 20:06:34.164261 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:34.164253    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9kdnl"
Sep 18 20:06:34.192709 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:34.192663    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9kdnl" podStartSLOduration=1.192647345 podStartE2EDuration="1.192647345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:06:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:06:34.189450786 +0000 UTC m=+579.143191955" watchObservedRunningTime="2026-09-18 20:06:34.192647345 +0000 UTC m=+579.146388512"
Sep 18 20:06:44.167144 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:44.167109    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gwfpk"
Sep 18 20:06:45.171822 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:45.171788    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9kdnl"
Sep 18 20:06:45.263405 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:45.263295    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gwfpk"]
Sep 18 20:06:45.266560 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:45.263826    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gwfpk" podUID="2e848b7b-03e2-423b-ab89-92c0a5b22782" containerName="manager" containerID="cri-o://6797808ab43f940aef816bbc8a588b919a1e004393b86df533f324442b93fefd" gracePeriod=10
Sep 18 20:06:45.513027 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:45.513004    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gwfpk"
Sep 18 20:06:45.570005 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:45.569980    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2e848b7b-03e2-423b-ab89-92c0a5b22782-extensions-socket-volume\") pod \"2e848b7b-03e2-423b-ab89-92c0a5b22782\" (UID: \"2e848b7b-03e2-423b-ab89-92c0a5b22782\") "
Sep 18 20:06:45.570159 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:45.570073    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q5lgg\" (UniqueName: \"kubernetes.io/projected/2e848b7b-03e2-423b-ab89-92c0a5b22782-kube-api-access-q5lgg\") pod \"2e848b7b-03e2-423b-ab89-92c0a5b22782\" (UID: \"2e848b7b-03e2-423b-ab89-92c0a5b22782\") "
Sep 18 20:06:45.570403 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:45.570367    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e848b7b-03e2-423b-ab89-92c0a5b22782-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "2e848b7b-03e2-423b-ab89-92c0a5b22782" (UID: "2e848b7b-03e2-423b-ab89-92c0a5b22782"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:06:45.572558 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:45.572526    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e848b7b-03e2-423b-ab89-92c0a5b22782-kube-api-access-q5lgg" (OuterVolumeSpecName: "kube-api-access-q5lgg") pod "2e848b7b-03e2-423b-ab89-92c0a5b22782" (UID: "2e848b7b-03e2-423b-ab89-92c0a5b22782"). InnerVolumeSpecName "kube-api-access-q5lgg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:06:45.671509 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:45.671483    2549 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2e848b7b-03e2-423b-ab89-92c0a5b22782-extensions-socket-volume\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:06:45.671509 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:45.671506    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5lgg\" (UniqueName: \"kubernetes.io/projected/2e848b7b-03e2-423b-ab89-92c0a5b22782-kube-api-access-q5lgg\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:06:46.226866 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:46.226831    2549 generic.go:356] "Generic (PLEG): container finished" podID="2e848b7b-03e2-423b-ab89-92c0a5b22782" containerID="6797808ab43f940aef816bbc8a588b919a1e004393b86df533f324442b93fefd" exitCode=0
Sep 18 20:06:46.227253 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:46.226875    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gwfpk"
Sep 18 20:06:46.227253 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:46.226917    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gwfpk" event={"ID":"2e848b7b-03e2-423b-ab89-92c0a5b22782","Type":"ContainerDied","Data":"6797808ab43f940aef816bbc8a588b919a1e004393b86df533f324442b93fefd"}
Sep 18 20:06:46.227253 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:46.226965    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gwfpk" event={"ID":"2e848b7b-03e2-423b-ab89-92c0a5b22782","Type":"ContainerDied","Data":"460ce989d60fcb8e838265572c679fde9f0c24413c289ddd2de5ed5e4dee4cc7"}
Sep 18 20:06:46.227253 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:46.226985    2549 scope.go:117] "RemoveContainer" containerID="6797808ab43f940aef816bbc8a588b919a1e004393b86df533f324442b93fefd"
Sep 18 20:06:46.236112 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:46.236094    2549 scope.go:117] "RemoveContainer" containerID="6797808ab43f940aef816bbc8a588b919a1e004393b86df533f324442b93fefd"
Sep 18 20:06:46.236371 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:06:46.236356    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6797808ab43f940aef816bbc8a588b919a1e004393b86df533f324442b93fefd\": container with ID starting with 6797808ab43f940aef816bbc8a588b919a1e004393b86df533f324442b93fefd not found: ID does not exist" containerID="6797808ab43f940aef816bbc8a588b919a1e004393b86df533f324442b93fefd"
Sep 18 20:06:46.236452 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:46.236393    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6797808ab43f940aef816bbc8a588b919a1e004393b86df533f324442b93fefd"} err="failed to get container status \"6797808ab43f940aef816bbc8a588b919a1e004393b86df533f324442b93fefd\": rpc error: code = NotFound desc = could not find container \"6797808ab43f940aef816bbc8a588b919a1e004393b86df533f324442b93fefd\": container with ID starting with 6797808ab43f940aef816bbc8a588b919a1e004393b86df533f324442b93fefd not found: ID does not exist"
Sep 18 20:06:46.253364 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:46.253341    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gwfpk"]
Sep 18 20:06:46.260553 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:46.260533    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gwfpk"]
Sep 18 20:06:47.520708 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:47.520677    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e848b7b-03e2-423b-ab89-92c0a5b22782" path="/var/lib/kubelet/pods/2e848b7b-03e2-423b-ab89-92c0a5b22782/volumes"
Sep 18 20:06:55.438749 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:55.438720    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-scn7p_a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48/ovn-acl-logging/0.log"
Sep 18 20:06:55.440595 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:06:55.440575    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-scn7p_a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48/ovn-acl-logging/0.log"
Sep 18 20:07:01.672700 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:07:01.672668    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thwfx"]
Sep 18 20:07:01.674032 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:07:01.673018    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e848b7b-03e2-423b-ab89-92c0a5b22782" containerName="manager"
Sep 18 20:07:01.674032 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:07:01.673029    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e848b7b-03e2-423b-ab89-92c0a5b22782" containerName="manager"
Sep 18 20:07:01.674032 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:07:01.673090    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e848b7b-03e2-423b-ab89-92c0a5b22782" containerName="manager"
Sep 18 20:07:01.674893 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:07:01.674875    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thwfx"
Sep 18 20:07:01.679561 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:07:01.679537    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-pmmz6\""
Sep 18 20:07:01.694169 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:07:01.694143    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thwfx"]
Sep 18 20:07:01.809424 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:07:01.809376    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3193c781-85d8-48b0-956b-d6d3b0b31fd8-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-thwfx\" (UID: \"3193c781-85d8-48b0-956b-d6d3b0b31fd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thwfx"
Sep 18 20:07:01.809424 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:07:01.809424    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3193c781-85d8-48b0-956b-d6d3b0b31fd8-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-thwfx\" (UID: \"3193c781-85d8-48b0-956b-d6d3b0b31fd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thwfx"
Sep 18 20:07:01.809617 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:07:01.809519    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3193c781-85d8-48b0-956b-d6d3b0b31fd8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-thwfx\" (UID: \"3193c781-85d8-48b0-956b-d6d3b0b31fd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thwfx"
Sep 18 20:07:01.809617 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:07:01.809552    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgwsg\" (UniqueName: \"kubernetes.io/projected/3193c781-85d8-48b0-956b-d6d3b0b31fd8-kube-api-access-hgwsg\") pod \"maas-default-gateway-openshift-default-78bb78ff78-thwfx\" (UID: \"3193c781-85d8-48b0-956b-d6d3b0b31fd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thwfx"
Sep 18 20:07:01.809617 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:07:01.809584    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3193c781-85d8-48b0-956b-d6d3b0b31fd8-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-thwfx\" (UID: \"3193c781-85d8-48b0-956b-d6d3b0b31fd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thwfx"
Sep 18 20:07:01.809714 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:07:01.809630    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3193c781-85d8-48b0-956b-d6d3b0b31fd8-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-thwfx\" (UID: \"3193c781-85d8-48b0-956b-d6d3b0b31fd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thwfx"
Sep 18 20:07:01.809714 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:07:01.809662    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3193c781-85d8-48b0-956b-d6d3b0b31fd8-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-thwfx\" (UID: \"3193c781-85d8-48b0-956b-d6d3b0b31fd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thwfx"
Sep 18 20:07:01.809776 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:07:01.809708    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3193c781-85d8-48b0-956b-d6d3b0b31fd8-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-thwfx\" (UID: \"3193c781-85d8-48b0-956b-d6d3b0b31fd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thwfx"
Sep 18 20:07:01.809776 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:07:01.809734    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3193c781-85d8-48b0-956b-d6d3b0b31fd8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-thwfx\" (UID: \"3193c781-85d8-48b0-956b-d6d3b0b31fd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thwfx"
Sep 18 20:07:01.910976 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:07:01.910937    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3193c781-85d8-48b0-956b-d6d3b0b31fd8-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-thwfx\" (UID: \"3193c781-85d8-48b0-956b-d6d3b0b31fd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thwfx"
Sep 18 20:07:01.910976 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:07:01.910981    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3193c781-85d8-48b0-956b-d6d3b0b31fd8-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-thwfx\" (UID: \"3193c781-85d8-48b0-956b-d6d3b0b31fd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thwfx"
Sep 18 20:07:01.911221 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:07:01.911019    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3193c781-85d8-48b0-956b-d6d3b0b31fd8-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-thwfx\" (UID: \"3193c781-85d8-48b0-956b-d6d3b0b31fd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thwfx"
Sep 18 20:07:01.911221 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:07:01.911043    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3193c781-85d8-48b0-956b-d6d3b0b31fd8-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-thwfx\" (UID: \"3193c781-85d8-48b0-956b-d6d3b0b31fd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thwfx"
Sep 18 20:07:01.911221 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:07:01.911066    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3193c781-85d8-48b0-956b-d6d3b0b31fd8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-thwfx\" (UID: \"3193c781-85d8-48b0-956b-d6d3b0b31fd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thwfx"
Sep 18 20:07:01.911221 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:07:01.911137    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3193c781-85d8-48b0-956b-d6d3b0b31fd8-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-thwfx\" (UID: \"3193c781-85d8-48b0-956b-d6d3b0b31fd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thwfx"
Sep 18 20:07:01.911221 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:07:01.911177    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3193c781-85d8-48b0-956b-d6d3b0b31fd8-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-thwfx\" (UID: \"3193c781-85d8-48b0-956b-d6d3b0b31fd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thwfx"
Sep 18 20:07:01.911508 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:07:01.911232    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3193c781-85d8-48b0-956b-d6d3b0b31fd8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-thwfx\" (UID: \"3193c781-85d8-48b0-956b-d6d3b0b31fd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thwfx"
Sep 18 20:07:01.911508 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:07:01.911261    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgwsg\" (UniqueName: \"kubernetes.io/projected/3193c781-85d8-48b0-956b-d6d3b0b31fd8-kube-api-access-hgwsg\") pod \"maas-default-gateway-openshift-default-78bb78ff78-thwfx\" (UID: \"3193c781-85d8-48b0-956b-d6d3b0b31fd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thwfx"
Sep 18 20:07:01.911508 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:07:01.911449    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3193c781-85d8-48b0-956b-d6d3b0b31fd8-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-thwfx\" (UID: \"3193c781-85d8-48b0-956b-d6d3b0b31fd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thwfx"
Sep 18 20:07:01.911659 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:07:01.911548    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3193c781-85d8-48b0-956b-d6d3b0b31fd8-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-thwfx\" (UID: \"3193c781-85d8-48b0-956b-d6d3b0b31fd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thwfx"
Sep 18 20:07:01.911701 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:07:01.911659    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3193c781-85d8-48b0-956b-d6d3b0b31fd8-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-thwfx\" (UID: \"3193c781-85d8-48b0-956b-d6d3b0b31fd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thwfx"
Sep 18 20:07:01.911784 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:07:01.911767    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3193c781-85d8-48b0-956b-d6d3b0b31fd8-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-thwfx\" (UID: \"3193c781-85d8-48b0-956b-d6d3b0b31fd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thwfx"
Sep 18 20:07:01.912008 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:07:01.911985    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3193c781-85d8-48b0-956b-d6d3b0b31fd8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-thwfx\" (UID: \"3193c781-85d8-48b0-956b-d6d3b0b31fd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thwfx"
Sep 18 20:07:01.913442 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:07:01.913412    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3193c781-85d8-48b0-956b-d6d3b0b31fd8-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-thwfx\" (UID: \"3193c781-85d8-48b0-956b-d6d3b0b31fd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thwfx"
Sep 18 20:07:01.913728 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:07:01.913712    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3193c781-85d8-48b0-956b-d6d3b0b31fd8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-thwfx\" (UID: \"3193c781-85d8-48b0-956b-d6d3b0b31fd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thwfx"
Sep 18 20:07:01.919662 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:07:01.919643    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgwsg\" (UniqueName: \"kubernetes.io/projected/3193c781-85d8-48b0-956b-d6d3b0b31fd8-kube-api-access-hgwsg\") pod \"maas-default-gateway-openshift-default-78bb78ff78-thwfx\" (UID: \"3193c781-85d8-48b0-956b-d6d3b0b31fd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thwfx"
Sep 18 20:07:01.919758 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:07:01.919669    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3193c781-85d8-48b0-956b-d6d3b0b31fd8-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-thwfx\" (UID: \"3193c781-85d8-48b0-956b-d6d3b0b31fd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thwfx"
Sep 18 20:07:01.984745 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:07:01.984670    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thwfx"
Sep 18 20:07:02.259931 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:07:02.259904    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thwfx"]
Sep 18 20:07:02.261356 ip-10-0-140-178 kubenswrapper[2549]: W0918 20:07:02.261327    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3193c781_85d8_48b0_956b_d6d3b0b31fd8.slice/crio-848c7d48d0054e965a276f341556e6f304d6a255c0ccd43a7cabc41e60d38d34 WatchSource:0}: Error finding container 848c7d48d0054e965a276f341556e6f304d6a255c0ccd43a7cabc41e60d38d34: Status 404 returned error can't find the container with id 848c7d48d0054e965a276f341556e6f304d6a255c0ccd43a7cabc41e60d38d34
Sep 18 20:07:02.263516 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:07:02.263483    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:07:02.263628 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:07:02.263559    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:07:02.263628 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:07:02.263602    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:07:02.312326 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:07:02.312299    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thwfx" event={"ID":"3193c781-85d8-48b0-956b-d6d3b0b31fd8","Type":"ContainerStarted","Data":"848c7d48d0054e965a276f341556e6f304d6a255c0ccd43a7cabc41e60d38d34"}
Sep 18 20:07:03.318605 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:07:03.318570    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thwfx" event={"ID":"3193c781-85d8-48b0-956b-d6d3b0b31fd8","Type":"ContainerStarted","Data":"d95c651e055aecdd111f13794da0a9aa02d935db353d4771b4582d2a62d279b9"}
Sep 18 20:07:03.345093 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:07:03.345045    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thwfx" podStartSLOduration=2.345028901 podStartE2EDuration="2.345028901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:07:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:07:03.34229572 +0000 UTC m=+608.296036888" watchObservedRunningTime="2026-09-18 20:07:03.345028901 +0000 UTC m=+608.298770068"
Sep 18 20:07:03.985892 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:07:03.985864    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thwfx"
Sep 18 20:07:03.990355 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:07:03.990328    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thwfx"
Sep 18 20:07:04.324989 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:07:04.324960    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thwfx"
Sep 18 20:07:04.325899 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:07:04.325881    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thwfx"
Sep 18 20:07:04.686401 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:07:04.686320    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:07:04.688842 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:07:04.688822    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"
Sep 18 20:07:04.692367 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:07:04.692348    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 18 20:07:04.699885 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:07:04.699860    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:07:04.741262 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:07:04.741240    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:07:04.839304 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:07:04.839273    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/95ac679b-9bdc-493a-8c1f-b758fff7d092-config-file\") pod \"limitador-limitador-84994f7ffc-jf6z5\" (UID: \"95ac679b-9bdc-493a-8c1f-b758fff7d092\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"
Sep 18 20:07:04.839304 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:07:04.839305    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhl6n\" (UniqueName: \"kubernetes.io/projected/95ac679b-9bdc-493a-8c1f-b758fff7d092-kube-api-access-bhl6n\") pod \"limitador-limitador-84994f7ffc-jf6z5\" (UID: \"95ac679b-9bdc-493a-8c1f-b758fff7d092\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"
Sep 18 20:07:04.940215 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:07:04.940139    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/95ac679b-9bdc-493a-8c1f-b758fff7d092-config-file\") pod \"limitador-limitador-84994f7ffc-jf6z5\" (UID: \"95ac679b-9bdc-493a-8c1f-b758fff7d092\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"
Sep 18 20:07:04.940215 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:07:04.940174    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhl6n\" (UniqueName: \"kubernetes.io/projected/95ac679b-9bdc-493a-8c1f-b758fff7d092-kube-api-access-bhl6n\") pod \"limitador-limitador-84994f7ffc-jf6z5\" (UID: \"95ac679b-9bdc-493a-8c1f-b758fff7d092\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"
Sep 18 20:07:04.940851 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:07:04.940831    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/95ac679b-9bdc-493a-8c1f-b758fff7d092-config-file\") pod \"limitador-limitador-84994f7ffc-jf6z5\" (UID: \"95ac679b-9bdc-493a-8c1f-b758fff7d092\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"
Sep 18 20:07:04.960616 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:07:04.960585    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhl6n\" (UniqueName: \"kubernetes.io/projected/95ac679b-9bdc-493a-8c1f-b758fff7d092-kube-api-access-bhl6n\") pod \"limitador-limitador-84994f7ffc-jf6z5\" (UID: \"95ac679b-9bdc-493a-8c1f-b758fff7d092\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"
Sep 18 20:07:04.999900 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:07:04.999873    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"
Sep 18 20:07:05.277919 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:07:05.277712    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:07:05.279997 ip-10-0-140-178 kubenswrapper[2549]: W0918 20:07:05.279972    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95ac679b_9bdc_493a_8c1f_b758fff7d092.slice/crio-9a87bfa063ac46b15f727e04586e8bb2a6aca0bcd7472130aa889ce1cf42f271 WatchSource:0}: Error finding container 9a87bfa063ac46b15f727e04586e8bb2a6aca0bcd7472130aa889ce1cf42f271: Status 404 returned error can't find the container with id 9a87bfa063ac46b15f727e04586e8bb2a6aca0bcd7472130aa889ce1cf42f271
Sep 18 20:07:05.331025 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:07:05.330996    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-jf6z5" event={"ID":"95ac679b-9bdc-493a-8c1f-b758fff7d092","Type":"ContainerStarted","Data":"9a87bfa063ac46b15f727e04586e8bb2a6aca0bcd7472130aa889ce1cf42f271"}
Sep 18 20:07:06.340760 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:07:06.340724    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-jf6z5" event={"ID":"95ac679b-9bdc-493a-8c1f-b758fff7d092","Type":"ContainerStarted","Data":"eb1de0cac46e0c79300fa1a38fc0dacc804df5e9f44f65f678cfea89bfb9e82a"}
Sep 18 20:07:06.341127 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:07:06.340930    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"
Sep 18 20:07:06.370762 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:07:06.370719    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-jf6z5" podStartSLOduration=1.4002857450000001 podStartE2EDuration="2.370704165s" totalImagesPullingTime="970.41842ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:07:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:07:06.369631823 +0000 UTC m=+611.323372991" watchObservedRunningTime="2026-09-18 20:07:06.370704165 +0000 UTC m=+611.324445332"
Sep 18 20:07:17.346637 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:07:17.346607    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"
Sep 18 20:08:17.542619 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:08:17.542589    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:08:26.758527 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:08:26.758431    2549 generic.go:356] "Generic (PLEG): container finished" podID="3193c781-85d8-48b0-956b-d6d3b0b31fd8" containerID="d95c651e055aecdd111f13794da0a9aa02d935db353d4771b4582d2a62d279b9" exitCode=0
Sep 18 20:08:26.758527 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:08:26.758500    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thwfx" event={"ID":"3193c781-85d8-48b0-956b-d6d3b0b31fd8","Type":"ContainerDied","Data":"d95c651e055aecdd111f13794da0a9aa02d935db353d4771b4582d2a62d279b9"}
Sep 18 20:08:26.764769 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:08:26.759433    2549 scope.go:117] "RemoveContainer" containerID="d95c651e055aecdd111f13794da0a9aa02d935db353d4771b4582d2a62d279b9"
Sep 18 20:08:26.765608 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:08:26.765574    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:08:26.765750 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:08:26.765656    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:08:26.765750 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:08:26.765690    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:08:27.770372 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:08:27.769526    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thwfx" event={"ID":"3193c781-85d8-48b0-956b-d6d3b0b31fd8","Type":"ContainerStarted","Data":"bef4cb1f4d8113d8a92c875b008d12f1cf2c4c3bedb23a1f67475c43a8ad75a1"}
Sep 18 20:08:27.985810 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:08:27.985765    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thwfx"
Sep 18 20:08:27.986016 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:08:27.985988    2549 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-thwfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Sep 18 20:08:27.986104 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:08:27.986053    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thwfx" podUID="3193c781-85d8-48b0-956b-d6d3b0b31fd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Sep 18 20:08:28.984986 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:08:28.984952    2549 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-thwfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Sep 18 20:08:28.985366 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:08:28.985005    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thwfx" podUID="3193c781-85d8-48b0-956b-d6d3b0b31fd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Sep 18 20:08:29.985625 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:08:29.985586    2549 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-thwfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Sep 18 20:08:29.985999 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:08:29.985646    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thwfx" podUID="3193c781-85d8-48b0-956b-d6d3b0b31fd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Sep 18 20:08:30.985898 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:08:30.985860    2549 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-thwfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Sep 18 20:08:30.986289 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:08:30.985944    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thwfx" podUID="3193c781-85d8-48b0-956b-d6d3b0b31fd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Sep 18 20:08:31.985441 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:08:31.985377    2549 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-thwfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Sep 18 20:08:31.985610 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:08:31.985452    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thwfx" podUID="3193c781-85d8-48b0-956b-d6d3b0b31fd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Sep 18 20:08:32.989692 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:08:32.989659    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thwfx"
Sep 18 20:08:32.990070 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:08:32.989945    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thwfx"
Sep 18 20:08:32.990563 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:08:32.990548    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thwfx"
Sep 18 20:08:53.711336 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:08:53.711301    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:09:05.738760 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:09:05.738725    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:09:15.766437 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:09:15.766396    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:09:28.112265 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:09:28.112234    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:09:40.350852 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:09:40.350815    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:09:47.421789 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:09:47.421757    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:09:53.840189 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:09:53.840158    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:11:18.644374 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:18.644340    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48"]
Sep 18 20:11:18.647342 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:18.647306    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48"
Sep 18 20:11:18.653245 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:11:18.653211    2549 status_manager.go:1018] "Failed to get status for pod" err="pods \"e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48\" is forbidden: User \"system:node:ip-10-0-140-178.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-140-178.ec2.internal' and this object" podUID="e773b934-ce8f-43ce-899c-a2c425da0c33" pod="openshift-ingress/e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48"
Sep 18 20:11:18.654095 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:11:18.654057    2549 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"e2e-worker-w3-4b633d-openshift-default-dockercfg-48l5m\" is forbidden: User \"system:node:ip-10-0-140-178.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-140-178.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-4b633d-openshift-default-dockercfg-48l5m\"" type="*v1.Secret"
Sep 18 20:11:18.689195 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:18.689166    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48"]
Sep 18 20:11:18.771740 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:18.771711    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e773b934-ce8f-43ce-899c-a2c425da0c33-credential-socket\") pod \"e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48\" (UID: \"e773b934-ce8f-43ce-899c-a2c425da0c33\") " pod="openshift-ingress/e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48"
Sep 18 20:11:18.771740 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:18.771745    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e773b934-ce8f-43ce-899c-a2c425da0c33-istio-data\") pod \"e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48\" (UID: \"e773b934-ce8f-43ce-899c-a2c425da0c33\") " pod="openshift-ingress/e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48"
Sep 18 20:11:18.771932 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:18.771817    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klnpm\" (UniqueName: \"kubernetes.io/projected/e773b934-ce8f-43ce-899c-a2c425da0c33-kube-api-access-klnpm\") pod \"e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48\" (UID: \"e773b934-ce8f-43ce-899c-a2c425da0c33\") " pod="openshift-ingress/e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48"
Sep 18 20:11:18.771932 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:18.771856    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e773b934-ce8f-43ce-899c-a2c425da0c33-workload-certs\") pod \"e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48\" (UID: \"e773b934-ce8f-43ce-899c-a2c425da0c33\") " pod="openshift-ingress/e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48"
Sep 18 20:11:18.771932 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:18.771876    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e773b934-ce8f-43ce-899c-a2c425da0c33-istio-token\") pod \"e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48\" (UID: \"e773b934-ce8f-43ce-899c-a2c425da0c33\") " pod="openshift-ingress/e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48"
Sep 18 20:11:18.771932 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:18.771894    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e773b934-ce8f-43ce-899c-a2c425da0c33-workload-socket\") pod \"e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48\" (UID: \"e773b934-ce8f-43ce-899c-a2c425da0c33\") " pod="openshift-ingress/e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48"
Sep 18 20:11:18.772070 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:18.771970    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e773b934-ce8f-43ce-899c-a2c425da0c33-istio-envoy\") pod \"e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48\" (UID: \"e773b934-ce8f-43ce-899c-a2c425da0c33\") " pod="openshift-ingress/e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48"
Sep 18 20:11:18.772070 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:18.771995    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e773b934-ce8f-43ce-899c-a2c425da0c33-istio-podinfo\") pod \"e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48\" (UID: \"e773b934-ce8f-43ce-899c-a2c425da0c33\") " pod="openshift-ingress/e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48"
Sep 18 20:11:18.772070 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:18.772015    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e773b934-ce8f-43ce-899c-a2c425da0c33-istiod-ca-cert\") pod \"e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48\" (UID: \"e773b934-ce8f-43ce-899c-a2c425da0c33\") " pod="openshift-ingress/e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48"
Sep 18 20:11:18.873048 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:18.873012    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klnpm\" (UniqueName: \"kubernetes.io/projected/e773b934-ce8f-43ce-899c-a2c425da0c33-kube-api-access-klnpm\") pod \"e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48\" (UID: \"e773b934-ce8f-43ce-899c-a2c425da0c33\") " pod="openshift-ingress/e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48"
Sep 18 20:11:18.873048 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:18.873050    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e773b934-ce8f-43ce-899c-a2c425da0c33-workload-certs\") pod \"e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48\" (UID: \"e773b934-ce8f-43ce-899c-a2c425da0c33\") " pod="openshift-ingress/e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48"
Sep 18 20:11:18.873283 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:18.873076    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e773b934-ce8f-43ce-899c-a2c425da0c33-istio-token\") pod \"e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48\" (UID: \"e773b934-ce8f-43ce-899c-a2c425da0c33\") " pod="openshift-ingress/e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48"
Sep 18 20:11:18.873283 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:18.873103    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e773b934-ce8f-43ce-899c-a2c425da0c33-workload-socket\") pod \"e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48\" (UID: \"e773b934-ce8f-43ce-899c-a2c425da0c33\") " pod="openshift-ingress/e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48"
Sep 18 20:11:18.873283 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:18.873182    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e773b934-ce8f-43ce-899c-a2c425da0c33-istio-envoy\") pod \"e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48\" (UID: \"e773b934-ce8f-43ce-899c-a2c425da0c33\") " pod="openshift-ingress/e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48"
Sep 18 20:11:18.873283 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:18.873219    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e773b934-ce8f-43ce-899c-a2c425da0c33-istio-podinfo\") pod \"e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48\" (UID: \"e773b934-ce8f-43ce-899c-a2c425da0c33\") " pod="openshift-ingress/e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48"
Sep 18 20:11:18.873283 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:18.873247    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e773b934-ce8f-43ce-899c-a2c425da0c33-istiod-ca-cert\") pod \"e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48\" (UID: \"e773b934-ce8f-43ce-899c-a2c425da0c33\") " pod="openshift-ingress/e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48"
Sep 18 20:11:18.873566 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:18.873308    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e773b934-ce8f-43ce-899c-a2c425da0c33-credential-socket\") pod \"e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48\" (UID: \"e773b934-ce8f-43ce-899c-a2c425da0c33\") " pod="openshift-ingress/e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48"
Sep 18 20:11:18.873566 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:18.873331    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e773b934-ce8f-43ce-899c-a2c425da0c33-istio-data\") pod \"e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48\" (UID: \"e773b934-ce8f-43ce-899c-a2c425da0c33\") " pod="openshift-ingress/e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48"
Sep 18 20:11:18.873669 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:18.873599    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e773b934-ce8f-43ce-899c-a2c425da0c33-workload-socket\") pod \"e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48\" (UID: \"e773b934-ce8f-43ce-899c-a2c425da0c33\") " pod="openshift-ingress/e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48"
Sep 18 20:11:18.873669 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:18.873652    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e773b934-ce8f-43ce-899c-a2c425da0c33-workload-certs\") pod \"e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48\" (UID: \"e773b934-ce8f-43ce-899c-a2c425da0c33\") " pod="openshift-ingress/e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48"
Sep 18 20:11:18.873735 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:18.873689    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e773b934-ce8f-43ce-899c-a2c425da0c33-credential-socket\") pod \"e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48\" (UID: \"e773b934-ce8f-43ce-899c-a2c425da0c33\") " pod="openshift-ingress/e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48"
Sep 18 20:11:18.874018 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:18.873997    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e773b934-ce8f-43ce-899c-a2c425da0c33-istio-data\") pod \"e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48\" (UID: \"e773b934-ce8f-43ce-899c-a2c425da0c33\") " pod="openshift-ingress/e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48"
Sep 18 20:11:18.874108 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:18.874043    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e773b934-ce8f-43ce-899c-a2c425da0c33-istiod-ca-cert\") pod \"e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48\" (UID: \"e773b934-ce8f-43ce-899c-a2c425da0c33\") " pod="openshift-ingress/e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48"
Sep 18 20:11:18.875948 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:18.875913    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e773b934-ce8f-43ce-899c-a2c425da0c33-istio-podinfo\") pod \"e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48\" (UID: \"e773b934-ce8f-43ce-899c-a2c425da0c33\") " pod="openshift-ingress/e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48"
Sep 18 20:11:18.876089 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:18.876069    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk"]
Sep 18 20:11:18.876147 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:18.876089    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e773b934-ce8f-43ce-899c-a2c425da0c33-istio-envoy\") pod \"e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48\" (UID: \"e773b934-ce8f-43ce-899c-a2c425da0c33\") " pod="openshift-ingress/e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48"
Sep 18 20:11:18.878597 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:18.878582    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk"
Sep 18 20:11:18.885575 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:18.885556    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-3a8052-openshift-default-dockercfg-9s26p\""
Sep 18 20:11:18.914883 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:18.914857    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-klnpm\" (UniqueName: \"kubernetes.io/projected/e773b934-ce8f-43ce-899c-a2c425da0c33-kube-api-access-klnpm\") pod \"e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48\" (UID: \"e773b934-ce8f-43ce-899c-a2c425da0c33\") " pod="openshift-ingress/e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48"
Sep 18 20:11:18.915460 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:18.915444    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e773b934-ce8f-43ce-899c-a2c425da0c33-istio-token\") pod \"e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48\" (UID: \"e773b934-ce8f-43ce-899c-a2c425da0c33\") " pod="openshift-ingress/e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48"
Sep 18 20:11:18.938752 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:18.938725    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk"]
Sep 18 20:11:18.974089 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:18.974058    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3d791d0-5f42-41d0-9ca2-05d11123c178-workload-socket\") pod \"e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk\" (UID: \"f3d791d0-5f42-41d0-9ca2-05d11123c178\") " pod="openshift-ingress/e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk"
Sep 18 20:11:18.974241 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:18.974095    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3d791d0-5f42-41d0-9ca2-05d11123c178-istio-podinfo\") pod \"e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk\" (UID: \"f3d791d0-5f42-41d0-9ca2-05d11123c178\") " pod="openshift-ingress/e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk"
Sep 18 20:11:18.974241 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:18.974131    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3d791d0-5f42-41d0-9ca2-05d11123c178-credential-socket\") pod \"e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk\" (UID: \"f3d791d0-5f42-41d0-9ca2-05d11123c178\") " pod="openshift-ingress/e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk"
Sep 18 20:11:18.974241 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:18.974147    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3d791d0-5f42-41d0-9ca2-05d11123c178-istiod-ca-cert\") pod \"e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk\" (UID: \"f3d791d0-5f42-41d0-9ca2-05d11123c178\") " pod="openshift-ingress/e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk"
Sep 18 20:11:18.974241 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:18.974177    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3d791d0-5f42-41d0-9ca2-05d11123c178-workload-certs\") pod \"e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk\" (UID: \"f3d791d0-5f42-41d0-9ca2-05d11123c178\") " pod="openshift-ingress/e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk"
Sep 18 20:11:18.974241 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:18.974222    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3d791d0-5f42-41d0-9ca2-05d11123c178-istio-envoy\") pod \"e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk\" (UID: \"f3d791d0-5f42-41d0-9ca2-05d11123c178\") " pod="openshift-ingress/e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk"
Sep 18 20:11:18.974514 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:18.974268    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3d791d0-5f42-41d0-9ca2-05d11123c178-istio-data\") pod \"e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk\" (UID: \"f3d791d0-5f42-41d0-9ca2-05d11123c178\") " pod="openshift-ingress/e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk"
Sep 18 20:11:18.974514 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:18.974303    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6djjg\" (UniqueName: \"kubernetes.io/projected/f3d791d0-5f42-41d0-9ca2-05d11123c178-kube-api-access-6djjg\") pod \"e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk\" (UID: \"f3d791d0-5f42-41d0-9ca2-05d11123c178\") " pod="openshift-ingress/e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk"
Sep 18 20:11:18.974514 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:18.974349    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3d791d0-5f42-41d0-9ca2-05d11123c178-istio-token\") pod \"e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk\" (UID: \"f3d791d0-5f42-41d0-9ca2-05d11123c178\") " pod="openshift-ingress/e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk"
Sep 18 20:11:19.075248 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:19.075214    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3d791d0-5f42-41d0-9ca2-05d11123c178-workload-certs\") pod \"e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk\" (UID: \"f3d791d0-5f42-41d0-9ca2-05d11123c178\") " pod="openshift-ingress/e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk"
Sep 18 20:11:19.075248 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:19.075254    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3d791d0-5f42-41d0-9ca2-05d11123c178-istio-envoy\") pod \"e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk\" (UID: \"f3d791d0-5f42-41d0-9ca2-05d11123c178\") " pod="openshift-ingress/e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk"
Sep 18 20:11:19.075482 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:19.075276    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3d791d0-5f42-41d0-9ca2-05d11123c178-istio-data\") pod \"e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk\" (UID: \"f3d791d0-5f42-41d0-9ca2-05d11123c178\") " pod="openshift-ingress/e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk"
Sep 18 20:11:19.075482 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:19.075407    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6djjg\" (UniqueName: \"kubernetes.io/projected/f3d791d0-5f42-41d0-9ca2-05d11123c178-kube-api-access-6djjg\") pod \"e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk\" (UID: \"f3d791d0-5f42-41d0-9ca2-05d11123c178\") " pod="openshift-ingress/e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk"
Sep 18 20:11:19.075482 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:19.075475    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3d791d0-5f42-41d0-9ca2-05d11123c178-istio-token\") pod \"e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk\" (UID: \"f3d791d0-5f42-41d0-9ca2-05d11123c178\") " pod="openshift-ingress/e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk"
Sep 18 20:11:19.075647 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:19.075523    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3d791d0-5f42-41d0-9ca2-05d11123c178-workload-socket\") pod \"e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk\" (UID: \"f3d791d0-5f42-41d0-9ca2-05d11123c178\") " pod="openshift-ingress/e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk"
Sep 18 20:11:19.075647 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:19.075556    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3d791d0-5f42-41d0-9ca2-05d11123c178-istio-podinfo\") pod \"e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk\" (UID: \"f3d791d0-5f42-41d0-9ca2-05d11123c178\") " pod="openshift-ingress/e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk"
Sep 18 20:11:19.075647 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:19.075631    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3d791d0-5f42-41d0-9ca2-05d11123c178-credential-socket\") pod \"e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk\" (UID: \"f3d791d0-5f42-41d0-9ca2-05d11123c178\") " pod="openshift-ingress/e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk"
Sep 18 20:11:19.075803 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:19.075659    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3d791d0-5f42-41d0-9ca2-05d11123c178-istiod-ca-cert\") pod \"e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk\" (UID: \"f3d791d0-5f42-41d0-9ca2-05d11123c178\") " pod="openshift-ingress/e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk"
Sep 18 20:11:19.075803 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:19.075660    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3d791d0-5f42-41d0-9ca2-05d11123c178-istio-data\") pod \"e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk\" (UID: \"f3d791d0-5f42-41d0-9ca2-05d11123c178\") " pod="openshift-ingress/e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk"
Sep 18 20:11:19.075900 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:19.075843    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3d791d0-5f42-41d0-9ca2-05d11123c178-workload-certs\") pod \"e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk\" (UID: \"f3d791d0-5f42-41d0-9ca2-05d11123c178\") " pod="openshift-ingress/e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk"
Sep 18 20:11:19.075983 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:19.075960    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3d791d0-5f42-41d0-9ca2-05d11123c178-workload-socket\") pod \"e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk\" (UID: \"f3d791d0-5f42-41d0-9ca2-05d11123c178\") " pod="openshift-ingress/e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk"
Sep 18 20:11:19.076216 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:19.076195    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3d791d0-5f42-41d0-9ca2-05d11123c178-credential-socket\") pod \"e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk\" (UID: \"f3d791d0-5f42-41d0-9ca2-05d11123c178\") " pod="openshift-ingress/e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk"
Sep 18 20:11:19.076401 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:19.076369    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3d791d0-5f42-41d0-9ca2-05d11123c178-istiod-ca-cert\") pod \"e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk\" (UID: \"f3d791d0-5f42-41d0-9ca2-05d11123c178\") " pod="openshift-ingress/e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk"
Sep 18 20:11:19.077634 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:19.077610    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3d791d0-5f42-41d0-9ca2-05d11123c178-istio-envoy\") pod \"e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk\" (UID: \"f3d791d0-5f42-41d0-9ca2-05d11123c178\") " pod="openshift-ingress/e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk"
Sep 18 20:11:19.077945 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:19.077930    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3d791d0-5f42-41d0-9ca2-05d11123c178-istio-podinfo\") pod \"e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk\" (UID: \"f3d791d0-5f42-41d0-9ca2-05d11123c178\") " pod="openshift-ingress/e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk"
Sep 18 20:11:19.098672 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:19.098649    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3d791d0-5f42-41d0-9ca2-05d11123c178-istio-token\") pod \"e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk\" (UID: \"f3d791d0-5f42-41d0-9ca2-05d11123c178\") " pod="openshift-ingress/e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk"
Sep 18 20:11:19.113461 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:19.113441    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6djjg\" (UniqueName: \"kubernetes.io/projected/f3d791d0-5f42-41d0-9ca2-05d11123c178-kube-api-access-6djjg\") pod \"e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk\" (UID: \"f3d791d0-5f42-41d0-9ca2-05d11123c178\") " pod="openshift-ingress/e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk"
Sep 18 20:11:19.199803 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:19.199731    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk"
Sep 18 20:11:19.202850 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:19.202826    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr"]
Sep 18 20:11:19.205094 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:19.205074    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr"
Sep 18 20:11:19.209626 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:19.209602    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-5284dc-openshift-default-dockercfg-7cvkd\""
Sep 18 20:11:19.277368 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:19.277338    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/374c8e70-b164-4b98-8c5d-f7994db126f2-credential-socket\") pod \"e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr\" (UID: \"374c8e70-b164-4b98-8c5d-f7994db126f2\") " pod="openshift-ingress/e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr"
Sep 18 20:11:19.277518 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:19.277421    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/374c8e70-b164-4b98-8c5d-f7994db126f2-istiod-ca-cert\") pod \"e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr\" (UID: \"374c8e70-b164-4b98-8c5d-f7994db126f2\") " pod="openshift-ingress/e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr"
Sep 18 20:11:19.277518 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:19.277468    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cp6t\" (UniqueName: \"kubernetes.io/projected/374c8e70-b164-4b98-8c5d-f7994db126f2-kube-api-access-6cp6t\") pod \"e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr\" (UID: \"374c8e70-b164-4b98-8c5d-f7994db126f2\") " pod="openshift-ingress/e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr"
Sep 18 20:11:19.277628 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:19.277525    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/374c8e70-b164-4b98-8c5d-f7994db126f2-workload-certs\") pod \"e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr\" (UID: \"374c8e70-b164-4b98-8c5d-f7994db126f2\") " pod="openshift-ingress/e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr"
Sep 18 20:11:19.277628 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:19.277545    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/374c8e70-b164-4b98-8c5d-f7994db126f2-istio-envoy\") pod \"e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr\" (UID: \"374c8e70-b164-4b98-8c5d-f7994db126f2\") " pod="openshift-ingress/e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr"
Sep 18 20:11:19.277628 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:19.277558    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/374c8e70-b164-4b98-8c5d-f7994db126f2-istio-data\") pod \"e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr\" (UID: \"374c8e70-b164-4b98-8c5d-f7994db126f2\") " pod="openshift-ingress/e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr"
Sep 18 20:11:19.277744 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:19.277659    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/374c8e70-b164-4b98-8c5d-f7994db126f2-istio-token\") pod \"e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr\" (UID: \"374c8e70-b164-4b98-8c5d-f7994db126f2\") " pod="openshift-ingress/e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr"
Sep 18 20:11:19.277744 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:19.277702    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/374c8e70-b164-4b98-8c5d-f7994db126f2-workload-socket\") pod \"e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr\" (UID: \"374c8e70-b164-4b98-8c5d-f7994db126f2\") " pod="openshift-ingress/e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr"
Sep 18 20:11:19.277744 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:19.277731    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/374c8e70-b164-4b98-8c5d-f7994db126f2-istio-podinfo\") pod \"e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr\" (UID: \"374c8e70-b164-4b98-8c5d-f7994db126f2\") " pod="openshift-ingress/e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr"
Sep 18 20:11:19.344220 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:19.344185    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr"]
Sep 18 20:11:19.378166 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:19.378134    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/374c8e70-b164-4b98-8c5d-f7994db126f2-workload-certs\") pod \"e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr\" (UID: \"374c8e70-b164-4b98-8c5d-f7994db126f2\") " pod="openshift-ingress/e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr"
Sep 18 20:11:19.378166 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:19.378169    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/374c8e70-b164-4b98-8c5d-f7994db126f2-istio-envoy\") pod \"e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr\" (UID: \"374c8e70-b164-4b98-8c5d-f7994db126f2\") " pod="openshift-ingress/e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr"
Sep 18 20:11:19.378365 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:19.378185    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/374c8e70-b164-4b98-8c5d-f7994db126f2-istio-data\") pod \"e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr\" (UID: \"374c8e70-b164-4b98-8c5d-f7994db126f2\") " pod="openshift-ingress/e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr"
Sep 18 20:11:19.378365 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:19.378241    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/374c8e70-b164-4b98-8c5d-f7994db126f2-istio-token\") pod \"e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr\" (UID: \"374c8e70-b164-4b98-8c5d-f7994db126f2\") " pod="openshift-ingress/e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr"
Sep 18 20:11:19.378365 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:19.378291    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/374c8e70-b164-4b98-8c5d-f7994db126f2-workload-socket\") pod \"e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr\" (UID: \"374c8e70-b164-4b98-8c5d-f7994db126f2\") " pod="openshift-ingress/e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr"
Sep 18 20:11:19.378365 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:19.378331    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/374c8e70-b164-4b98-8c5d-f7994db126f2-istio-podinfo\") pod \"e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr\" (UID: \"374c8e70-b164-4b98-8c5d-f7994db126f2\") " pod="openshift-ingress/e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr"
Sep 18 20:11:19.378591 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:19.378376    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/374c8e70-b164-4b98-8c5d-f7994db126f2-credential-socket\") pod \"e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr\" (UID: \"374c8e70-b164-4b98-8c5d-f7994db126f2\") " pod="openshift-ingress/e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr"
Sep 18 20:11:19.378591 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:19.378442    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/374c8e70-b164-4b98-8c5d-f7994db126f2-istiod-ca-cert\") pod \"e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr\" (UID: \"374c8e70-b164-4b98-8c5d-f7994db126f2\") " pod="openshift-ingress/e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr"
Sep 18 20:11:19.378702 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:19.378615    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/374c8e70-b164-4b98-8c5d-f7994db126f2-workload-certs\") pod \"e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr\" (UID: \"374c8e70-b164-4b98-8c5d-f7994db126f2\") " pod="openshift-ingress/e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr"
Sep 18 20:11:19.378702 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:19.378635    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cp6t\" (UniqueName: \"kubernetes.io/projected/374c8e70-b164-4b98-8c5d-f7994db126f2-kube-api-access-6cp6t\") pod \"e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr\" (UID: \"374c8e70-b164-4b98-8c5d-f7994db126f2\") " pod="openshift-ingress/e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr"
Sep 18 20:11:19.378794 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:19.378716    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/374c8e70-b164-4b98-8c5d-f7994db126f2-istio-data\") pod \"e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr\" (UID: \"374c8e70-b164-4b98-8c5d-f7994db126f2\") " pod="openshift-ingress/e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr"
Sep 18 20:11:19.378973 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:19.378953    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/374c8e70-b164-4b98-8c5d-f7994db126f2-credential-socket\") pod \"e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr\" (UID: \"374c8e70-b164-4b98-8c5d-f7994db126f2\") " pod="openshift-ingress/e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr"
Sep 18 20:11:19.379034 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:19.378972    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/374c8e70-b164-4b98-8c5d-f7994db126f2-workload-socket\") pod \"e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr\" (UID: \"374c8e70-b164-4b98-8c5d-f7994db126f2\") " pod="openshift-ingress/e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr"
Sep 18 20:11:19.379107 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:19.379089    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/374c8e70-b164-4b98-8c5d-f7994db126f2-istiod-ca-cert\") pod \"e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr\" (UID: \"374c8e70-b164-4b98-8c5d-f7994db126f2\") " pod="openshift-ingress/e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr"
Sep 18 20:11:19.380530 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:19.380512    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/374c8e70-b164-4b98-8c5d-f7994db126f2-istio-envoy\") pod \"e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr\" (UID: \"374c8e70-b164-4b98-8c5d-f7994db126f2\") " pod="openshift-ingress/e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr"
Sep 18 20:11:19.380859 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:19.380839    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/374c8e70-b164-4b98-8c5d-f7994db126f2-istio-podinfo\") pod \"e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr\" (UID: \"374c8e70-b164-4b98-8c5d-f7994db126f2\") " pod="openshift-ingress/e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr"
Sep 18 20:11:19.401592 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:19.401568    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/374c8e70-b164-4b98-8c5d-f7994db126f2-istio-token\") pod \"e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr\" (UID: \"374c8e70-b164-4b98-8c5d-f7994db126f2\") " pod="openshift-ingress/e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr"
Sep 18 20:11:19.403841 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:19.403824    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cp6t\" (UniqueName: \"kubernetes.io/projected/374c8e70-b164-4b98-8c5d-f7994db126f2-kube-api-access-6cp6t\") pod \"e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr\" (UID: \"374c8e70-b164-4b98-8c5d-f7994db126f2\") " pod="openshift-ingress/e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr"
Sep 18 20:11:19.498745 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:19.498719    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk"]
Sep 18 20:11:19.501129 ip-10-0-140-178 kubenswrapper[2549]: W0918 20:11:19.501103    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3d791d0_5f42_41d0_9ca2_05d11123c178.slice/crio-ff5be8fd843d89b2c451174509b869c90aa2f61fdf451a122e30bff940a6898b WatchSource:0}: Error finding container ff5be8fd843d89b2c451174509b869c90aa2f61fdf451a122e30bff940a6898b: Status 404 returned error can't find the container with id ff5be8fd843d89b2c451174509b869c90aa2f61fdf451a122e30bff940a6898b
Sep 18 20:11:19.503264 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:19.503232    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:11:19.503364 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:19.503309    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:11:19.503364 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:19.503352    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:11:19.516502 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:19.516480    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr"
Sep 18 20:11:19.656445 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:19.656402    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk" event={"ID":"f3d791d0-5f42-41d0-9ca2-05d11123c178","Type":"ContainerStarted","Data":"f816bfed502fbeec9741f216d42b39e63756affcc99a168fdb5a0fe07e0cbe37"}
Sep 18 20:11:19.656806 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:19.656475    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk" event={"ID":"f3d791d0-5f42-41d0-9ca2-05d11123c178","Type":"ContainerStarted","Data":"ff5be8fd843d89b2c451174509b869c90aa2f61fdf451a122e30bff940a6898b"}
Sep 18 20:11:19.680155 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:19.680130    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-4b633d-openshift-default-dockercfg-48l5m\""
Sep 18 20:11:19.682263 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:19.682238    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48"
Sep 18 20:11:19.765352 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:19.765042    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk" podStartSLOduration=1.765023325 podStartE2EDuration="1.765023325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:11:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:11:19.760337689 +0000 UTC m=+864.714078858" watchObservedRunningTime="2026-09-18 20:11:19.765023325 +0000 UTC m=+864.718764493"
Sep 18 20:11:19.841873 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:19.841847    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr"]
Sep 18 20:11:19.843511 ip-10-0-140-178 kubenswrapper[2549]: W0918 20:11:19.843484    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod374c8e70_b164_4b98_8c5d_f7994db126f2.slice/crio-0ea2e05bdda724e84ab7dd16702f8d4ef9508aed5079adbc9cb7f5afb6238d40 WatchSource:0}: Error finding container 0ea2e05bdda724e84ab7dd16702f8d4ef9508aed5079adbc9cb7f5afb6238d40: Status 404 returned error can't find the container with id 0ea2e05bdda724e84ab7dd16702f8d4ef9508aed5079adbc9cb7f5afb6238d40
Sep 18 20:11:19.848284 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:19.845939    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:11:19.848284 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:19.846016    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:11:19.848284 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:19.846057    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:11:20.015986 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:20.015928    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48"]
Sep 18 20:11:20.016839 ip-10-0-140-178 kubenswrapper[2549]: W0918 20:11:20.016810    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode773b934_ce8f_43ce_899c_a2c425da0c33.slice/crio-ebd67750f59425ef2ee0ab8c5f438e1e15eb663215c62867f1b7a6fa2c404611 WatchSource:0}: Error finding container ebd67750f59425ef2ee0ab8c5f438e1e15eb663215c62867f1b7a6fa2c404611: Status 404 returned error can't find the container with id ebd67750f59425ef2ee0ab8c5f438e1e15eb663215c62867f1b7a6fa2c404611
Sep 18 20:11:20.019669 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:20.019636    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:11:20.019784 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:20.019706    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:11:20.019784 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:20.019734    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:11:20.200229 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:20.200196    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk"
Sep 18 20:11:20.205026 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:20.205001    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk"
Sep 18 20:11:20.403335 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:20.403299    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7747-8c2db0"]
Sep 18 20:11:20.406184 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:20.406157    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7747-8c2db0"
Sep 18 20:11:20.410181 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:20.410162    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-jcdsw\""
Sep 18 20:11:20.425793 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:20.425762    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7747-8c2db0"]
Sep 18 20:11:20.490149 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:20.490115    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s58lc\" (UniqueName: \"kubernetes.io/projected/a79b8d82-526b-45d0-844b-dc0c2bf6ee98-kube-api-access-s58lc\") pod \"test-curl-7747-8c2db0\" (UID: \"a79b8d82-526b-45d0-844b-dc0c2bf6ee98\") " pod="opendatahub/test-curl-7747-8c2db0"
Sep 18 20:11:20.586832 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:20.586798    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:11:20.590587 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:20.590563    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s58lc\" (UniqueName: \"kubernetes.io/projected/a79b8d82-526b-45d0-844b-dc0c2bf6ee98-kube-api-access-s58lc\") pod \"test-curl-7747-8c2db0\" (UID: \"a79b8d82-526b-45d0-844b-dc0c2bf6ee98\") " pod="opendatahub/test-curl-7747-8c2db0"
Sep 18 20:11:20.626041 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:20.626014    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s58lc\" (UniqueName: \"kubernetes.io/projected/a79b8d82-526b-45d0-844b-dc0c2bf6ee98-kube-api-access-s58lc\") pod \"test-curl-7747-8c2db0\" (UID: \"a79b8d82-526b-45d0-844b-dc0c2bf6ee98\") " pod="opendatahub/test-curl-7747-8c2db0"
Sep 18 20:11:20.665490 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:20.665399    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48" event={"ID":"e773b934-ce8f-43ce-899c-a2c425da0c33","Type":"ContainerStarted","Data":"0ecd23622c0958500be173983ee355784faaaf0494e238debce8c4acd460d3c1"}
Sep 18 20:11:20.665490 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:20.665462    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48" event={"ID":"e773b934-ce8f-43ce-899c-a2c425da0c33","Type":"ContainerStarted","Data":"ebd67750f59425ef2ee0ab8c5f438e1e15eb663215c62867f1b7a6fa2c404611"}
Sep 18 20:11:20.667020 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:20.666995    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr" event={"ID":"374c8e70-b164-4b98-8c5d-f7994db126f2","Type":"ContainerStarted","Data":"72d061c1a843da03a5e6768c6bae567b761ff73cfe8163e9d982404a1b36b202"}
Sep 18 20:11:20.667123 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:20.667035    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr" event={"ID":"374c8e70-b164-4b98-8c5d-f7994db126f2","Type":"ContainerStarted","Data":"0ea2e05bdda724e84ab7dd16702f8d4ef9508aed5079adbc9cb7f5afb6238d40"}
Sep 18 20:11:20.667327 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:20.667310    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk"
Sep 18 20:11:20.668113 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:20.668096    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk"
Sep 18 20:11:20.682623 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:20.682602    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48"
Sep 18 20:11:20.716090 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:20.716050    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48" podStartSLOduration=2.716036177 podStartE2EDuration="2.716036177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:11:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:11:20.714073215 +0000 UTC m=+865.667814383" watchObservedRunningTime="2026-09-18 20:11:20.716036177 +0000 UTC m=+865.669777344"
Sep 18 20:11:20.716256 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:20.716242    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7747-8c2db0"
Sep 18 20:11:20.816883 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:20.816825    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr" podStartSLOduration=1.816809264 podStartE2EDuration="1.816809264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:11:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:11:20.816573645 +0000 UTC m=+865.770314813" watchObservedRunningTime="2026-09-18 20:11:20.816809264 +0000 UTC m=+865.770550431"
Sep 18 20:11:21.001161 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:21.001130    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7747-8c2db0"]
Sep 18 20:11:21.001972 ip-10-0-140-178 kubenswrapper[2549]: W0918 20:11:21.001945    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda79b8d82_526b_45d0_844b_dc0c2bf6ee98.slice/crio-adb5a47822cf7d517282c035da5cdb27d9af3617ce5a53a5aa9be53e0c00cd54 WatchSource:0}: Error finding container adb5a47822cf7d517282c035da5cdb27d9af3617ce5a53a5aa9be53e0c00cd54: Status 404 returned error can't find the container with id adb5a47822cf7d517282c035da5cdb27d9af3617ce5a53a5aa9be53e0c00cd54
Sep 18 20:11:21.004114 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:21.004091    2549 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 20:11:21.526234 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:21.526201    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr"
Sep 18 20:11:21.538421 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:21.538369    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr"
Sep 18 20:11:21.674337 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:21.674305    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7747-8c2db0" event={"ID":"a79b8d82-526b-45d0-844b-dc0c2bf6ee98","Type":"ContainerStarted","Data":"adb5a47822cf7d517282c035da5cdb27d9af3617ce5a53a5aa9be53e0c00cd54"}
Sep 18 20:11:21.674750 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:21.674732    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr"
Sep 18 20:11:21.675770 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:21.675751    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr"
Sep 18 20:11:21.687485 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:21.687465    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48"
Sep 18 20:11:22.681457 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:22.681419    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7747-8c2db0" event={"ID":"a79b8d82-526b-45d0-844b-dc0c2bf6ee98","Type":"ContainerStarted","Data":"d5dd007b1fc3a263d6f273dd756449f677ca49f0825d50d037badeeaa7206a4c"}
Sep 18 20:11:22.681918 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:22.681848    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48"
Sep 18 20:11:22.682647 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:22.682626    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48"
Sep 18 20:11:22.722070 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:22.722026    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7747-8c2db0" podStartSLOduration=1.969013208 podStartE2EDuration="2.722010791s" totalImagesPullingTime="752.997583ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:11:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:11:22.721141157 +0000 UTC m=+867.674882322" watchObservedRunningTime="2026-09-18 20:11:22.722010791 +0000 UTC m=+867.675751962"
Sep 18 20:11:23.954159 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:23.954125    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7747-8c2db0"]
Sep 18 20:11:23.972258 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:23.972230    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7747-8c2db0"]
Sep 18 20:11:24.147049 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:24.147010    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7747-45ebd2"]
Sep 18 20:11:24.147531 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:24.147511    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a79b8d82-526b-45d0-844b-dc0c2bf6ee98" containerName="test-curl-7747-8c2db0"
Sep 18 20:11:24.147531 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:24.147532    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="a79b8d82-526b-45d0-844b-dc0c2bf6ee98" containerName="test-curl-7747-8c2db0"
Sep 18 20:11:24.147703 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:24.147648    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="a79b8d82-526b-45d0-844b-dc0c2bf6ee98" containerName="test-curl-7747-8c2db0"
Sep 18 20:11:24.150530 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:24.150509    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7747-45ebd2"
Sep 18 20:11:24.165558 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:24.165533    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7747-45ebd2"]
Sep 18 20:11:24.227005 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:24.226939    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dktdw\" (UniqueName: \"kubernetes.io/projected/70f2937f-37e4-457b-b6a5-9d2ab5f88eb5-kube-api-access-dktdw\") pod \"test-curl-7747-45ebd2\" (UID: \"70f2937f-37e4-457b-b6a5-9d2ab5f88eb5\") " pod="opendatahub/test-curl-7747-45ebd2"
Sep 18 20:11:24.327373 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:24.327344    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dktdw\" (UniqueName: \"kubernetes.io/projected/70f2937f-37e4-457b-b6a5-9d2ab5f88eb5-kube-api-access-dktdw\") pod \"test-curl-7747-45ebd2\" (UID: \"70f2937f-37e4-457b-b6a5-9d2ab5f88eb5\") " pod="opendatahub/test-curl-7747-45ebd2"
Sep 18 20:11:24.348461 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:24.348429    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dktdw\" (UniqueName: \"kubernetes.io/projected/70f2937f-37e4-457b-b6a5-9d2ab5f88eb5-kube-api-access-dktdw\") pod \"test-curl-7747-45ebd2\" (UID: \"70f2937f-37e4-457b-b6a5-9d2ab5f88eb5\") " pod="opendatahub/test-curl-7747-45ebd2"
Sep 18 20:11:24.459530 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:24.459506    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7747-45ebd2"
Sep 18 20:11:24.692642 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:24.692609    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7747-8c2db0" podUID="a79b8d82-526b-45d0-844b-dc0c2bf6ee98" containerName="test-curl-7747-8c2db0" containerID="cri-o://d5dd007b1fc3a263d6f273dd756449f677ca49f0825d50d037badeeaa7206a4c" gracePeriod=2
Sep 18 20:11:24.699294 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:11:24.699268    2549 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7747-8c2db0\" is forbidden: User \"system:node:ip-10-0-140-178.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-178.ec2.internal' and this object" podUID="a79b8d82-526b-45d0-844b-dc0c2bf6ee98" pod="opendatahub/test-curl-7747-8c2db0"
Sep 18 20:11:24.745264 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:24.745236    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7747-45ebd2"]
Sep 18 20:11:24.745446 ip-10-0-140-178 kubenswrapper[2549]: W0918 20:11:24.745421    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod70f2937f_37e4_457b_b6a5_9d2ab5f88eb5.slice/crio-7884a099ebb9f1722a0ee5b7758b30bda1e9dbf5e259a65560eba5444add30ed WatchSource:0}: Error finding container 7884a099ebb9f1722a0ee5b7758b30bda1e9dbf5e259a65560eba5444add30ed: Status 404 returned error can't find the container with id 7884a099ebb9f1722a0ee5b7758b30bda1e9dbf5e259a65560eba5444add30ed
Sep 18 20:11:25.521368 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:11:25.521332    2549 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7747-8c2db0\" is forbidden: User \"system:node:ip-10-0-140-178.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-178.ec2.internal' and this object" podUID="a79b8d82-526b-45d0-844b-dc0c2bf6ee98" pod="opendatahub/test-curl-7747-8c2db0"
Sep 18 20:11:25.699453 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:25.699421    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7747-45ebd2" event={"ID":"70f2937f-37e4-457b-b6a5-9d2ab5f88eb5","Type":"ContainerStarted","Data":"ff68e84ac38acc00ccbc6fd30d4a78ce7398cf34eeea6d5f32feff5a5e3099b6"}
Sep 18 20:11:25.699453 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:25.699467    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7747-45ebd2" event={"ID":"70f2937f-37e4-457b-b6a5-9d2ab5f88eb5","Type":"ContainerStarted","Data":"7884a099ebb9f1722a0ee5b7758b30bda1e9dbf5e259a65560eba5444add30ed"}
Sep 18 20:11:26.639459 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:26.639405    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7747-45ebd2" podStartSLOduration=2.6393760029999997 podStartE2EDuration="2.639376003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:11:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:11:25.735888087 +0000 UTC m=+870.689629254" watchObservedRunningTime="2026-09-18 20:11:26.639376003 +0000 UTC m=+871.593117171"
Sep 18 20:11:26.639938 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:26.639792    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7747-45ebd2"]
Sep 18 20:11:26.655004 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:26.654970    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7747-45ebd2"]
Sep 18 20:11:26.705619 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:11:26.705592    2549 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7747-45ebd2" container="test-curl-7747-45ebd2"
Sep 18 20:11:26.884229 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:26.884204    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7747-8c2db0"
Sep 18 20:11:26.953462 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:26.953396    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s58lc\" (UniqueName: \"kubernetes.io/projected/a79b8d82-526b-45d0-844b-dc0c2bf6ee98-kube-api-access-s58lc\") pod \"a79b8d82-526b-45d0-844b-dc0c2bf6ee98\" (UID: \"a79b8d82-526b-45d0-844b-dc0c2bf6ee98\") "
Sep 18 20:11:26.955520 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:26.955491    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a79b8d82-526b-45d0-844b-dc0c2bf6ee98-kube-api-access-s58lc" (OuterVolumeSpecName: "kube-api-access-s58lc") pod "a79b8d82-526b-45d0-844b-dc0c2bf6ee98" (UID: "a79b8d82-526b-45d0-844b-dc0c2bf6ee98"). InnerVolumeSpecName "kube-api-access-s58lc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:11:27.054579 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:27.054546    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s58lc\" (UniqueName: \"kubernetes.io/projected/a79b8d82-526b-45d0-844b-dc0c2bf6ee98-kube-api-access-s58lc\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:11:27.527530 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:27.527498    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a79b8d82-526b-45d0-844b-dc0c2bf6ee98" path="/var/lib/kubelet/pods/a79b8d82-526b-45d0-844b-dc0c2bf6ee98/volumes"
Sep 18 20:11:27.711776 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:27.711742    2549 generic.go:356] "Generic (PLEG): container finished" podID="a79b8d82-526b-45d0-844b-dc0c2bf6ee98" containerID="d5dd007b1fc3a263d6f273dd756449f677ca49f0825d50d037badeeaa7206a4c" exitCode=137
Sep 18 20:11:27.712197 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:27.711803    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7747-8c2db0"
Sep 18 20:11:27.712197 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:27.711843    2549 scope.go:117] "RemoveContainer" containerID="d5dd007b1fc3a263d6f273dd756449f677ca49f0825d50d037badeeaa7206a4c"
Sep 18 20:11:27.712197 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:27.712078    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7747-45ebd2" podUID="70f2937f-37e4-457b-b6a5-9d2ab5f88eb5" containerName="test-curl-7747-45ebd2" containerID="cri-o://ff68e84ac38acc00ccbc6fd30d4a78ce7398cf34eeea6d5f32feff5a5e3099b6" gracePeriod=2
Sep 18 20:11:27.720418 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:27.720369    2549 scope.go:117] "RemoveContainer" containerID="d5dd007b1fc3a263d6f273dd756449f677ca49f0825d50d037badeeaa7206a4c"
Sep 18 20:11:27.720705 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:11:27.720684    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5dd007b1fc3a263d6f273dd756449f677ca49f0825d50d037badeeaa7206a4c\": container with ID starting with d5dd007b1fc3a263d6f273dd756449f677ca49f0825d50d037badeeaa7206a4c not found: ID does not exist" containerID="d5dd007b1fc3a263d6f273dd756449f677ca49f0825d50d037badeeaa7206a4c"
Sep 18 20:11:27.720764 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:27.720714    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5dd007b1fc3a263d6f273dd756449f677ca49f0825d50d037badeeaa7206a4c"} err="failed to get container status \"d5dd007b1fc3a263d6f273dd756449f677ca49f0825d50d037badeeaa7206a4c\": rpc error: code = NotFound desc = could not find container \"d5dd007b1fc3a263d6f273dd756449f677ca49f0825d50d037badeeaa7206a4c\": container with ID starting with d5dd007b1fc3a263d6f273dd756449f677ca49f0825d50d037badeeaa7206a4c not found: ID does not exist"
Sep 18 20:11:27.815912 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:27.815832    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr"]
Sep 18 20:11:27.816225 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:27.816213    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70f2937f-37e4-457b-b6a5-9d2ab5f88eb5" containerName="test-curl-7747-45ebd2"
Sep 18 20:11:27.816268 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:27.816227    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="70f2937f-37e4-457b-b6a5-9d2ab5f88eb5" containerName="test-curl-7747-45ebd2"
Sep 18 20:11:27.816308 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:27.816299    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="70f2937f-37e4-457b-b6a5-9d2ab5f88eb5" containerName="test-curl-7747-45ebd2"
Sep 18 20:11:27.820467 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:27.820453    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr"
Sep 18 20:11:27.839352 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:27.839128    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-d0d77c-openshift-default-dockercfg-mt7fl\""
Sep 18 20:11:27.867415 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:27.867370    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr"]
Sep 18 20:11:27.962645 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:27.962613    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eeadd113-df76-4c65-83e6-c99756b64684-istio-podinfo\") pod \"e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr\" (UID: \"eeadd113-df76-4c65-83e6-c99756b64684\") " pod="openshift-ingress/e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr"
Sep 18 20:11:27.962796 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:27.962662    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeadd113-df76-4c65-83e6-c99756b64684-workload-socket\") pod \"e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr\" (UID: \"eeadd113-df76-4c65-83e6-c99756b64684\") " pod="openshift-ingress/e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr"
Sep 18 20:11:27.962796 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:27.962685    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eeadd113-df76-4c65-83e6-c99756b64684-istio-data\") pod \"e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr\" (UID: \"eeadd113-df76-4c65-83e6-c99756b64684\") " pod="openshift-ingress/e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr"
Sep 18 20:11:27.962796 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:27.962734    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eeadd113-df76-4c65-83e6-c99756b64684-workload-certs\") pod \"e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr\" (UID: \"eeadd113-df76-4c65-83e6-c99756b64684\") " pod="openshift-ingress/e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr"
Sep 18 20:11:27.962796 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:27.962751    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eeadd113-df76-4c65-83e6-c99756b64684-istio-token\") pod \"e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr\" (UID: \"eeadd113-df76-4c65-83e6-c99756b64684\") " pod="openshift-ingress/e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr"
Sep 18 20:11:27.962796 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:27.962767    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eeadd113-df76-4c65-83e6-c99756b64684-istiod-ca-cert\") pod \"e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr\" (UID: \"eeadd113-df76-4c65-83e6-c99756b64684\") " pod="openshift-ingress/e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr"
Sep 18 20:11:27.962796 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:27.962784    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeadd113-df76-4c65-83e6-c99756b64684-credential-socket\") pod \"e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr\" (UID: \"eeadd113-df76-4c65-83e6-c99756b64684\") " pod="openshift-ingress/e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr"
Sep 18 20:11:27.962992 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:27.962812    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eeadd113-df76-4c65-83e6-c99756b64684-istio-envoy\") pod \"e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr\" (UID: \"eeadd113-df76-4c65-83e6-c99756b64684\") " pod="openshift-ingress/e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr"
Sep 18 20:11:27.962992 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:27.962881    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6gg9\" (UniqueName: \"kubernetes.io/projected/eeadd113-df76-4c65-83e6-c99756b64684-kube-api-access-l6gg9\") pod \"e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr\" (UID: \"eeadd113-df76-4c65-83e6-c99756b64684\") " pod="openshift-ingress/e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr"
Sep 18 20:11:28.063931 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:28.063898    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeadd113-df76-4c65-83e6-c99756b64684-credential-socket\") pod \"e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr\" (UID: \"eeadd113-df76-4c65-83e6-c99756b64684\") " pod="openshift-ingress/e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr"
Sep 18 20:11:28.064177 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:28.063942    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eeadd113-df76-4c65-83e6-c99756b64684-istio-envoy\") pod \"e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr\" (UID: \"eeadd113-df76-4c65-83e6-c99756b64684\") " pod="openshift-ingress/e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr"
Sep 18 20:11:28.064177 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:28.063964    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6gg9\" (UniqueName: \"kubernetes.io/projected/eeadd113-df76-4c65-83e6-c99756b64684-kube-api-access-l6gg9\") pod \"e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr\" (UID: \"eeadd113-df76-4c65-83e6-c99756b64684\") " pod="openshift-ingress/e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr"
Sep 18 20:11:28.064177 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:28.064013    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eeadd113-df76-4c65-83e6-c99756b64684-istio-podinfo\") pod \"e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr\" (UID: \"eeadd113-df76-4c65-83e6-c99756b64684\") " pod="openshift-ingress/e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr"
Sep 18 20:11:28.064177 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:28.064048    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeadd113-df76-4c65-83e6-c99756b64684-workload-socket\") pod \"e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr\" (UID: \"eeadd113-df76-4c65-83e6-c99756b64684\") " pod="openshift-ingress/e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr"
Sep 18 20:11:28.064177 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:28.064061    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eeadd113-df76-4c65-83e6-c99756b64684-istio-data\") pod \"e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr\" (UID: \"eeadd113-df76-4c65-83e6-c99756b64684\") " pod="openshift-ingress/e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr"
Sep 18 20:11:28.064496 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:28.064220    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eeadd113-df76-4c65-83e6-c99756b64684-workload-certs\") pod \"e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr\" (UID: \"eeadd113-df76-4c65-83e6-c99756b64684\") " pod="openshift-ingress/e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr"
Sep 18 20:11:28.064496 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:28.064264    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eeadd113-df76-4c65-83e6-c99756b64684-istio-token\") pod \"e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr\" (UID: \"eeadd113-df76-4c65-83e6-c99756b64684\") " pod="openshift-ingress/e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr"
Sep 18 20:11:28.064496 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:28.064290    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eeadd113-df76-4c65-83e6-c99756b64684-istiod-ca-cert\") pod \"e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr\" (UID: \"eeadd113-df76-4c65-83e6-c99756b64684\") " pod="openshift-ingress/e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr"
Sep 18 20:11:28.064496 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:28.064362    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeadd113-df76-4c65-83e6-c99756b64684-credential-socket\") pod \"e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr\" (UID: \"eeadd113-df76-4c65-83e6-c99756b64684\") " pod="openshift-ingress/e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr"
Sep 18 20:11:28.064496 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:28.064477    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eeadd113-df76-4c65-83e6-c99756b64684-istio-data\") pod \"e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr\" (UID: \"eeadd113-df76-4c65-83e6-c99756b64684\") " pod="openshift-ingress/e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr"
Sep 18 20:11:28.064759 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:28.064530    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeadd113-df76-4c65-83e6-c99756b64684-workload-socket\") pod \"e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr\" (UID: \"eeadd113-df76-4c65-83e6-c99756b64684\") " pod="openshift-ingress/e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr"
Sep 18 20:11:28.064759 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:28.064583    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eeadd113-df76-4c65-83e6-c99756b64684-workload-certs\") pod \"e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr\" (UID: \"eeadd113-df76-4c65-83e6-c99756b64684\") " pod="openshift-ingress/e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr"
Sep 18 20:11:28.065027 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:28.065006    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eeadd113-df76-4c65-83e6-c99756b64684-istiod-ca-cert\") pod \"e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr\" (UID: \"eeadd113-df76-4c65-83e6-c99756b64684\") " pod="openshift-ingress/e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr"
Sep 18 20:11:28.066604 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:28.066555    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eeadd113-df76-4c65-83e6-c99756b64684-istio-envoy\") pod \"e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr\" (UID: \"eeadd113-df76-4c65-83e6-c99756b64684\") " pod="openshift-ingress/e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr"
Sep 18 20:11:28.066604 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:28.066580    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eeadd113-df76-4c65-83e6-c99756b64684-istio-podinfo\") pod \"e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr\" (UID: \"eeadd113-df76-4c65-83e6-c99756b64684\") " pod="openshift-ingress/e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr"
Sep 18 20:11:28.089647 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:28.089616    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eeadd113-df76-4c65-83e6-c99756b64684-istio-token\") pod \"e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr\" (UID: \"eeadd113-df76-4c65-83e6-c99756b64684\") " pod="openshift-ingress/e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr"
Sep 18 20:11:28.089759 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:28.089743    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6gg9\" (UniqueName: \"kubernetes.io/projected/eeadd113-df76-4c65-83e6-c99756b64684-kube-api-access-l6gg9\") pod \"e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr\" (UID: \"eeadd113-df76-4c65-83e6-c99756b64684\") " pod="openshift-ingress/e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr"
Sep 18 20:11:28.131566 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:28.131207    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr"
Sep 18 20:11:28.429022 ip-10-0-140-178 kubenswrapper[2549]: W0918 20:11:28.428992    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeadd113_df76_4c65_83e6_c99756b64684.slice/crio-97f080c4979f7b0698bcc411ecc2dceddae3a91c247f6fa923b5bdefcf7ee05d WatchSource:0}: Error finding container 97f080c4979f7b0698bcc411ecc2dceddae3a91c247f6fa923b5bdefcf7ee05d: Status 404 returned error can't find the container with id 97f080c4979f7b0698bcc411ecc2dceddae3a91c247f6fa923b5bdefcf7ee05d
Sep 18 20:11:28.431279 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:28.431246    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:11:28.431413 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:28.431313    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:11:28.431413 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:28.431342    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:11:28.434845 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:28.434822    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr"]
Sep 18 20:11:28.725253 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:28.724955    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr" event={"ID":"eeadd113-df76-4c65-83e6-c99756b64684","Type":"ContainerStarted","Data":"105f0cea652267717428f2c9330d0b08b62f15c33a01d78affa90cab0d87152e"}
Sep 18 20:11:28.725253 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:28.725017    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr" event={"ID":"eeadd113-df76-4c65-83e6-c99756b64684","Type":"ContainerStarted","Data":"97f080c4979f7b0698bcc411ecc2dceddae3a91c247f6fa923b5bdefcf7ee05d"}
Sep 18 20:11:28.809884 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:28.809822    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr" podStartSLOduration=1.809805079 podStartE2EDuration="1.809805079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:11:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:11:28.806810387 +0000 UTC m=+873.760551556" watchObservedRunningTime="2026-09-18 20:11:28.809805079 +0000 UTC m=+873.763546246"
Sep 18 20:11:29.132588 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:29.132548    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr"
Sep 18 20:11:29.134141 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:29.134116    2549 patch_prober.go:28] interesting pod/e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Sep 18 20:11:29.134270 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:29.134174    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr" podUID="eeadd113-df76-4c65-83e6-c99756b64684" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Sep 18 20:11:29.890317 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:29.890289    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7747-45ebd2"
Sep 18 20:11:29.984560 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:29.984471    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dktdw\" (UniqueName: \"kubernetes.io/projected/70f2937f-37e4-457b-b6a5-9d2ab5f88eb5-kube-api-access-dktdw\") pod \"70f2937f-37e4-457b-b6a5-9d2ab5f88eb5\" (UID: \"70f2937f-37e4-457b-b6a5-9d2ab5f88eb5\") "
Sep 18 20:11:29.986486 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:29.986461    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70f2937f-37e4-457b-b6a5-9d2ab5f88eb5-kube-api-access-dktdw" (OuterVolumeSpecName: "kube-api-access-dktdw") pod "70f2937f-37e4-457b-b6a5-9d2ab5f88eb5" (UID: "70f2937f-37e4-457b-b6a5-9d2ab5f88eb5"). InnerVolumeSpecName "kube-api-access-dktdw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:11:30.085284 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:30.085255    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dktdw\" (UniqueName: \"kubernetes.io/projected/70f2937f-37e4-457b-b6a5-9d2ab5f88eb5-kube-api-access-dktdw\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:11:30.135947 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:30.135916    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr"
Sep 18 20:11:30.738657 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:30.738621    2549 generic.go:356] "Generic (PLEG): container finished" podID="70f2937f-37e4-457b-b6a5-9d2ab5f88eb5" containerID="ff68e84ac38acc00ccbc6fd30d4a78ce7398cf34eeea6d5f32feff5a5e3099b6" exitCode=137
Sep 18 20:11:30.738832 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:30.738667    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7747-45ebd2"
Sep 18 20:11:30.738832 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:30.738715    2549 scope.go:117] "RemoveContainer" containerID="ff68e84ac38acc00ccbc6fd30d4a78ce7398cf34eeea6d5f32feff5a5e3099b6"
Sep 18 20:11:30.739214 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:30.739190    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr"
Sep 18 20:11:30.740194 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:30.740177    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr"
Sep 18 20:11:30.747970 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:30.747955    2549 scope.go:117] "RemoveContainer" containerID="ff68e84ac38acc00ccbc6fd30d4a78ce7398cf34eeea6d5f32feff5a5e3099b6"
Sep 18 20:11:30.748234 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:11:30.748215    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff68e84ac38acc00ccbc6fd30d4a78ce7398cf34eeea6d5f32feff5a5e3099b6\": container with ID starting with ff68e84ac38acc00ccbc6fd30d4a78ce7398cf34eeea6d5f32feff5a5e3099b6 not found: ID does not exist" containerID="ff68e84ac38acc00ccbc6fd30d4a78ce7398cf34eeea6d5f32feff5a5e3099b6"
Sep 18 20:11:30.748282 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:30.748241    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff68e84ac38acc00ccbc6fd30d4a78ce7398cf34eeea6d5f32feff5a5e3099b6"} err="failed to get container status \"ff68e84ac38acc00ccbc6fd30d4a78ce7398cf34eeea6d5f32feff5a5e3099b6\": rpc error: code = NotFound desc = could not find container \"ff68e84ac38acc00ccbc6fd30d4a78ce7398cf34eeea6d5f32feff5a5e3099b6\": container with ID starting with ff68e84ac38acc00ccbc6fd30d4a78ce7398cf34eeea6d5f32feff5a5e3099b6 not found: ID does not exist"
Sep 18 20:11:30.812173 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:11:30.812137    2549 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7747-45ebd2\" is forbidden: User \"system:node:ip-10-0-140-178.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-178.ec2.internal' and this object" podUID="70f2937f-37e4-457b-b6a5-9d2ab5f88eb5" pod="opendatahub/test-curl-7747-45ebd2"
Sep 18 20:11:31.521933 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:31.521900    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70f2937f-37e4-457b-b6a5-9d2ab5f88eb5" path="/var/lib/kubelet/pods/70f2937f-37e4-457b-b6a5-9d2ab5f88eb5/volumes"
Sep 18 20:11:33.350032 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:33.349987    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:11:36.257353 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:36.257318    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-0a7e3fae-openshift-default-656fdc8c44-vp26n"]
Sep 18 20:11:36.263792 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:36.263764    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0a7e3fae-openshift-default-656fdc8c44-vp26n"
Sep 18 20:11:36.271155 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:36.271132    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-0a7e3fae-openshift-default-dockercfg-8nm94\""
Sep 18 20:11:36.315660 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:36.315627    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0a7e3fae-openshift-default-656fdc8c44-vp26n"]
Sep 18 20:11:36.345952 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:36.345908    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7e73c75-ba46-4a2b-98c9-974242035854-istiod-ca-cert\") pod \"e2e-ait-0a7e3fae-openshift-default-656fdc8c44-vp26n\" (UID: \"e7e73c75-ba46-4a2b-98c9-974242035854\") " pod="openshift-ingress/e2e-ait-0a7e3fae-openshift-default-656fdc8c44-vp26n"
Sep 18 20:11:36.346288 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:36.346265    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7e73c75-ba46-4a2b-98c9-974242035854-credential-socket\") pod \"e2e-ait-0a7e3fae-openshift-default-656fdc8c44-vp26n\" (UID: \"e7e73c75-ba46-4a2b-98c9-974242035854\") " pod="openshift-ingress/e2e-ait-0a7e3fae-openshift-default-656fdc8c44-vp26n"
Sep 18 20:11:36.346445 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:36.346428    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7e73c75-ba46-4a2b-98c9-974242035854-istio-data\") pod \"e2e-ait-0a7e3fae-openshift-default-656fdc8c44-vp26n\" (UID: \"e7e73c75-ba46-4a2b-98c9-974242035854\") " pod="openshift-ingress/e2e-ait-0a7e3fae-openshift-default-656fdc8c44-vp26n"
Sep 18 20:11:36.346569 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:36.346555    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7e73c75-ba46-4a2b-98c9-974242035854-istio-podinfo\") pod \"e2e-ait-0a7e3fae-openshift-default-656fdc8c44-vp26n\" (UID: \"e7e73c75-ba46-4a2b-98c9-974242035854\") " pod="openshift-ingress/e2e-ait-0a7e3fae-openshift-default-656fdc8c44-vp26n"
Sep 18 20:11:36.346676 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:36.346663    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4t77\" (UniqueName: \"kubernetes.io/projected/e7e73c75-ba46-4a2b-98c9-974242035854-kube-api-access-r4t77\") pod \"e2e-ait-0a7e3fae-openshift-default-656fdc8c44-vp26n\" (UID: \"e7e73c75-ba46-4a2b-98c9-974242035854\") " pod="openshift-ingress/e2e-ait-0a7e3fae-openshift-default-656fdc8c44-vp26n"
Sep 18 20:11:36.346827 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:36.346814    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7e73c75-ba46-4a2b-98c9-974242035854-workload-socket\") pod \"e2e-ait-0a7e3fae-openshift-default-656fdc8c44-vp26n\" (UID: \"e7e73c75-ba46-4a2b-98c9-974242035854\") " pod="openshift-ingress/e2e-ait-0a7e3fae-openshift-default-656fdc8c44-vp26n"
Sep 18 20:11:36.346925 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:36.346913    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7e73c75-ba46-4a2b-98c9-974242035854-istio-envoy\") pod \"e2e-ait-0a7e3fae-openshift-default-656fdc8c44-vp26n\" (UID: \"e7e73c75-ba46-4a2b-98c9-974242035854\") " pod="openshift-ingress/e2e-ait-0a7e3fae-openshift-default-656fdc8c44-vp26n"
Sep 18 20:11:36.347047 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:36.347032    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7e73c75-ba46-4a2b-98c9-974242035854-workload-certs\") pod \"e2e-ait-0a7e3fae-openshift-default-656fdc8c44-vp26n\" (UID: \"e7e73c75-ba46-4a2b-98c9-974242035854\") " pod="openshift-ingress/e2e-ait-0a7e3fae-openshift-default-656fdc8c44-vp26n"
Sep 18 20:11:36.347178 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:36.347164    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7e73c75-ba46-4a2b-98c9-974242035854-istio-token\") pod \"e2e-ait-0a7e3fae-openshift-default-656fdc8c44-vp26n\" (UID: \"e7e73c75-ba46-4a2b-98c9-974242035854\") " pod="openshift-ingress/e2e-ait-0a7e3fae-openshift-default-656fdc8c44-vp26n"
Sep 18 20:11:36.449933 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:36.449895    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7e73c75-ba46-4a2b-98c9-974242035854-workload-certs\") pod \"e2e-ait-0a7e3fae-openshift-default-656fdc8c44-vp26n\" (UID: \"e7e73c75-ba46-4a2b-98c9-974242035854\") " pod="openshift-ingress/e2e-ait-0a7e3fae-openshift-default-656fdc8c44-vp26n"
Sep 18 20:11:36.450674 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:36.450649    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7e73c75-ba46-4a2b-98c9-974242035854-istio-token\") pod \"e2e-ait-0a7e3fae-openshift-default-656fdc8c44-vp26n\" (UID: \"e7e73c75-ba46-4a2b-98c9-974242035854\") " pod="openshift-ingress/e2e-ait-0a7e3fae-openshift-default-656fdc8c44-vp26n"
Sep 18 20:11:36.451215 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:36.451194    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7e73c75-ba46-4a2b-98c9-974242035854-istiod-ca-cert\") pod \"e2e-ait-0a7e3fae-openshift-default-656fdc8c44-vp26n\" (UID: \"e7e73c75-ba46-4a2b-98c9-974242035854\") " pod="openshift-ingress/e2e-ait-0a7e3fae-openshift-default-656fdc8c44-vp26n"
Sep 18 20:11:36.452067 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:36.452049    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7e73c75-ba46-4a2b-98c9-974242035854-credential-socket\") pod \"e2e-ait-0a7e3fae-openshift-default-656fdc8c44-vp26n\" (UID: \"e7e73c75-ba46-4a2b-98c9-974242035854\") " pod="openshift-ingress/e2e-ait-0a7e3fae-openshift-default-656fdc8c44-vp26n"
Sep 18 20:11:36.452502 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:36.452485    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7e73c75-ba46-4a2b-98c9-974242035854-istio-data\") pod \"e2e-ait-0a7e3fae-openshift-default-656fdc8c44-vp26n\" (UID: \"e7e73c75-ba46-4a2b-98c9-974242035854\") " pod="openshift-ingress/e2e-ait-0a7e3fae-openshift-default-656fdc8c44-vp26n"
Sep 18 20:11:36.452889 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:36.452872    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7e73c75-ba46-4a2b-98c9-974242035854-istio-podinfo\") pod \"e2e-ait-0a7e3fae-openshift-default-656fdc8c44-vp26n\" (UID: \"e7e73c75-ba46-4a2b-98c9-974242035854\") " pod="openshift-ingress/e2e-ait-0a7e3fae-openshift-default-656fdc8c44-vp26n"
Sep 18 20:11:36.453452 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:36.453431    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4t77\" (UniqueName: \"kubernetes.io/projected/e7e73c75-ba46-4a2b-98c9-974242035854-kube-api-access-r4t77\") pod \"e2e-ait-0a7e3fae-openshift-default-656fdc8c44-vp26n\" (UID: \"e7e73c75-ba46-4a2b-98c9-974242035854\") " pod="openshift-ingress/e2e-ait-0a7e3fae-openshift-default-656fdc8c44-vp26n"
Sep 18 20:11:36.453882 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:36.451987    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7e73c75-ba46-4a2b-98c9-974242035854-istiod-ca-cert\") pod \"e2e-ait-0a7e3fae-openshift-default-656fdc8c44-vp26n\" (UID: \"e7e73c75-ba46-4a2b-98c9-974242035854\") " pod="openshift-ingress/e2e-ait-0a7e3fae-openshift-default-656fdc8c44-vp26n"
Sep 18 20:11:36.453993 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:36.452828    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7e73c75-ba46-4a2b-98c9-974242035854-istio-data\") pod \"e2e-ait-0a7e3fae-openshift-default-656fdc8c44-vp26n\" (UID: \"e7e73c75-ba46-4a2b-98c9-974242035854\") " pod="openshift-ingress/e2e-ait-0a7e3fae-openshift-default-656fdc8c44-vp26n"
Sep 18 20:11:36.454082 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:36.450559    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7e73c75-ba46-4a2b-98c9-974242035854-workload-certs\") pod \"e2e-ait-0a7e3fae-openshift-default-656fdc8c44-vp26n\" (UID: \"e7e73c75-ba46-4a2b-98c9-974242035854\") " pod="openshift-ingress/e2e-ait-0a7e3fae-openshift-default-656fdc8c44-vp26n"
Sep 18 20:11:36.454157 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:36.452426    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7e73c75-ba46-4a2b-98c9-974242035854-credential-socket\") pod \"e2e-ait-0a7e3fae-openshift-default-656fdc8c44-vp26n\" (UID: \"e7e73c75-ba46-4a2b-98c9-974242035854\") " pod="openshift-ingress/e2e-ait-0a7e3fae-openshift-default-656fdc8c44-vp26n"
Sep 18 20:11:36.454348 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:36.454331    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7e73c75-ba46-4a2b-98c9-974242035854-workload-socket\") pod \"e2e-ait-0a7e3fae-openshift-default-656fdc8c44-vp26n\" (UID: \"e7e73c75-ba46-4a2b-98c9-974242035854\") " pod="openshift-ingress/e2e-ait-0a7e3fae-openshift-default-656fdc8c44-vp26n"
Sep 18 20:11:36.454792 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:36.454765    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7e73c75-ba46-4a2b-98c9-974242035854-istio-envoy\") pod \"e2e-ait-0a7e3fae-openshift-default-656fdc8c44-vp26n\" (UID: \"e7e73c75-ba46-4a2b-98c9-974242035854\") " pod="openshift-ingress/e2e-ait-0a7e3fae-openshift-default-656fdc8c44-vp26n"
Sep 18 20:11:36.455236 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:36.454701    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7e73c75-ba46-4a2b-98c9-974242035854-workload-socket\") pod \"e2e-ait-0a7e3fae-openshift-default-656fdc8c44-vp26n\" (UID: \"e7e73c75-ba46-4a2b-98c9-974242035854\") " pod="openshift-ingress/e2e-ait-0a7e3fae-openshift-default-656fdc8c44-vp26n"
Sep 18 20:11:36.457819 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:36.457782    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7e73c75-ba46-4a2b-98c9-974242035854-istio-envoy\") pod \"e2e-ait-0a7e3fae-openshift-default-656fdc8c44-vp26n\" (UID: \"e7e73c75-ba46-4a2b-98c9-974242035854\") " pod="openshift-ingress/e2e-ait-0a7e3fae-openshift-default-656fdc8c44-vp26n"
Sep 18 20:11:36.461885 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:36.461864    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7e73c75-ba46-4a2b-98c9-974242035854-istio-podinfo\") pod \"e2e-ait-0a7e3fae-openshift-default-656fdc8c44-vp26n\" (UID: \"e7e73c75-ba46-4a2b-98c9-974242035854\") " pod="openshift-ingress/e2e-ait-0a7e3fae-openshift-default-656fdc8c44-vp26n"
Sep 18 20:11:36.487794 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:36.487736    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7e73c75-ba46-4a2b-98c9-974242035854-istio-token\") pod \"e2e-ait-0a7e3fae-openshift-default-656fdc8c44-vp26n\" (UID: \"e7e73c75-ba46-4a2b-98c9-974242035854\") " pod="openshift-ingress/e2e-ait-0a7e3fae-openshift-default-656fdc8c44-vp26n"
Sep 18 20:11:36.492065 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:36.492006    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4t77\" (UniqueName: \"kubernetes.io/projected/e7e73c75-ba46-4a2b-98c9-974242035854-kube-api-access-r4t77\") pod \"e2e-ait-0a7e3fae-openshift-default-656fdc8c44-vp26n\" (UID: \"e7e73c75-ba46-4a2b-98c9-974242035854\") " pod="openshift-ingress/e2e-ait-0a7e3fae-openshift-default-656fdc8c44-vp26n"
Sep 18 20:11:36.602526 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:36.602484    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0a7e3fae-openshift-default-656fdc8c44-vp26n"
Sep 18 20:11:37.010765 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:37.010229    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0a7e3fae-openshift-default-656fdc8c44-vp26n"]
Sep 18 20:11:37.019572 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:37.018199    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:11:37.019572 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:37.018299    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:11:37.019572 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:37.018358    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:11:37.889019 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:37.888971    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0a7e3fae-openshift-default-656fdc8c44-vp26n" event={"ID":"e7e73c75-ba46-4a2b-98c9-974242035854","Type":"ContainerStarted","Data":"30e5c724ef615cd7308c151509887b159c490f5385e412ba90f704e15530aa69"}
Sep 18 20:11:37.889702 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:37.889667    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0a7e3fae-openshift-default-656fdc8c44-vp26n" event={"ID":"e7e73c75-ba46-4a2b-98c9-974242035854","Type":"ContainerStarted","Data":"77d1e70253712c1ca1c5005658446cce63c9d633b0ca27a681634427f2bd07b7"}
Sep 18 20:11:38.604928 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:38.604887    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-0a7e3fae-openshift-default-656fdc8c44-vp26n"
Sep 18 20:11:38.614052 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:38.614015    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-0a7e3fae-openshift-default-656fdc8c44-vp26n"
Sep 18 20:11:38.693792 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:38.693714    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-0a7e3fae-openshift-default-656fdc8c44-vp26n" podStartSLOduration=2.693678002 podStartE2EDuration="2.693678002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:11:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:11:37.96161986 +0000 UTC m=+882.915361030" watchObservedRunningTime="2026-09-18 20:11:38.693678002 +0000 UTC m=+883.647419174"
Sep 18 20:11:38.898476 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:38.898366    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-0a7e3fae-openshift-default-656fdc8c44-vp26n"
Sep 18 20:11:38.903892 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:38.903861    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-0a7e3fae-openshift-default-656fdc8c44-vp26n"
Sep 18 20:11:45.947333 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:45.947299    2549 generic.go:356] "Generic (PLEG): container finished" podID="374c8e70-b164-4b98-8c5d-f7994db126f2" containerID="72d061c1a843da03a5e6768c6bae567b761ff73cfe8163e9d982404a1b36b202" exitCode=0
Sep 18 20:11:45.947829 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:45.947395    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr" event={"ID":"374c8e70-b164-4b98-8c5d-f7994db126f2","Type":"ContainerDied","Data":"72d061c1a843da03a5e6768c6bae567b761ff73cfe8163e9d982404a1b36b202"}
Sep 18 20:11:45.947930 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:45.947910    2549 scope.go:117] "RemoveContainer" containerID="72d061c1a843da03a5e6768c6bae567b761ff73cfe8163e9d982404a1b36b202"
Sep 18 20:11:45.949150 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:45.949121    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:11:45.949259 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:45.949190    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:11:45.949259 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:45.949237    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:11:45.949346 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:45.949325    2549 generic.go:356] "Generic (PLEG): container finished" podID="f3d791d0-5f42-41d0-9ca2-05d11123c178" containerID="f816bfed502fbeec9741f216d42b39e63756affcc99a168fdb5a0fe07e0cbe37" exitCode=0
Sep 18 20:11:45.949439 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:45.949409    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk" event={"ID":"f3d791d0-5f42-41d0-9ca2-05d11123c178","Type":"ContainerDied","Data":"f816bfed502fbeec9741f216d42b39e63756affcc99a168fdb5a0fe07e0cbe37"}
Sep 18 20:11:45.949770 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:45.949754    2549 scope.go:117] "RemoveContainer" containerID="f816bfed502fbeec9741f216d42b39e63756affcc99a168fdb5a0fe07e0cbe37"
Sep 18 20:11:45.950862 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:45.950831    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:11:45.950965 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:45.950899    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:11:45.950965 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:45.950936    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:11:45.951232 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:45.951215    2549 generic.go:356] "Generic (PLEG): container finished" podID="e773b934-ce8f-43ce-899c-a2c425da0c33" containerID="0ecd23622c0958500be173983ee355784faaaf0494e238debce8c4acd460d3c1" exitCode=0
Sep 18 20:11:45.951326 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:45.951241    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48" event={"ID":"e773b934-ce8f-43ce-899c-a2c425da0c33","Type":"ContainerDied","Data":"0ecd23622c0958500be173983ee355784faaaf0494e238debce8c4acd460d3c1"}
Sep 18 20:11:45.951625 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:45.951547    2549 scope.go:117] "RemoveContainer" containerID="0ecd23622c0958500be173983ee355784faaaf0494e238debce8c4acd460d3c1"
Sep 18 20:11:45.952649 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:45.952611    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:11:45.952789 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:45.952685    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:11:45.952789 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:45.952731    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:11:46.963647 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:46.962587    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk" event={"ID":"f3d791d0-5f42-41d0-9ca2-05d11123c178","Type":"ContainerStarted","Data":"901490bdb2fa08401b24c7ab3d701dfa5390ed1ad355f911516ca9cdd50e5ead"}
Sep 18 20:11:46.966987 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:46.966958    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48" event={"ID":"e773b934-ce8f-43ce-899c-a2c425da0c33","Type":"ContainerStarted","Data":"3bedc9afbef4d4d01605b9660cc4639b3e70f994b85f0df2e97591fb2b004d29"}
Sep 18 20:11:46.970520 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:46.970494    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr" event={"ID":"374c8e70-b164-4b98-8c5d-f7994db126f2","Type":"ContainerStarted","Data":"92a6cfb5fdec34475d9f59d4357957edbc092711a9b5e350d2e82e3a3b91c183"}
Sep 18 20:11:47.199987 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:47.199940    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk"
Sep 18 20:11:47.201198 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:47.201173    2549 patch_prober.go:28] interesting pod/e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Sep 18 20:11:47.201259 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:47.201232    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk" podUID="f3d791d0-5f42-41d0-9ca2-05d11123c178" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Sep 18 20:11:47.518245 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:47.518209    2549 patch_prober.go:28] interesting pod/e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Sep 18 20:11:47.518435 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:47.518273    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr" podUID="374c8e70-b164-4b98-8c5d-f7994db126f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Sep 18 20:11:47.521341 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:47.521320    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr"
Sep 18 20:11:47.682944 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:47.682906    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48"
Sep 18 20:11:47.683366 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:47.683334    2549 patch_prober.go:28] interesting pod/e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Sep 18 20:11:47.683518 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:47.683414    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48" podUID="e773b934-ce8f-43ce-899c-a2c425da0c33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Sep 18 20:11:47.724277 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:47.722581    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm"]
Sep 18 20:11:47.726173 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:47.726153    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm"
Sep 18 20:11:47.735962 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:47.735940    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-c34fad-openshift-default-dockercfg-lfvkh\""
Sep 18 20:11:47.808556 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:47.808457    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm"]
Sep 18 20:11:47.877407 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:47.869959    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2cb0038-fbd2-4eff-a731-227ed00eed57-istio-podinfo\") pod \"e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm\" (UID: \"d2cb0038-fbd2-4eff-a731-227ed00eed57\") " pod="openshift-ingress/e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm"
Sep 18 20:11:47.877407 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:47.870038    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2cb0038-fbd2-4eff-a731-227ed00eed57-credential-socket\") pod \"e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm\" (UID: \"d2cb0038-fbd2-4eff-a731-227ed00eed57\") " pod="openshift-ingress/e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm"
Sep 18 20:11:47.877407 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:47.870083    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2cb0038-fbd2-4eff-a731-227ed00eed57-istio-token\") pod \"e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm\" (UID: \"d2cb0038-fbd2-4eff-a731-227ed00eed57\") " pod="openshift-ingress/e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm"
Sep 18 20:11:47.877407 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:47.870115    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhnj4\" (UniqueName: \"kubernetes.io/projected/d2cb0038-fbd2-4eff-a731-227ed00eed57-kube-api-access-fhnj4\") pod \"e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm\" (UID: \"d2cb0038-fbd2-4eff-a731-227ed00eed57\") " pod="openshift-ingress/e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm"
Sep 18 20:11:47.877407 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:47.870140    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2cb0038-fbd2-4eff-a731-227ed00eed57-istio-data\") pod \"e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm\" (UID: \"d2cb0038-fbd2-4eff-a731-227ed00eed57\") " pod="openshift-ingress/e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm"
Sep 18 20:11:47.877407 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:47.870185    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2cb0038-fbd2-4eff-a731-227ed00eed57-istiod-ca-cert\") pod \"e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm\" (UID: \"d2cb0038-fbd2-4eff-a731-227ed00eed57\") " pod="openshift-ingress/e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm"
Sep 18 20:11:47.877407 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:47.870226    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2cb0038-fbd2-4eff-a731-227ed00eed57-workload-socket\") pod \"e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm\" (UID: \"d2cb0038-fbd2-4eff-a731-227ed00eed57\") " pod="openshift-ingress/e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm"
Sep 18 20:11:47.877407 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:47.870285    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2cb0038-fbd2-4eff-a731-227ed00eed57-workload-certs\") pod \"e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm\" (UID: \"d2cb0038-fbd2-4eff-a731-227ed00eed57\") " pod="openshift-ingress/e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm"
Sep 18 20:11:47.877407 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:47.870320    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2cb0038-fbd2-4eff-a731-227ed00eed57-istio-envoy\") pod \"e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm\" (UID: \"d2cb0038-fbd2-4eff-a731-227ed00eed57\") " pod="openshift-ingress/e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm"
Sep 18 20:11:47.970932 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:47.970892    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2cb0038-fbd2-4eff-a731-227ed00eed57-workload-certs\") pod \"e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm\" (UID: \"d2cb0038-fbd2-4eff-a731-227ed00eed57\") " pod="openshift-ingress/e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm"
Sep 18 20:11:47.972072 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:47.972046    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2cb0038-fbd2-4eff-a731-227ed00eed57-istio-envoy\") pod \"e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm\" (UID: \"d2cb0038-fbd2-4eff-a731-227ed00eed57\") " pod="openshift-ingress/e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm"
Sep 18 20:11:47.972300 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:47.972282    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2cb0038-fbd2-4eff-a731-227ed00eed57-istio-podinfo\") pod \"e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm\" (UID: \"d2cb0038-fbd2-4eff-a731-227ed00eed57\") " pod="openshift-ingress/e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm"
Sep 18 20:11:47.972470 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:47.972451    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2cb0038-fbd2-4eff-a731-227ed00eed57-credential-socket\") pod \"e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm\" (UID: \"d2cb0038-fbd2-4eff-a731-227ed00eed57\") " pod="openshift-ingress/e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm"
Sep 18 20:11:47.972957 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:47.972938    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2cb0038-fbd2-4eff-a731-227ed00eed57-istio-token\") pod \"e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm\" (UID: \"d2cb0038-fbd2-4eff-a731-227ed00eed57\") " pod="openshift-ingress/e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm"
Sep 18 20:11:47.973072 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:47.973050    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2cb0038-fbd2-4eff-a731-227ed00eed57-credential-socket\") pod \"e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm\" (UID: \"d2cb0038-fbd2-4eff-a731-227ed00eed57\") " pod="openshift-ingress/e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm"
Sep 18 20:11:47.973154 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:47.971983    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2cb0038-fbd2-4eff-a731-227ed00eed57-workload-certs\") pod \"e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm\" (UID: \"d2cb0038-fbd2-4eff-a731-227ed00eed57\") " pod="openshift-ingress/e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm"
Sep 18 20:11:47.973259 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:47.973241    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhnj4\" (UniqueName: \"kubernetes.io/projected/d2cb0038-fbd2-4eff-a731-227ed00eed57-kube-api-access-fhnj4\") pod \"e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm\" (UID: \"d2cb0038-fbd2-4eff-a731-227ed00eed57\") " pod="openshift-ingress/e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm"
Sep 18 20:11:47.973356 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:47.973340    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2cb0038-fbd2-4eff-a731-227ed00eed57-istio-data\") pod \"e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm\" (UID: \"d2cb0038-fbd2-4eff-a731-227ed00eed57\") " pod="openshift-ingress/e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm"
Sep 18 20:11:47.973508 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:47.973494    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2cb0038-fbd2-4eff-a731-227ed00eed57-istiod-ca-cert\") pod \"e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm\" (UID: \"d2cb0038-fbd2-4eff-a731-227ed00eed57\") " pod="openshift-ingress/e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm"
Sep 18 20:11:47.973626 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:47.973612    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2cb0038-fbd2-4eff-a731-227ed00eed57-workload-socket\") pod \"e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm\" (UID: \"d2cb0038-fbd2-4eff-a731-227ed00eed57\") " pod="openshift-ingress/e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm"
Sep 18 20:11:47.974999 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:47.974729    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2cb0038-fbd2-4eff-a731-227ed00eed57-istiod-ca-cert\") pod \"e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm\" (UID: \"d2cb0038-fbd2-4eff-a731-227ed00eed57\") " pod="openshift-ingress/e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm"
Sep 18 20:11:47.975375 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:47.975355    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2cb0038-fbd2-4eff-a731-227ed00eed57-workload-socket\") pod \"e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm\" (UID: \"d2cb0038-fbd2-4eff-a731-227ed00eed57\") " pod="openshift-ingress/e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm"
Sep 18 20:11:47.977181 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:47.975720    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2cb0038-fbd2-4eff-a731-227ed00eed57-istio-envoy\") pod \"e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm\" (UID: \"d2cb0038-fbd2-4eff-a731-227ed00eed57\") " pod="openshift-ingress/e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm"
Sep 18 20:11:47.977181 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:47.976730    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2cb0038-fbd2-4eff-a731-227ed00eed57-istio-data\") pod \"e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm\" (UID: \"d2cb0038-fbd2-4eff-a731-227ed00eed57\") " pod="openshift-ingress/e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm"
Sep 18 20:11:47.982694 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:47.982669    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2cb0038-fbd2-4eff-a731-227ed00eed57-istio-podinfo\") pod \"e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm\" (UID: \"d2cb0038-fbd2-4eff-a731-227ed00eed57\") " pod="openshift-ingress/e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm"
Sep 18 20:11:47.985785 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:47.985737    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2cb0038-fbd2-4eff-a731-227ed00eed57-istio-token\") pod \"e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm\" (UID: \"d2cb0038-fbd2-4eff-a731-227ed00eed57\") " pod="openshift-ingress/e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm"
Sep 18 20:11:47.988498 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:47.986608    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhnj4\" (UniqueName: \"kubernetes.io/projected/d2cb0038-fbd2-4eff-a731-227ed00eed57-kube-api-access-fhnj4\") pod \"e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm\" (UID: \"d2cb0038-fbd2-4eff-a731-227ed00eed57\") " pod="openshift-ingress/e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm"
Sep 18 20:11:48.037698 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:48.037657    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm"
Sep 18 20:11:48.200779 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:48.200286    2549 patch_prober.go:28] interesting pod/e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Sep 18 20:11:48.200779 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:48.200345    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk" podUID="f3d791d0-5f42-41d0-9ca2-05d11123c178" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Sep 18 20:11:48.514277 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:48.514214    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm"]
Sep 18 20:11:48.517856 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:48.517818    2549 patch_prober.go:28] interesting pod/e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Sep 18 20:11:48.517985 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:48.517894    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr" podUID="374c8e70-b164-4b98-8c5d-f7994db126f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Sep 18 20:11:48.527404 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:48.524122    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:11:48.527404 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:48.524208    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:11:48.527404 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:48.524249    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:11:48.687305 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:48.684766    2549 patch_prober.go:28] interesting pod/e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Sep 18 20:11:48.687305 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:48.684827    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48" podUID="e773b934-ce8f-43ce-899c-a2c425da0c33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Sep 18 20:11:48.996006 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:48.995908    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm" event={"ID":"d2cb0038-fbd2-4eff-a731-227ed00eed57","Type":"ContainerStarted","Data":"474e7a33c8ee247f675562129318c3a865174d5d26ecbd96ea8cf2f6b027af8b"}
Sep 18 20:11:48.996006 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:48.995982    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm" event={"ID":"d2cb0038-fbd2-4eff-a731-227ed00eed57","Type":"ContainerStarted","Data":"67cbb3a04ba98372260ea8ec3e6b79741633246debfa21c35dea85d4407cffe8"}
Sep 18 20:11:49.023078 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:49.023010    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm" podStartSLOduration=2.022985113 podStartE2EDuration="2.022985113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:11:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:11:49.020136155 +0000 UTC m=+893.973877323" watchObservedRunningTime="2026-09-18 20:11:49.022985113 +0000 UTC m=+893.976726282"
Sep 18 20:11:49.040811 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:49.040763    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm"
Sep 18 20:11:49.041002 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:49.040914    2549 patch_prober.go:28] interesting pod/e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Sep 18 20:11:49.041002 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:49.040978    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm" podUID="d2cb0038-fbd2-4eff-a731-227ed00eed57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Sep 18 20:11:49.201060 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:49.201014    2549 patch_prober.go:28] interesting pod/e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Sep 18 20:11:49.201242 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:49.201086    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk" podUID="f3d791d0-5f42-41d0-9ca2-05d11123c178" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Sep 18 20:11:49.530407 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:49.529947    2549 patch_prober.go:28] interesting pod/e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Sep 18 20:11:49.530407 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:49.530024    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr" podUID="374c8e70-b164-4b98-8c5d-f7994db126f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Sep 18 20:11:49.683362 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:49.683320    2549 patch_prober.go:28] interesting pod/e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Sep 18 20:11:49.683580 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:49.683404    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48" podUID="e773b934-ce8f-43ce-899c-a2c425da0c33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Sep 18 20:11:50.052273 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:50.052231    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm"
Sep 18 20:11:50.200450 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:50.200405    2549 patch_prober.go:28] interesting pod/e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Sep 18 20:11:50.200625 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:50.200489    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk" podUID="f3d791d0-5f42-41d0-9ca2-05d11123c178" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Sep 18 20:11:50.517374 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:50.517329    2549 patch_prober.go:28] interesting pod/e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Sep 18 20:11:50.517604 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:50.517414    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr" podUID="374c8e70-b164-4b98-8c5d-f7994db126f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Sep 18 20:11:50.668424 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:50.667630    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk"
Sep 18 20:11:50.684102 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:50.683637    2549 patch_prober.go:28] interesting pod/e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Sep 18 20:11:50.684102 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:50.683707    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48" podUID="e773b934-ce8f-43ce-899c-a2c425da0c33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Sep 18 20:11:51.014846 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:51.014809    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm"
Sep 18 20:11:51.024233 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:51.024207    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm"
Sep 18 20:11:51.201939 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:51.201903    2549 patch_prober.go:28] interesting pod/e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Sep 18 20:11:51.202556 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:51.202503    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk" podUID="f3d791d0-5f42-41d0-9ca2-05d11123c178" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Sep 18 20:11:51.519449 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:51.519022    2549 patch_prober.go:28] interesting pod/e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Sep 18 20:11:51.519449 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:51.519081    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr" podUID="374c8e70-b164-4b98-8c5d-f7994db126f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Sep 18 20:11:51.675367 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:51.675323    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr"
Sep 18 20:11:51.683165 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:51.683129    2549 patch_prober.go:28] interesting pod/e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Sep 18 20:11:51.683449 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:51.683420    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48" podUID="e773b934-ce8f-43ce-899c-a2c425da0c33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Sep 18 20:11:52.201078 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:52.201033    2549 patch_prober.go:28] interesting pod/e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Sep 18 20:11:52.201257 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:52.201102    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk" podUID="f3d791d0-5f42-41d0-9ca2-05d11123c178" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Sep 18 20:11:52.517254 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:52.517152    2549 patch_prober.go:28] interesting pod/e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Sep 18 20:11:52.517878 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:52.517825    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr" podUID="374c8e70-b164-4b98-8c5d-f7994db126f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Sep 18 20:11:52.682708 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:52.682670    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48"
Sep 18 20:11:52.684439 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:52.684412    2549 patch_prober.go:28] interesting pod/e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Sep 18 20:11:52.684686 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:52.684647    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48" podUID="e773b934-ce8f-43ce-899c-a2c425da0c33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Sep 18 20:11:53.200759 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:53.200713    2549 patch_prober.go:28] interesting pod/e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Sep 18 20:11:53.200966 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:53.200784    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk" podUID="f3d791d0-5f42-41d0-9ca2-05d11123c178" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Sep 18 20:11:53.522278 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:53.522184    2549 patch_prober.go:28] interesting pod/e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Sep 18 20:11:53.522833 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:53.522799    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr" podUID="374c8e70-b164-4b98-8c5d-f7994db126f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Sep 18 20:11:53.683633 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:53.683596    2549 patch_prober.go:28] interesting pod/e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Sep 18 20:11:53.683980 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:53.683950    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48" podUID="e773b934-ce8f-43ce-899c-a2c425da0c33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Sep 18 20:11:54.342532 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:54.342500    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk"
Sep 18 20:11:54.343917 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:54.343897    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk"
Sep 18 20:11:54.517457 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:54.517427    2549 patch_prober.go:28] interesting pod/e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Sep 18 20:11:54.517619 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:54.517483    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr" podUID="374c8e70-b164-4b98-8c5d-f7994db126f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Sep 18 20:11:54.688109 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:54.688019    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48"
Sep 18 20:11:54.689330 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:54.689310    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48"
Sep 18 20:11:55.479234 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:55.479207    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-scn7p_a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48/ovn-acl-logging/0.log"
Sep 18 20:11:55.481995 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:55.481973    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-scn7p_a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48/ovn-acl-logging/0.log"
Sep 18 20:11:55.521826 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:55.521801    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr"
Sep 18 20:11:55.522915 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:55.522894    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr"
Sep 18 20:11:56.118936 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:56.118903    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:11:57.078576 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:57.076522    2549 generic.go:356] "Generic (PLEG): container finished" podID="eeadd113-df76-4c65-83e6-c99756b64684" containerID="105f0cea652267717428f2c9330d0b08b62f15c33a01d78affa90cab0d87152e" exitCode=0
Sep 18 20:11:57.078576 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:57.076720    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr" event={"ID":"eeadd113-df76-4c65-83e6-c99756b64684","Type":"ContainerDied","Data":"105f0cea652267717428f2c9330d0b08b62f15c33a01d78affa90cab0d87152e"}
Sep 18 20:11:57.078576 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:57.077264    2549 scope.go:117] "RemoveContainer" containerID="105f0cea652267717428f2c9330d0b08b62f15c33a01d78affa90cab0d87152e"
Sep 18 20:11:57.080193 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:57.080158    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:11:57.080343 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:57.080253    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:11:57.080343 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:57.080305    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:11:58.088737 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:58.088693    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr" event={"ID":"eeadd113-df76-4c65-83e6-c99756b64684","Type":"ContainerStarted","Data":"65716d550e61a8d07528a1dd04ef02b529c9a28ed9563d00d6d568cb3a6402d5"}
Sep 18 20:11:58.132399 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:58.132360    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr"
Sep 18 20:11:58.133497 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:58.133477    2549 patch_prober.go:28] interesting pod/e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Sep 18 20:11:58.133559 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:58.133520    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr" podUID="eeadd113-df76-4c65-83e6-c99756b64684" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Sep 18 20:11:59.132590 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:59.132552    2549 patch_prober.go:28] interesting pod/e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Sep 18 20:11:59.132994 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:11:59.132615    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr" podUID="eeadd113-df76-4c65-83e6-c99756b64684" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Sep 18 20:12:00.131904 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:00.131871    2549 patch_prober.go:28] interesting pod/e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Sep 18 20:12:00.132084 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:00.131936    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr" podUID="eeadd113-df76-4c65-83e6-c99756b64684" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Sep 18 20:12:00.739155 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:00.739118    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr"
Sep 18 20:12:01.132526 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:01.132490    2549 patch_prober.go:28] interesting pod/e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Sep 18 20:12:01.132695 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:01.132568    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr" podUID="eeadd113-df76-4c65-83e6-c99756b64684" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Sep 18 20:12:02.132767 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:02.132731    2549 patch_prober.go:28] interesting pod/e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Sep 18 20:12:02.133123 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:02.132797    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr" podUID="eeadd113-df76-4c65-83e6-c99756b64684" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Sep 18 20:12:03.132639 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:03.132607    2549 patch_prober.go:28] interesting pod/e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Sep 18 20:12:03.132838 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:03.132665    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr" podUID="eeadd113-df76-4c65-83e6-c99756b64684" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Sep 18 20:12:04.141925 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:04.141889    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr"
Sep 18 20:12:04.142901 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:04.142878    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr"
Sep 18 20:12:05.288267 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:05.288231    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a7e3fae-gxhhj"]
Sep 18 20:12:05.291218 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:05.291191    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a7e3fae-gxhhj"
Sep 18 20:12:05.295828 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:05.295805    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bhssz\""
Sep 18 20:12:05.295828 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:05.295811    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 20:12:05.303436 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:05.303411    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8295fca-d6bb-47e9-aeb8-55c659e68097-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0a7e3fae-gxhhj\" (UID: \"e8295fca-d6bb-47e9-aeb8-55c659e68097\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a7e3fae-gxhhj"
Sep 18 20:12:05.306549 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:05.306528    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a7e3fae-gxhhj"]
Sep 18 20:12:05.404475 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:05.404437    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8295fca-d6bb-47e9-aeb8-55c659e68097-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0a7e3fae-gxhhj\" (UID: \"e8295fca-d6bb-47e9-aeb8-55c659e68097\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a7e3fae-gxhhj"
Sep 18 20:12:05.405071 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:05.405044    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8295fca-d6bb-47e9-aeb8-55c659e68097-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0a7e3fae-gxhhj\" (UID: \"e8295fca-d6bb-47e9-aeb8-55c659e68097\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a7e3fae-gxhhj"
Sep 18 20:12:05.602239 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:05.602212    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a7e3fae-gxhhj"
Sep 18 20:12:05.895428 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:05.895399    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a7e3fae-gxhhj"]
Sep 18 20:12:05.895882 ip-10-0-140-178 kubenswrapper[2549]: W0918 20:12:05.895858    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8295fca_d6bb_47e9_aeb8_55c659e68097.slice/crio-c28f56b65ff2e1b5183e3b10e9ca3c11233460c8f5f8ad0253888bd0efa566c8 WatchSource:0}: Error finding container c28f56b65ff2e1b5183e3b10e9ca3c11233460c8f5f8ad0253888bd0efa566c8: Status 404 returned error can't find the container with id c28f56b65ff2e1b5183e3b10e9ca3c11233460c8f5f8ad0253888bd0efa566c8
Sep 18 20:12:06.154841 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:06.154760    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a7e3fae-gxhhj" event={"ID":"e8295fca-d6bb-47e9-aeb8-55c659e68097","Type":"ContainerStarted","Data":"c28f56b65ff2e1b5183e3b10e9ca3c11233460c8f5f8ad0253888bd0efa566c8"}
Sep 18 20:12:07.168309 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:07.168275    2549 generic.go:356] "Generic (PLEG): container finished" podID="e7e73c75-ba46-4a2b-98c9-974242035854" containerID="30e5c724ef615cd7308c151509887b159c490f5385e412ba90f704e15530aa69" exitCode=0
Sep 18 20:12:07.168884 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:07.168333    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0a7e3fae-openshift-default-656fdc8c44-vp26n" event={"ID":"e7e73c75-ba46-4a2b-98c9-974242035854","Type":"ContainerDied","Data":"30e5c724ef615cd7308c151509887b159c490f5385e412ba90f704e15530aa69"}
Sep 18 20:12:07.168884 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:07.168833    2549 scope.go:117] "RemoveContainer" containerID="30e5c724ef615cd7308c151509887b159c490f5385e412ba90f704e15530aa69"
Sep 18 20:12:07.170217 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:07.170186    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:12:07.170316 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:07.170300    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:12:07.170398 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:07.170340    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:12:08.181682 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:08.181645    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0a7e3fae-openshift-default-656fdc8c44-vp26n" event={"ID":"e7e73c75-ba46-4a2b-98c9-974242035854","Type":"ContainerStarted","Data":"6c4bf1cc34ba51ad44ef25e67f37d38342ff603ebc6e72e19112b2687fc9f6f9"}
Sep 18 20:12:08.602872 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:08.602843    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-0a7e3fae-openshift-default-656fdc8c44-vp26n"
Sep 18 20:12:08.603773 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:08.603741    2549 patch_prober.go:28] interesting pod/e2e-ait-0a7e3fae-openshift-default-656fdc8c44-vp26n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Sep 18 20:12:08.603902 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:08.603794    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0a7e3fae-openshift-default-656fdc8c44-vp26n" podUID="e7e73c75-ba46-4a2b-98c9-974242035854" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Sep 18 20:12:08.898994 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:08.898962    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-0a7e3fae-openshift-default-656fdc8c44-vp26n"
Sep 18 20:12:09.214446 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:09.214328    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a7e3fae-gxhhj" event={"ID":"e8295fca-d6bb-47e9-aeb8-55c659e68097","Type":"ContainerStarted","Data":"c7dd8980f037cc888ad7d53c342c5cfc6f3ad2fd88c8dd63fdc4db0a36832317"}
Sep 18 20:12:09.239558 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:09.239510    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a7e3fae-gxhhj" podStartSLOduration=1.825550008 podStartE2EDuration="4.239493437s" totalImagesPullingTime="2.413943429s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:12:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:12:09.23722167 +0000 UTC m=+914.190962838" watchObservedRunningTime="2026-09-18 20:12:09.239493437 +0000 UTC m=+914.193234604"
Sep 18 20:12:09.603277 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:09.603238    2549 patch_prober.go:28] interesting pod/e2e-ait-0a7e3fae-openshift-default-656fdc8c44-vp26n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Sep 18 20:12:09.603466 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:09.603302    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0a7e3fae-openshift-default-656fdc8c44-vp26n" podUID="e7e73c75-ba46-4a2b-98c9-974242035854" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Sep 18 20:12:10.221591 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:10.221553    2549 generic.go:356] "Generic (PLEG): container finished" podID="e8295fca-d6bb-47e9-aeb8-55c659e68097" containerID="c7dd8980f037cc888ad7d53c342c5cfc6f3ad2fd88c8dd63fdc4db0a36832317" exitCode=0
Sep 18 20:12:10.222114 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:10.222090    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a7e3fae-gxhhj" event={"ID":"e8295fca-d6bb-47e9-aeb8-55c659e68097","Type":"ContainerDied","Data":"c7dd8980f037cc888ad7d53c342c5cfc6f3ad2fd88c8dd63fdc4db0a36832317"}
Sep 18 20:12:10.603514 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:10.603476    2549 patch_prober.go:28] interesting pod/e2e-ait-0a7e3fae-openshift-default-656fdc8c44-vp26n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Sep 18 20:12:10.603729 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:10.603545    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0a7e3fae-openshift-default-656fdc8c44-vp26n" podUID="e7e73c75-ba46-4a2b-98c9-974242035854" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Sep 18 20:12:11.497096 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:11.497024    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a7e3fae-gxhhj"
Sep 18 20:12:11.598328 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:11.598296    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8295fca-d6bb-47e9-aeb8-55c659e68097-maas-api-service-ca\") pod \"e8295fca-d6bb-47e9-aeb8-55c659e68097\" (UID: \"e8295fca-d6bb-47e9-aeb8-55c659e68097\") "
Sep 18 20:12:11.599537 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:11.599503    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8295fca-d6bb-47e9-aeb8-55c659e68097-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e8295fca-d6bb-47e9-aeb8-55c659e68097" (UID: "e8295fca-d6bb-47e9-aeb8-55c659e68097"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:12:11.603708 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:11.603682    2549 patch_prober.go:28] interesting pod/e2e-ait-0a7e3fae-openshift-default-656fdc8c44-vp26n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Sep 18 20:12:11.603815 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:11.603745    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0a7e3fae-openshift-default-656fdc8c44-vp26n" podUID="e7e73c75-ba46-4a2b-98c9-974242035854" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Sep 18 20:12:11.700091 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:11.700047    2549 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8295fca-d6bb-47e9-aeb8-55c659e68097-maas-api-service-ca\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:12:12.234771 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:12.234747    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a7e3fae-gxhhj"
Sep 18 20:12:12.235036 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:12.234790    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a7e3fae-gxhhj" event={"ID":"e8295fca-d6bb-47e9-aeb8-55c659e68097","Type":"ContainerDied","Data":"c28f56b65ff2e1b5183e3b10e9ca3c11233460c8f5f8ad0253888bd0efa566c8"}
Sep 18 20:12:12.235204 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:12.235191    2549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c28f56b65ff2e1b5183e3b10e9ca3c11233460c8f5f8ad0253888bd0efa566c8"
Sep 18 20:12:12.603551 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:12.603513    2549 patch_prober.go:28] interesting pod/e2e-ait-0a7e3fae-openshift-default-656fdc8c44-vp26n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Sep 18 20:12:12.603902 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:12.603568    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0a7e3fae-openshift-default-656fdc8c44-vp26n" podUID="e7e73c75-ba46-4a2b-98c9-974242035854" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Sep 18 20:12:13.607854 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:13.607823    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-0a7e3fae-openshift-default-656fdc8c44-vp26n"
Sep 18 20:12:13.608866 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:13.608845    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-0a7e3fae-openshift-default-656fdc8c44-vp26n"
Sep 18 20:12:16.463454 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:16.463421    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:12:18.273364 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:18.273334    2549 generic.go:356] "Generic (PLEG): container finished" podID="d2cb0038-fbd2-4eff-a731-227ed00eed57" containerID="474e7a33c8ee247f675562129318c3a865174d5d26ecbd96ea8cf2f6b027af8b" exitCode=0
Sep 18 20:12:18.273791 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:18.273407    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm" event={"ID":"d2cb0038-fbd2-4eff-a731-227ed00eed57","Type":"ContainerDied","Data":"474e7a33c8ee247f675562129318c3a865174d5d26ecbd96ea8cf2f6b027af8b"}
Sep 18 20:12:18.273791 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:18.273770    2549 scope.go:117] "RemoveContainer" containerID="474e7a33c8ee247f675562129318c3a865174d5d26ecbd96ea8cf2f6b027af8b"
Sep 18 20:12:18.274901 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:18.274873    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:12:18.275019 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:18.274951    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:12:18.275019 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:18.274999    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:12:19.281764 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:19.281729    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm" event={"ID":"d2cb0038-fbd2-4eff-a731-227ed00eed57","Type":"ContainerStarted","Data":"43097b18c843857228ac00df6755bf900dca3812563fae55946a2d88f2f3caab"}
Sep 18 20:12:20.038951 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:20.038913    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm"
Sep 18 20:12:20.039961 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:20.039939    2549 patch_prober.go:28] interesting pod/e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Sep 18 20:12:20.040049 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:20.039985    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm" podUID="d2cb0038-fbd2-4eff-a731-227ed00eed57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Sep 18 20:12:21.014806 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:21.014772    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm"
Sep 18 20:12:21.038345 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:21.038314    2549 patch_prober.go:28] interesting pod/e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Sep 18 20:12:21.038481 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:21.038397    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm" podUID="d2cb0038-fbd2-4eff-a731-227ed00eed57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Sep 18 20:12:22.038579 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:22.038546    2549 patch_prober.go:28] interesting pod/e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Sep 18 20:12:22.038965 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:22.038601    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm" podUID="d2cb0038-fbd2-4eff-a731-227ed00eed57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Sep 18 20:12:23.039016 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:23.038983    2549 patch_prober.go:28] interesting pod/e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Sep 18 20:12:23.039400 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:23.039039    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm" podUID="d2cb0038-fbd2-4eff-a731-227ed00eed57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Sep 18 20:12:23.090936 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:23.090901    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a7e3fae-gxhhj"]
Sep 18 20:12:23.098398 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:23.098356    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a7e3fae-gxhhj"]
Sep 18 20:12:23.242157 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:23.242128    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0a7e3fae-openshift-default-656fdc8c44-vp26n"]
Sep 18 20:12:23.242332 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:23.242312    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-0a7e3fae-openshift-default-656fdc8c44-vp26n" podUID="e7e73c75-ba46-4a2b-98c9-974242035854" containerName="istio-proxy" containerID="cri-o://6c4bf1cc34ba51ad44ef25e67f37d38342ff603ebc6e72e19112b2687fc9f6f9" gracePeriod=30
Sep 18 20:12:23.522763 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:23.522725    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8295fca-d6bb-47e9-aeb8-55c659e68097" path="/var/lib/kubelet/pods/e8295fca-d6bb-47e9-aeb8-55c659e68097/volumes"
Sep 18 20:12:24.038601 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:24.038566    2549 patch_prober.go:28] interesting pod/e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Sep 18 20:12:24.038770 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:24.038625    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm" podUID="d2cb0038-fbd2-4eff-a731-227ed00eed57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Sep 18 20:12:24.289053 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:24.288944    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h"]
Sep 18 20:12:24.289543 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:24.289460    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8295fca-d6bb-47e9-aeb8-55c659e68097" containerName="revoke-keys"
Sep 18 20:12:24.289543 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:24.289475    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8295fca-d6bb-47e9-aeb8-55c659e68097" containerName="revoke-keys"
Sep 18 20:12:24.289667 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:24.289551    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8295fca-d6bb-47e9-aeb8-55c659e68097" containerName="revoke-keys"
Sep 18 20:12:24.291671 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:24.291649    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h"
Sep 18 20:12:24.299356 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:24.299334    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-7bed33dc-gw-openshift-default-dockercfg-z42x5\""
Sep 18 20:12:24.310741 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:24.310713    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h"]
Sep 18 20:12:24.430299 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:24.430270    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e079f67-e825-45ef-b75d-14dbf11bb9f3-credential-socket\") pod \"e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h\" (UID: \"7e079f67-e825-45ef-b75d-14dbf11bb9f3\") " pod="openshift-ingress/e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h"
Sep 18 20:12:24.430299 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:24.430303    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e079f67-e825-45ef-b75d-14dbf11bb9f3-istio-data\") pod \"e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h\" (UID: \"7e079f67-e825-45ef-b75d-14dbf11bb9f3\") " pod="openshift-ingress/e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h"
Sep 18 20:12:24.430543 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:24.430321    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e079f67-e825-45ef-b75d-14dbf11bb9f3-istio-token\") pod \"e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h\" (UID: \"7e079f67-e825-45ef-b75d-14dbf11bb9f3\") " pod="openshift-ingress/e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h"
Sep 18 20:12:24.430543 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:24.430375    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e079f67-e825-45ef-b75d-14dbf11bb9f3-istio-podinfo\") pod \"e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h\" (UID: \"7e079f67-e825-45ef-b75d-14dbf11bb9f3\") " pod="openshift-ingress/e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h"
Sep 18 20:12:24.430543 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:24.430426    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76j42\" (UniqueName: \"kubernetes.io/projected/7e079f67-e825-45ef-b75d-14dbf11bb9f3-kube-api-access-76j42\") pod \"e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h\" (UID: \"7e079f67-e825-45ef-b75d-14dbf11bb9f3\") " pod="openshift-ingress/e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h"
Sep 18 20:12:24.430543 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:24.430466    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e079f67-e825-45ef-b75d-14dbf11bb9f3-istio-envoy\") pod \"e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h\" (UID: \"7e079f67-e825-45ef-b75d-14dbf11bb9f3\") " pod="openshift-ingress/e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h"
Sep 18 20:12:24.430543 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:24.430487    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e079f67-e825-45ef-b75d-14dbf11bb9f3-workload-certs\") pod \"e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h\" (UID: \"7e079f67-e825-45ef-b75d-14dbf11bb9f3\") " pod="openshift-ingress/e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h"
Sep 18 20:12:24.430543 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:24.430503    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e079f67-e825-45ef-b75d-14dbf11bb9f3-workload-socket\") pod \"e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h\" (UID: \"7e079f67-e825-45ef-b75d-14dbf11bb9f3\") " pod="openshift-ingress/e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h"
Sep 18 20:12:24.430543 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:24.430529    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e079f67-e825-45ef-b75d-14dbf11bb9f3-istiod-ca-cert\") pod \"e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h\" (UID: \"7e079f67-e825-45ef-b75d-14dbf11bb9f3\") " pod="openshift-ingress/e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h"
Sep 18 20:12:24.531906 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:24.531873    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e079f67-e825-45ef-b75d-14dbf11bb9f3-credential-socket\") pod \"e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h\" (UID: \"7e079f67-e825-45ef-b75d-14dbf11bb9f3\") " pod="openshift-ingress/e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h"
Sep 18 20:12:24.532088 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:24.531910    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e079f67-e825-45ef-b75d-14dbf11bb9f3-istio-data\") pod \"e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h\" (UID: \"7e079f67-e825-45ef-b75d-14dbf11bb9f3\") " pod="openshift-ingress/e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h"
Sep 18 20:12:24.532088 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:24.531936    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e079f67-e825-45ef-b75d-14dbf11bb9f3-istio-token\") pod \"e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h\" (UID: \"7e079f67-e825-45ef-b75d-14dbf11bb9f3\") " pod="openshift-ingress/e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h"
Sep 18 20:12:24.532088 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:24.531969    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e079f67-e825-45ef-b75d-14dbf11bb9f3-istio-podinfo\") pod \"e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h\" (UID: \"7e079f67-e825-45ef-b75d-14dbf11bb9f3\") " pod="openshift-ingress/e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h"
Sep 18 20:12:24.532088 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:24.532017    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76j42\" (UniqueName: \"kubernetes.io/projected/7e079f67-e825-45ef-b75d-14dbf11bb9f3-kube-api-access-76j42\") pod \"e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h\" (UID: \"7e079f67-e825-45ef-b75d-14dbf11bb9f3\") " pod="openshift-ingress/e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h"
Sep 18 20:12:24.532088 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:24.532072    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e079f67-e825-45ef-b75d-14dbf11bb9f3-istio-envoy\") pod \"e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h\" (UID: \"7e079f67-e825-45ef-b75d-14dbf11bb9f3\") " pod="openshift-ingress/e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h"
Sep 18 20:12:24.532357 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:24.532106    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e079f67-e825-45ef-b75d-14dbf11bb9f3-workload-certs\") pod \"e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h\" (UID: \"7e079f67-e825-45ef-b75d-14dbf11bb9f3\") " pod="openshift-ingress/e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h"
Sep 18 20:12:24.532357 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:24.532132    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e079f67-e825-45ef-b75d-14dbf11bb9f3-workload-socket\") pod \"e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h\" (UID: \"7e079f67-e825-45ef-b75d-14dbf11bb9f3\") " pod="openshift-ingress/e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h"
Sep 18 20:12:24.532357 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:24.532163    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e079f67-e825-45ef-b75d-14dbf11bb9f3-istiod-ca-cert\") pod \"e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h\" (UID: \"7e079f67-e825-45ef-b75d-14dbf11bb9f3\") " pod="openshift-ingress/e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h"
Sep 18 20:12:24.532357 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:24.532283    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e079f67-e825-45ef-b75d-14dbf11bb9f3-credential-socket\") pod \"e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h\" (UID: \"7e079f67-e825-45ef-b75d-14dbf11bb9f3\") " pod="openshift-ingress/e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h"
Sep 18 20:12:24.532357 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:24.532351    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e079f67-e825-45ef-b75d-14dbf11bb9f3-istio-data\") pod \"e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h\" (UID: \"7e079f67-e825-45ef-b75d-14dbf11bb9f3\") " pod="openshift-ingress/e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h"
Sep 18 20:12:24.532642 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:24.532533    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e079f67-e825-45ef-b75d-14dbf11bb9f3-workload-certs\") pod \"e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h\" (UID: \"7e079f67-e825-45ef-b75d-14dbf11bb9f3\") " pod="openshift-ingress/e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h"
Sep 18 20:12:24.532642 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:24.532569    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e079f67-e825-45ef-b75d-14dbf11bb9f3-workload-socket\") pod \"e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h\" (UID: \"7e079f67-e825-45ef-b75d-14dbf11bb9f3\") " pod="openshift-ingress/e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h"
Sep 18 20:12:24.532918 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:24.532896    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e079f67-e825-45ef-b75d-14dbf11bb9f3-istiod-ca-cert\") pod \"e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h\" (UID: \"7e079f67-e825-45ef-b75d-14dbf11bb9f3\") " pod="openshift-ingress/e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h"
Sep 18 20:12:24.534366 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:24.534342    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e079f67-e825-45ef-b75d-14dbf11bb9f3-istio-envoy\") pod \"e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h\" (UID: \"7e079f67-e825-45ef-b75d-14dbf11bb9f3\") " pod="openshift-ingress/e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h"
Sep 18 20:12:24.534645 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:24.534624    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e079f67-e825-45ef-b75d-14dbf11bb9f3-istio-podinfo\") pod \"e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h\" (UID: \"7e079f67-e825-45ef-b75d-14dbf11bb9f3\") " pod="openshift-ingress/e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h"
Sep 18 20:12:24.544157 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:24.544106    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e079f67-e825-45ef-b75d-14dbf11bb9f3-istio-token\") pod \"e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h\" (UID: \"7e079f67-e825-45ef-b75d-14dbf11bb9f3\") " pod="openshift-ingress/e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h"
Sep 18 20:12:24.553213 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:24.553192    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76j42\" (UniqueName: \"kubernetes.io/projected/7e079f67-e825-45ef-b75d-14dbf11bb9f3-kube-api-access-76j42\") pod \"e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h\" (UID: \"7e079f67-e825-45ef-b75d-14dbf11bb9f3\") " pod="openshift-ingress/e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h"
Sep 18 20:12:24.602118 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:24.602085    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h"
Sep 18 20:12:24.888921 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:24.888893    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h"]
Sep 18 20:12:24.889902 ip-10-0-140-178 kubenswrapper[2549]: W0918 20:12:24.889875    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e079f67_e825_45ef_b75d_14dbf11bb9f3.slice/crio-a9ec6e8b674b5ebd6d1ea3abb44ca9771c2be0c0ea7f7a9c21d2d5491ffdb20b WatchSource:0}: Error finding container a9ec6e8b674b5ebd6d1ea3abb44ca9771c2be0c0ea7f7a9c21d2d5491ffdb20b: Status 404 returned error can't find the container with id a9ec6e8b674b5ebd6d1ea3abb44ca9771c2be0c0ea7f7a9c21d2d5491ffdb20b
Sep 18 20:12:24.892262 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:24.892228    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:12:24.892355 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:24.892309    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:12:24.892355 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:24.892338    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:12:25.042276 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:25.042247    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm"
Sep 18 20:12:25.043293 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:25.043266    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm"
Sep 18 20:12:25.322463 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:25.322425    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h" event={"ID":"7e079f67-e825-45ef-b75d-14dbf11bb9f3","Type":"ContainerStarted","Data":"abbde195d3fb03b1a8d77cf2d4862caeeee16e9734858be7ade8479b5dd57e94"}
Sep 18 20:12:25.322867 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:25.322492    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h" event={"ID":"7e079f67-e825-45ef-b75d-14dbf11bb9f3","Type":"ContainerStarted","Data":"a9ec6e8b674b5ebd6d1ea3abb44ca9771c2be0c0ea7f7a9c21d2d5491ffdb20b"}
Sep 18 20:12:25.351814 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:25.351770    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h" podStartSLOduration=1.351752402 podStartE2EDuration="1.351752402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:12:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:12:25.348548292 +0000 UTC m=+930.302289472" watchObservedRunningTime="2026-09-18 20:12:25.351752402 +0000 UTC m=+930.305493569"
Sep 18 20:12:25.603315 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:25.603236    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h"
Sep 18 20:12:25.608267 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:25.608242    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h"
Sep 18 20:12:26.328765 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:26.328737    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h"
Sep 18 20:12:26.329603 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:26.329582    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h"
Sep 18 20:12:28.508069 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:28.508043    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0a7e3fae-openshift-default-656fdc8c44-vp26n"
Sep 18 20:12:28.672705 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:28.672663    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7e73c75-ba46-4a2b-98c9-974242035854-istio-token\") pod \"e7e73c75-ba46-4a2b-98c9-974242035854\" (UID: \"e7e73c75-ba46-4a2b-98c9-974242035854\") "
Sep 18 20:12:28.672705 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:28.672713    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7e73c75-ba46-4a2b-98c9-974242035854-istiod-ca-cert\") pod \"e7e73c75-ba46-4a2b-98c9-974242035854\" (UID: \"e7e73c75-ba46-4a2b-98c9-974242035854\") "
Sep 18 20:12:28.672961 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:28.672765    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7e73c75-ba46-4a2b-98c9-974242035854-credential-socket\") pod \"e7e73c75-ba46-4a2b-98c9-974242035854\" (UID: \"e7e73c75-ba46-4a2b-98c9-974242035854\") "
Sep 18 20:12:28.672961 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:28.672803    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7e73c75-ba46-4a2b-98c9-974242035854-istio-podinfo\") pod \"e7e73c75-ba46-4a2b-98c9-974242035854\" (UID: \"e7e73c75-ba46-4a2b-98c9-974242035854\") "
Sep 18 20:12:28.672961 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:28.672853    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r4t77\" (UniqueName: \"kubernetes.io/projected/e7e73c75-ba46-4a2b-98c9-974242035854-kube-api-access-r4t77\") pod \"e7e73c75-ba46-4a2b-98c9-974242035854\" (UID: \"e7e73c75-ba46-4a2b-98c9-974242035854\") "
Sep 18 20:12:28.672961 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:28.672887    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7e73c75-ba46-4a2b-98c9-974242035854-workload-socket\") pod \"e7e73c75-ba46-4a2b-98c9-974242035854\" (UID: \"e7e73c75-ba46-4a2b-98c9-974242035854\") "
Sep 18 20:12:28.672961 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:28.672920    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7e73c75-ba46-4a2b-98c9-974242035854-istio-data\") pod \"e7e73c75-ba46-4a2b-98c9-974242035854\" (UID: \"e7e73c75-ba46-4a2b-98c9-974242035854\") "
Sep 18 20:12:28.672961 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:28.672935    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7e73c75-ba46-4a2b-98c9-974242035854-workload-certs\") pod \"e7e73c75-ba46-4a2b-98c9-974242035854\" (UID: \"e7e73c75-ba46-4a2b-98c9-974242035854\") "
Sep 18 20:12:28.673277 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:28.672974    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7e73c75-ba46-4a2b-98c9-974242035854-istio-envoy\") pod \"e7e73c75-ba46-4a2b-98c9-974242035854\" (UID: \"e7e73c75-ba46-4a2b-98c9-974242035854\") "
Sep 18 20:12:28.673540 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:28.673513    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7e73c75-ba46-4a2b-98c9-974242035854-istio-data" (OuterVolumeSpecName: "istio-data") pod "e7e73c75-ba46-4a2b-98c9-974242035854" (UID: "e7e73c75-ba46-4a2b-98c9-974242035854"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:12:28.673778 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:28.673631    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7e73c75-ba46-4a2b-98c9-974242035854-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e7e73c75-ba46-4a2b-98c9-974242035854" (UID: "e7e73c75-ba46-4a2b-98c9-974242035854"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:12:28.673778 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:28.673735    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7e73c75-ba46-4a2b-98c9-974242035854-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e7e73c75-ba46-4a2b-98c9-974242035854" (UID: "e7e73c75-ba46-4a2b-98c9-974242035854"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:12:28.673901 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:28.673817    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7e73c75-ba46-4a2b-98c9-974242035854-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e7e73c75-ba46-4a2b-98c9-974242035854" (UID: "e7e73c75-ba46-4a2b-98c9-974242035854"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:12:28.675033 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:28.675010    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7e73c75-ba46-4a2b-98c9-974242035854-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e7e73c75-ba46-4a2b-98c9-974242035854" (UID: "e7e73c75-ba46-4a2b-98c9-974242035854"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:12:28.676188 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:28.676160    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7e73c75-ba46-4a2b-98c9-974242035854-kube-api-access-r4t77" (OuterVolumeSpecName: "kube-api-access-r4t77") pod "e7e73c75-ba46-4a2b-98c9-974242035854" (UID: "e7e73c75-ba46-4a2b-98c9-974242035854"). InnerVolumeSpecName "kube-api-access-r4t77". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:12:28.676298 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:28.676221    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7e73c75-ba46-4a2b-98c9-974242035854-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e7e73c75-ba46-4a2b-98c9-974242035854" (UID: "e7e73c75-ba46-4a2b-98c9-974242035854"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:12:28.676298 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:28.676233    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7e73c75-ba46-4a2b-98c9-974242035854-istio-token" (OuterVolumeSpecName: "istio-token") pod "e7e73c75-ba46-4a2b-98c9-974242035854" (UID: "e7e73c75-ba46-4a2b-98c9-974242035854"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:12:28.676639 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:28.676623    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e7e73c75-ba46-4a2b-98c9-974242035854-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e7e73c75-ba46-4a2b-98c9-974242035854" (UID: "e7e73c75-ba46-4a2b-98c9-974242035854"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 20:12:28.774345 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:28.774314    2549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7e73c75-ba46-4a2b-98c9-974242035854-istio-envoy\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:12:28.774345 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:28.774342    2549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7e73c75-ba46-4a2b-98c9-974242035854-istio-token\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:12:28.774345 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:28.774352    2549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7e73c75-ba46-4a2b-98c9-974242035854-istiod-ca-cert\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:12:28.774562 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:28.774361    2549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7e73c75-ba46-4a2b-98c9-974242035854-credential-socket\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:12:28.774562 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:28.774371    2549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7e73c75-ba46-4a2b-98c9-974242035854-istio-podinfo\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:12:28.774562 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:28.774391    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4t77\" (UniqueName: \"kubernetes.io/projected/e7e73c75-ba46-4a2b-98c9-974242035854-kube-api-access-r4t77\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:12:28.774562 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:28.774400    2549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7e73c75-ba46-4a2b-98c9-974242035854-workload-socket\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:12:28.774562 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:28.774408    2549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7e73c75-ba46-4a2b-98c9-974242035854-istio-data\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:12:28.774562 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:28.774416    2549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7e73c75-ba46-4a2b-98c9-974242035854-workload-certs\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:12:29.348349 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:29.348309    2549 generic.go:356] "Generic (PLEG): container finished" podID="e7e73c75-ba46-4a2b-98c9-974242035854" containerID="6c4bf1cc34ba51ad44ef25e67f37d38342ff603ebc6e72e19112b2687fc9f6f9" exitCode=0
Sep 18 20:12:29.348562 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:29.348378    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0a7e3fae-openshift-default-656fdc8c44-vp26n"
Sep 18 20:12:29.348562 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:29.348413    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0a7e3fae-openshift-default-656fdc8c44-vp26n" event={"ID":"e7e73c75-ba46-4a2b-98c9-974242035854","Type":"ContainerDied","Data":"6c4bf1cc34ba51ad44ef25e67f37d38342ff603ebc6e72e19112b2687fc9f6f9"}
Sep 18 20:12:29.348562 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:29.348508    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0a7e3fae-openshift-default-656fdc8c44-vp26n" event={"ID":"e7e73c75-ba46-4a2b-98c9-974242035854","Type":"ContainerDied","Data":"77d1e70253712c1ca1c5005658446cce63c9d633b0ca27a681634427f2bd07b7"}
Sep 18 20:12:29.348562 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:29.348538    2549 scope.go:117] "RemoveContainer" containerID="6c4bf1cc34ba51ad44ef25e67f37d38342ff603ebc6e72e19112b2687fc9f6f9"
Sep 18 20:12:29.359786 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:29.359764    2549 scope.go:117] "RemoveContainer" containerID="30e5c724ef615cd7308c151509887b159c490f5385e412ba90f704e15530aa69"
Sep 18 20:12:29.368154 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:29.368137    2549 scope.go:117] "RemoveContainer" containerID="6c4bf1cc34ba51ad44ef25e67f37d38342ff603ebc6e72e19112b2687fc9f6f9"
Sep 18 20:12:29.368449 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:12:29.368423    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c4bf1cc34ba51ad44ef25e67f37d38342ff603ebc6e72e19112b2687fc9f6f9\": container with ID starting with 6c4bf1cc34ba51ad44ef25e67f37d38342ff603ebc6e72e19112b2687fc9f6f9 not found: ID does not exist" containerID="6c4bf1cc34ba51ad44ef25e67f37d38342ff603ebc6e72e19112b2687fc9f6f9"
Sep 18 20:12:29.368546 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:29.368452    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c4bf1cc34ba51ad44ef25e67f37d38342ff603ebc6e72e19112b2687fc9f6f9"} err="failed to get container status \"6c4bf1cc34ba51ad44ef25e67f37d38342ff603ebc6e72e19112b2687fc9f6f9\": rpc error: code = NotFound desc = could not find container \"6c4bf1cc34ba51ad44ef25e67f37d38342ff603ebc6e72e19112b2687fc9f6f9\": container with ID starting with 6c4bf1cc34ba51ad44ef25e67f37d38342ff603ebc6e72e19112b2687fc9f6f9 not found: ID does not exist"
Sep 18 20:12:29.368546 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:29.368483    2549 scope.go:117] "RemoveContainer" containerID="30e5c724ef615cd7308c151509887b159c490f5385e412ba90f704e15530aa69"
Sep 18 20:12:29.368738 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:12:29.368720    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30e5c724ef615cd7308c151509887b159c490f5385e412ba90f704e15530aa69\": container with ID starting with 30e5c724ef615cd7308c151509887b159c490f5385e412ba90f704e15530aa69 not found: ID does not exist" containerID="30e5c724ef615cd7308c151509887b159c490f5385e412ba90f704e15530aa69"
Sep 18 20:12:29.368778 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:29.368746    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30e5c724ef615cd7308c151509887b159c490f5385e412ba90f704e15530aa69"} err="failed to get container status \"30e5c724ef615cd7308c151509887b159c490f5385e412ba90f704e15530aa69\": rpc error: code = NotFound desc = could not find container \"30e5c724ef615cd7308c151509887b159c490f5385e412ba90f704e15530aa69\": container with ID starting with 30e5c724ef615cd7308c151509887b159c490f5385e412ba90f704e15530aa69 not found: ID does not exist"
Sep 18 20:12:29.373648 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:29.373627    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0a7e3fae-openshift-default-656fdc8c44-vp26n"]
Sep 18 20:12:29.385150 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:29.385125    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-0a7e3fae-openshift-default-656fdc8c44-vp26n"]
Sep 18 20:12:29.521513 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:29.521484    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7e73c75-ba46-4a2b-98c9-974242035854" path="/var/lib/kubelet/pods/e7e73c75-ba46-4a2b-98c9-974242035854/volumes"
Sep 18 20:12:30.531176 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:30.531144    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:12:47.459667 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:47.459593    2549 generic.go:356] "Generic (PLEG): container finished" podID="7e079f67-e825-45ef-b75d-14dbf11bb9f3" containerID="abbde195d3fb03b1a8d77cf2d4862caeeee16e9734858be7ade8479b5dd57e94" exitCode=0
Sep 18 20:12:47.460038 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:47.459662    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h" event={"ID":"7e079f67-e825-45ef-b75d-14dbf11bb9f3","Type":"ContainerDied","Data":"abbde195d3fb03b1a8d77cf2d4862caeeee16e9734858be7ade8479b5dd57e94"}
Sep 18 20:12:47.460113 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:47.460098    2549 scope.go:117] "RemoveContainer" containerID="abbde195d3fb03b1a8d77cf2d4862caeeee16e9734858be7ade8479b5dd57e94"
Sep 18 20:12:47.461208 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:47.461183    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:12:47.461259 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:47.461248    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:12:47.461293 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:47.461277    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:12:48.468631 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:48.468578    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h" event={"ID":"7e079f67-e825-45ef-b75d-14dbf11bb9f3","Type":"ContainerStarted","Data":"b84ce33bbbe2d2a05750235d47f54cce628420814220a37eb343ba342b45413f"}
Sep 18 20:12:48.602583 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:48.602537    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h"
Sep 18 20:12:48.602752 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:48.602717    2549 patch_prober.go:28] interesting pod/e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Sep 18 20:12:48.602799 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:48.602763    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h" podUID="7e079f67-e825-45ef-b75d-14dbf11bb9f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Sep 18 20:12:48.624122 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:48.624089    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:12:49.603015 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:49.602979    2549 patch_prober.go:28] interesting pod/e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Sep 18 20:12:49.603449 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:49.603045    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h" podUID="7e079f67-e825-45ef-b75d-14dbf11bb9f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Sep 18 20:12:50.602839 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:50.602807    2549 patch_prober.go:28] interesting pod/e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Sep 18 20:12:50.603025 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:50.602872    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h" podUID="7e079f67-e825-45ef-b75d-14dbf11bb9f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Sep 18 20:12:51.602803 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:51.602767    2549 patch_prober.go:28] interesting pod/e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Sep 18 20:12:51.603176 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:51.602825    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h" podUID="7e079f67-e825-45ef-b75d-14dbf11bb9f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Sep 18 20:12:52.602747 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:52.602706    2549 patch_prober.go:28] interesting pod/e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Sep 18 20:12:52.602944 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:52.602775    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h" podUID="7e079f67-e825-45ef-b75d-14dbf11bb9f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Sep 18 20:12:53.602565 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:53.602529    2549 patch_prober.go:28] interesting pod/e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Sep 18 20:12:53.602749 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:53.602584    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h" podUID="7e079f67-e825-45ef-b75d-14dbf11bb9f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Sep 18 20:12:54.607341 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:54.607310    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h"
Sep 18 20:12:54.607754 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:54.607595    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h"
Sep 18 20:12:54.608324 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:54.608307    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h"
Sep 18 20:12:56.921256 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:56.921224    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7bed33dc-rkmrs"]
Sep 18 20:12:56.921724 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:56.921709    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7e73c75-ba46-4a2b-98c9-974242035854" containerName="istio-proxy"
Sep 18 20:12:56.921724 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:56.921725    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7e73c75-ba46-4a2b-98c9-974242035854" containerName="istio-proxy"
Sep 18 20:12:56.921801 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:56.921748    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7e73c75-ba46-4a2b-98c9-974242035854" containerName="istio-proxy"
Sep 18 20:12:56.921801 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:56.921757    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7e73c75-ba46-4a2b-98c9-974242035854" containerName="istio-proxy"
Sep 18 20:12:56.921867 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:56.921845    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7e73c75-ba46-4a2b-98c9-974242035854" containerName="istio-proxy"
Sep 18 20:12:56.921867 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:56.921855    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7e73c75-ba46-4a2b-98c9-974242035854" containerName="istio-proxy"
Sep 18 20:12:56.924524 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:56.924505    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7bed33dc-rkmrs"
Sep 18 20:12:56.927560 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:56.927537    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 20:12:56.927692 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:56.927540    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bhssz\""
Sep 18 20:12:56.934345 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:56.934316    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7bed33dc-rkmrs"]
Sep 18 20:12:57.061794 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:57.061763    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec64c66e-f4d5-4f17-956e-23753a58ff77-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-7bed33dc-rkmrs\" (UID: \"ec64c66e-f4d5-4f17-956e-23753a58ff77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7bed33dc-rkmrs"
Sep 18 20:12:57.162589 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:57.162555    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec64c66e-f4d5-4f17-956e-23753a58ff77-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-7bed33dc-rkmrs\" (UID: \"ec64c66e-f4d5-4f17-956e-23753a58ff77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7bed33dc-rkmrs"
Sep 18 20:12:57.163199 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:57.163180    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec64c66e-f4d5-4f17-956e-23753a58ff77-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-7bed33dc-rkmrs\" (UID: \"ec64c66e-f4d5-4f17-956e-23753a58ff77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7bed33dc-rkmrs"
Sep 18 20:12:57.236914 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:57.236844    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7bed33dc-rkmrs"
Sep 18 20:12:57.519869 ip-10-0-140-178 kubenswrapper[2549]: W0918 20:12:57.519830    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec64c66e_f4d5_4f17_956e_23753a58ff77.slice/crio-bfa417fecdc9d7f3747a205fff474457e7923dfb512c40cd960f69a13aaef54e WatchSource:0}: Error finding container bfa417fecdc9d7f3747a205fff474457e7923dfb512c40cd960f69a13aaef54e: Status 404 returned error can't find the container with id bfa417fecdc9d7f3747a205fff474457e7923dfb512c40cd960f69a13aaef54e
Sep 18 20:12:57.522674 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:57.522650    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7bed33dc-rkmrs"]
Sep 18 20:12:57.535114 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:57.535088    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7bed33dc-rkmrs" event={"ID":"ec64c66e-f4d5-4f17-956e-23753a58ff77","Type":"ContainerStarted","Data":"bfa417fecdc9d7f3747a205fff474457e7923dfb512c40cd960f69a13aaef54e"}
Sep 18 20:12:58.541654 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:58.541578    2549 generic.go:356] "Generic (PLEG): container finished" podID="ec64c66e-f4d5-4f17-956e-23753a58ff77" containerID="af2af129b91d52a1a95d03a9a049c7abdee7ba9a245526b8ef24e5211de85eb3" exitCode=0
Sep 18 20:12:58.542009 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:58.541657    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7bed33dc-rkmrs" event={"ID":"ec64c66e-f4d5-4f17-956e-23753a58ff77","Type":"ContainerDied","Data":"af2af129b91d52a1a95d03a9a049c7abdee7ba9a245526b8ef24e5211de85eb3"}
Sep 18 20:12:59.685747 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:59.685722    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7bed33dc-rkmrs"
Sep 18 20:12:59.789164 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:59.789132    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec64c66e-f4d5-4f17-956e-23753a58ff77-maas-api-service-ca\") pod \"ec64c66e-f4d5-4f17-956e-23753a58ff77\" (UID: \"ec64c66e-f4d5-4f17-956e-23753a58ff77\") "
Sep 18 20:12:59.789499 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:59.789478    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec64c66e-f4d5-4f17-956e-23753a58ff77-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ec64c66e-f4d5-4f17-956e-23753a58ff77" (UID: "ec64c66e-f4d5-4f17-956e-23753a58ff77"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:12:59.890577 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:12:59.890548    2549 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec64c66e-f4d5-4f17-956e-23753a58ff77-maas-api-service-ca\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:13:00.553878 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:00.553844    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7bed33dc-rkmrs" event={"ID":"ec64c66e-f4d5-4f17-956e-23753a58ff77","Type":"ContainerDied","Data":"bfa417fecdc9d7f3747a205fff474457e7923dfb512c40cd960f69a13aaef54e"}
Sep 18 20:13:00.553878 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:00.553866    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7bed33dc-rkmrs"
Sep 18 20:13:00.554086 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:00.553900    2549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bfa417fecdc9d7f3747a205fff474457e7923dfb512c40cd960f69a13aaef54e"
Sep 18 20:13:03.924896 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:03.924856    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-5284dc/simulator-w0-5284dc-kserve-84dc7dc7f-mpwht"]
Sep 18 20:13:03.925414 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:03.925355    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec64c66e-f4d5-4f17-956e-23753a58ff77" containerName="revoke-keys"
Sep 18 20:13:03.925414 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:03.925369    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec64c66e-f4d5-4f17-956e-23753a58ff77" containerName="revoke-keys"
Sep 18 20:13:03.925568 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:03.925513    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec64c66e-f4d5-4f17-956e-23753a58ff77" containerName="revoke-keys"
Sep 18 20:13:03.928755 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:03.928740    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-5284dc/simulator-w0-5284dc-kserve-84dc7dc7f-mpwht"
Sep 18 20:13:03.933133 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:03.933104    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-5284dc\"/\"simulator-w0-5284dc-kserve-self-signed-certs\""
Sep 18 20:13:03.933133 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:03.933130    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-5284dc\"/\"default-dockercfg-r2p8h\""
Sep 18 20:13:03.933326 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:03.933135    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-5284dc\"/\"kube-root-ca.crt\""
Sep 18 20:13:03.933326 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:03.933134    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-5284dc\"/\"openshift-service-ca.crt\""
Sep 18 20:13:03.941414 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:03.941372    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-5284dc/simulator-w0-5284dc-kserve-84dc7dc7f-mpwht"]
Sep 18 20:13:04.028488 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:04.028458    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/daf3d095-d0b4-462f-b05d-0a699a8eb26c-home\") pod \"simulator-w0-5284dc-kserve-84dc7dc7f-mpwht\" (UID: \"daf3d095-d0b4-462f-b05d-0a699a8eb26c\") " pod="e2e-models-e2e-worker-w0-5284dc/simulator-w0-5284dc-kserve-84dc7dc7f-mpwht"
Sep 18 20:13:04.028645 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:04.028501    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwrkb\" (UniqueName: \"kubernetes.io/projected/daf3d095-d0b4-462f-b05d-0a699a8eb26c-kube-api-access-wwrkb\") pod \"simulator-w0-5284dc-kserve-84dc7dc7f-mpwht\" (UID: \"daf3d095-d0b4-462f-b05d-0a699a8eb26c\") " pod="e2e-models-e2e-worker-w0-5284dc/simulator-w0-5284dc-kserve-84dc7dc7f-mpwht"
Sep 18 20:13:04.028645 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:04.028540    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/daf3d095-d0b4-462f-b05d-0a699a8eb26c-tmp-dir\") pod \"simulator-w0-5284dc-kserve-84dc7dc7f-mpwht\" (UID: \"daf3d095-d0b4-462f-b05d-0a699a8eb26c\") " pod="e2e-models-e2e-worker-w0-5284dc/simulator-w0-5284dc-kserve-84dc7dc7f-mpwht"
Sep 18 20:13:04.028645 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:04.028576    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/daf3d095-d0b4-462f-b05d-0a699a8eb26c-tls-certs\") pod \"simulator-w0-5284dc-kserve-84dc7dc7f-mpwht\" (UID: \"daf3d095-d0b4-462f-b05d-0a699a8eb26c\") " pod="e2e-models-e2e-worker-w0-5284dc/simulator-w0-5284dc-kserve-84dc7dc7f-mpwht"
Sep 18 20:13:04.028785 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:04.028691    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/daf3d095-d0b4-462f-b05d-0a699a8eb26c-model-cache\") pod \"simulator-w0-5284dc-kserve-84dc7dc7f-mpwht\" (UID: \"daf3d095-d0b4-462f-b05d-0a699a8eb26c\") " pod="e2e-models-e2e-worker-w0-5284dc/simulator-w0-5284dc-kserve-84dc7dc7f-mpwht"
Sep 18 20:13:04.028785 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:04.028730    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/daf3d095-d0b4-462f-b05d-0a699a8eb26c-dshm\") pod \"simulator-w0-5284dc-kserve-84dc7dc7f-mpwht\" (UID: \"daf3d095-d0b4-462f-b05d-0a699a8eb26c\") " pod="e2e-models-e2e-worker-w0-5284dc/simulator-w0-5284dc-kserve-84dc7dc7f-mpwht"
Sep 18 20:13:04.129493 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:04.129457    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/daf3d095-d0b4-462f-b05d-0a699a8eb26c-model-cache\") pod \"simulator-w0-5284dc-kserve-84dc7dc7f-mpwht\" (UID: \"daf3d095-d0b4-462f-b05d-0a699a8eb26c\") " pod="e2e-models-e2e-worker-w0-5284dc/simulator-w0-5284dc-kserve-84dc7dc7f-mpwht"
Sep 18 20:13:04.129648 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:04.129508    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/daf3d095-d0b4-462f-b05d-0a699a8eb26c-dshm\") pod \"simulator-w0-5284dc-kserve-84dc7dc7f-mpwht\" (UID: \"daf3d095-d0b4-462f-b05d-0a699a8eb26c\") " pod="e2e-models-e2e-worker-w0-5284dc/simulator-w0-5284dc-kserve-84dc7dc7f-mpwht"
Sep 18 20:13:04.129648 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:04.129564    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/daf3d095-d0b4-462f-b05d-0a699a8eb26c-home\") pod \"simulator-w0-5284dc-kserve-84dc7dc7f-mpwht\" (UID: \"daf3d095-d0b4-462f-b05d-0a699a8eb26c\") " pod="e2e-models-e2e-worker-w0-5284dc/simulator-w0-5284dc-kserve-84dc7dc7f-mpwht"
Sep 18 20:13:04.129648 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:04.129602    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwrkb\" (UniqueName: \"kubernetes.io/projected/daf3d095-d0b4-462f-b05d-0a699a8eb26c-kube-api-access-wwrkb\") pod \"simulator-w0-5284dc-kserve-84dc7dc7f-mpwht\" (UID: \"daf3d095-d0b4-462f-b05d-0a699a8eb26c\") " pod="e2e-models-e2e-worker-w0-5284dc/simulator-w0-5284dc-kserve-84dc7dc7f-mpwht"
Sep 18 20:13:04.129648 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:04.129637    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/daf3d095-d0b4-462f-b05d-0a699a8eb26c-tmp-dir\") pod \"simulator-w0-5284dc-kserve-84dc7dc7f-mpwht\" (UID: \"daf3d095-d0b4-462f-b05d-0a699a8eb26c\") " pod="e2e-models-e2e-worker-w0-5284dc/simulator-w0-5284dc-kserve-84dc7dc7f-mpwht"
Sep 18 20:13:04.129814 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:04.129673    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/daf3d095-d0b4-462f-b05d-0a699a8eb26c-tls-certs\") pod \"simulator-w0-5284dc-kserve-84dc7dc7f-mpwht\" (UID: \"daf3d095-d0b4-462f-b05d-0a699a8eb26c\") " pod="e2e-models-e2e-worker-w0-5284dc/simulator-w0-5284dc-kserve-84dc7dc7f-mpwht"
Sep 18 20:13:04.129943 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:04.129920    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/daf3d095-d0b4-462f-b05d-0a699a8eb26c-model-cache\") pod \"simulator-w0-5284dc-kserve-84dc7dc7f-mpwht\" (UID: \"daf3d095-d0b4-462f-b05d-0a699a8eb26c\") " pod="e2e-models-e2e-worker-w0-5284dc/simulator-w0-5284dc-kserve-84dc7dc7f-mpwht"
Sep 18 20:13:04.130008 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:04.129943    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/daf3d095-d0b4-462f-b05d-0a699a8eb26c-home\") pod \"simulator-w0-5284dc-kserve-84dc7dc7f-mpwht\" (UID: \"daf3d095-d0b4-462f-b05d-0a699a8eb26c\") " pod="e2e-models-e2e-worker-w0-5284dc/simulator-w0-5284dc-kserve-84dc7dc7f-mpwht"
Sep 18 20:13:04.130055 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:04.130033    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/daf3d095-d0b4-462f-b05d-0a699a8eb26c-tmp-dir\") pod \"simulator-w0-5284dc-kserve-84dc7dc7f-mpwht\" (UID: \"daf3d095-d0b4-462f-b05d-0a699a8eb26c\") " pod="e2e-models-e2e-worker-w0-5284dc/simulator-w0-5284dc-kserve-84dc7dc7f-mpwht"
Sep 18 20:13:04.131861 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:04.131835    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/daf3d095-d0b4-462f-b05d-0a699a8eb26c-dshm\") pod \"simulator-w0-5284dc-kserve-84dc7dc7f-mpwht\" (UID: \"daf3d095-d0b4-462f-b05d-0a699a8eb26c\") " pod="e2e-models-e2e-worker-w0-5284dc/simulator-w0-5284dc-kserve-84dc7dc7f-mpwht"
Sep 18 20:13:04.132003 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:04.131982    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/daf3d095-d0b4-462f-b05d-0a699a8eb26c-tls-certs\") pod \"simulator-w0-5284dc-kserve-84dc7dc7f-mpwht\" (UID: \"daf3d095-d0b4-462f-b05d-0a699a8eb26c\") " pod="e2e-models-e2e-worker-w0-5284dc/simulator-w0-5284dc-kserve-84dc7dc7f-mpwht"
Sep 18 20:13:04.144966 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:04.144941    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwrkb\" (UniqueName: \"kubernetes.io/projected/daf3d095-d0b4-462f-b05d-0a699a8eb26c-kube-api-access-wwrkb\") pod \"simulator-w0-5284dc-kserve-84dc7dc7f-mpwht\" (UID: \"daf3d095-d0b4-462f-b05d-0a699a8eb26c\") " pod="e2e-models-e2e-worker-w0-5284dc/simulator-w0-5284dc-kserve-84dc7dc7f-mpwht"
Sep 18 20:13:04.238031 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:04.237943    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-5284dc/simulator-w0-5284dc-kserve-84dc7dc7f-mpwht"
Sep 18 20:13:04.521374 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:04.521292    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-5284dc/simulator-w0-5284dc-kserve-84dc7dc7f-mpwht"]
Sep 18 20:13:04.524661 ip-10-0-140-178 kubenswrapper[2549]: W0918 20:13:04.524633    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddaf3d095_d0b4_462f_b05d_0a699a8eb26c.slice/crio-4a1e984fd07f3cf59d5eff4946fefe9e8843788e707729101130fe2eeb1dfa30 WatchSource:0}: Error finding container 4a1e984fd07f3cf59d5eff4946fefe9e8843788e707729101130fe2eeb1dfa30: Status 404 returned error can't find the container with id 4a1e984fd07f3cf59d5eff4946fefe9e8843788e707729101130fe2eeb1dfa30
Sep 18 20:13:04.579963 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:04.579932    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-5284dc/simulator-w0-5284dc-kserve-84dc7dc7f-mpwht" event={"ID":"daf3d095-d0b4-462f-b05d-0a699a8eb26c","Type":"ContainerStarted","Data":"4a1e984fd07f3cf59d5eff4946fefe9e8843788e707729101130fe2eeb1dfa30"}
Sep 18 20:13:06.596009 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:06.595972    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-5284dc/simulator-w0-5284dc-kserve-84dc7dc7f-mpwht" event={"ID":"daf3d095-d0b4-462f-b05d-0a699a8eb26c","Type":"ContainerStarted","Data":"a66e0dd6332a112e76bcdcf3d5edc88fdcbc315b2fe24c835a9c3c2bfe2a1818"}
Sep 18 20:13:06.636477 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:06.636429    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-5284dc/simulator-w0-5284dc-kserve-84dc7dc7f-mpwht" podStartSLOduration=2.479104762 podStartE2EDuration="3.636413439s" totalImagesPullingTime="1.157308677s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:13:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:13:06.635511534 +0000 UTC m=+971.589252701" watchObservedRunningTime="2026-09-18 20:13:06.636413439 +0000 UTC m=+971.590154606"
Sep 18 20:13:09.068680 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:09.068646    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7bed33dc-rkmrs"]
Sep 18 20:13:09.071711 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:09.071684    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7bed33dc-rkmrs"]
Sep 18 20:13:09.204974 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:09.204933    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h"]
Sep 18 20:13:09.205185 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:09.205159    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h" podUID="7e079f67-e825-45ef-b75d-14dbf11bb9f3" containerName="istio-proxy" containerID="cri-o://b84ce33bbbe2d2a05750235d47f54cce628420814220a37eb343ba342b45413f" gracePeriod=30
Sep 18 20:13:09.522400 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:09.522339    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec64c66e-f4d5-4f17-956e-23753a58ff77" path="/var/lib/kubelet/pods/ec64c66e-f4d5-4f17-956e-23753a58ff77/volumes"
Sep 18 20:13:09.610257 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:09.610224    2549 patch_prober.go:28] interesting pod/e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 18 20:13:09.610430 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:09.610280    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h" podUID="7e079f67-e825-45ef-b75d-14dbf11bb9f3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:13:09.754499 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:09.754460    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:13:09.992359 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:09.992326    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-f524d4ec-openshift-default-7cf9f69b46-jpkf2"]
Sep 18 20:13:10.001255 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:09.997319    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f524d4ec-openshift-default-7cf9f69b46-jpkf2"
Sep 18 20:13:10.001682 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:10.001658    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-f524d4ec-openshift-default-dockercfg-kmdbc\""
Sep 18 20:13:10.012610 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:10.012584    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f524d4ec-openshift-default-7cf9f69b46-jpkf2"]
Sep 18 20:13:10.097404 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:10.091184    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c2246a4-6b66-49ff-a268-b34347959d01-istio-data\") pod \"e2e-ait-f524d4ec-openshift-default-7cf9f69b46-jpkf2\" (UID: \"1c2246a4-6b66-49ff-a268-b34347959d01\") " pod="openshift-ingress/e2e-ait-f524d4ec-openshift-default-7cf9f69b46-jpkf2"
Sep 18 20:13:10.097404 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:10.091291    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c2246a4-6b66-49ff-a268-b34347959d01-istiod-ca-cert\") pod \"e2e-ait-f524d4ec-openshift-default-7cf9f69b46-jpkf2\" (UID: \"1c2246a4-6b66-49ff-a268-b34347959d01\") " pod="openshift-ingress/e2e-ait-f524d4ec-openshift-default-7cf9f69b46-jpkf2"
Sep 18 20:13:10.097404 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:10.091327    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c2246a4-6b66-49ff-a268-b34347959d01-workload-socket\") pod \"e2e-ait-f524d4ec-openshift-default-7cf9f69b46-jpkf2\" (UID: \"1c2246a4-6b66-49ff-a268-b34347959d01\") " pod="openshift-ingress/e2e-ait-f524d4ec-openshift-default-7cf9f69b46-jpkf2"
Sep 18 20:13:10.097404 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:10.091359    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c2246a4-6b66-49ff-a268-b34347959d01-workload-certs\") pod \"e2e-ait-f524d4ec-openshift-default-7cf9f69b46-jpkf2\" (UID: \"1c2246a4-6b66-49ff-a268-b34347959d01\") " pod="openshift-ingress/e2e-ait-f524d4ec-openshift-default-7cf9f69b46-jpkf2"
Sep 18 20:13:10.097404 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:10.091403    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c2246a4-6b66-49ff-a268-b34347959d01-credential-socket\") pod \"e2e-ait-f524d4ec-openshift-default-7cf9f69b46-jpkf2\" (UID: \"1c2246a4-6b66-49ff-a268-b34347959d01\") " pod="openshift-ingress/e2e-ait-f524d4ec-openshift-default-7cf9f69b46-jpkf2"
Sep 18 20:13:10.097404 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:10.091448    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkgsm\" (UniqueName: \"kubernetes.io/projected/1c2246a4-6b66-49ff-a268-b34347959d01-kube-api-access-kkgsm\") pod \"e2e-ait-f524d4ec-openshift-default-7cf9f69b46-jpkf2\" (UID: \"1c2246a4-6b66-49ff-a268-b34347959d01\") " pod="openshift-ingress/e2e-ait-f524d4ec-openshift-default-7cf9f69b46-jpkf2"
Sep 18 20:13:10.097404 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:10.091487    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c2246a4-6b66-49ff-a268-b34347959d01-istio-token\") pod \"e2e-ait-f524d4ec-openshift-default-7cf9f69b46-jpkf2\" (UID: \"1c2246a4-6b66-49ff-a268-b34347959d01\") " pod="openshift-ingress/e2e-ait-f524d4ec-openshift-default-7cf9f69b46-jpkf2"
Sep 18 20:13:10.097404 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:10.091521    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c2246a4-6b66-49ff-a268-b34347959d01-istio-podinfo\") pod \"e2e-ait-f524d4ec-openshift-default-7cf9f69b46-jpkf2\" (UID: \"1c2246a4-6b66-49ff-a268-b34347959d01\") " pod="openshift-ingress/e2e-ait-f524d4ec-openshift-default-7cf9f69b46-jpkf2"
Sep 18 20:13:10.097404 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:10.091581    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c2246a4-6b66-49ff-a268-b34347959d01-istio-envoy\") pod \"e2e-ait-f524d4ec-openshift-default-7cf9f69b46-jpkf2\" (UID: \"1c2246a4-6b66-49ff-a268-b34347959d01\") " pod="openshift-ingress/e2e-ait-f524d4ec-openshift-default-7cf9f69b46-jpkf2"
Sep 18 20:13:10.192937 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:10.192904    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c2246a4-6b66-49ff-a268-b34347959d01-istio-data\") pod \"e2e-ait-f524d4ec-openshift-default-7cf9f69b46-jpkf2\" (UID: \"1c2246a4-6b66-49ff-a268-b34347959d01\") " pod="openshift-ingress/e2e-ait-f524d4ec-openshift-default-7cf9f69b46-jpkf2"
Sep 18 20:13:10.193140 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:10.193073    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c2246a4-6b66-49ff-a268-b34347959d01-istiod-ca-cert\") pod \"e2e-ait-f524d4ec-openshift-default-7cf9f69b46-jpkf2\" (UID: \"1c2246a4-6b66-49ff-a268-b34347959d01\") " pod="openshift-ingress/e2e-ait-f524d4ec-openshift-default-7cf9f69b46-jpkf2"
Sep 18 20:13:10.193140 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:10.193112    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c2246a4-6b66-49ff-a268-b34347959d01-workload-socket\") pod \"e2e-ait-f524d4ec-openshift-default-7cf9f69b46-jpkf2\" (UID: \"1c2246a4-6b66-49ff-a268-b34347959d01\") " pod="openshift-ingress/e2e-ait-f524d4ec-openshift-default-7cf9f69b46-jpkf2"
Sep 18 20:13:10.193275 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:10.193228    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c2246a4-6b66-49ff-a268-b34347959d01-workload-certs\") pod \"e2e-ait-f524d4ec-openshift-default-7cf9f69b46-jpkf2\" (UID: \"1c2246a4-6b66-49ff-a268-b34347959d01\") " pod="openshift-ingress/e2e-ait-f524d4ec-openshift-default-7cf9f69b46-jpkf2"
Sep 18 20:13:10.193275 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:10.193263    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c2246a4-6b66-49ff-a268-b34347959d01-credential-socket\") pod \"e2e-ait-f524d4ec-openshift-default-7cf9f69b46-jpkf2\" (UID: \"1c2246a4-6b66-49ff-a268-b34347959d01\") " pod="openshift-ingress/e2e-ait-f524d4ec-openshift-default-7cf9f69b46-jpkf2"
Sep 18 20:13:10.193419 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:10.193299    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkgsm\" (UniqueName: \"kubernetes.io/projected/1c2246a4-6b66-49ff-a268-b34347959d01-kube-api-access-kkgsm\") pod \"e2e-ait-f524d4ec-openshift-default-7cf9f69b46-jpkf2\" (UID: \"1c2246a4-6b66-49ff-a268-b34347959d01\") " pod="openshift-ingress/e2e-ait-f524d4ec-openshift-default-7cf9f69b46-jpkf2"
Sep 18 20:13:10.193419 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:10.193327    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c2246a4-6b66-49ff-a268-b34347959d01-istio-token\") pod \"e2e-ait-f524d4ec-openshift-default-7cf9f69b46-jpkf2\" (UID: \"1c2246a4-6b66-49ff-a268-b34347959d01\") " pod="openshift-ingress/e2e-ait-f524d4ec-openshift-default-7cf9f69b46-jpkf2"
Sep 18 20:13:10.193419 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:10.193350    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c2246a4-6b66-49ff-a268-b34347959d01-istio-podinfo\") pod \"e2e-ait-f524d4ec-openshift-default-7cf9f69b46-jpkf2\" (UID: \"1c2246a4-6b66-49ff-a268-b34347959d01\") " pod="openshift-ingress/e2e-ait-f524d4ec-openshift-default-7cf9f69b46-jpkf2"
Sep 18 20:13:10.193419 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:10.193369    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c2246a4-6b66-49ff-a268-b34347959d01-istio-data\") pod \"e2e-ait-f524d4ec-openshift-default-7cf9f69b46-jpkf2\" (UID: \"1c2246a4-6b66-49ff-a268-b34347959d01\") " pod="openshift-ingress/e2e-ait-f524d4ec-openshift-default-7cf9f69b46-jpkf2"
Sep 18 20:13:10.193627 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:10.193438    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c2246a4-6b66-49ff-a268-b34347959d01-istio-envoy\") pod \"e2e-ait-f524d4ec-openshift-default-7cf9f69b46-jpkf2\" (UID: \"1c2246a4-6b66-49ff-a268-b34347959d01\") " pod="openshift-ingress/e2e-ait-f524d4ec-openshift-default-7cf9f69b46-jpkf2"
Sep 18 20:13:10.193627 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:10.193445    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c2246a4-6b66-49ff-a268-b34347959d01-workload-socket\") pod \"e2e-ait-f524d4ec-openshift-default-7cf9f69b46-jpkf2\" (UID: \"1c2246a4-6b66-49ff-a268-b34347959d01\") " pod="openshift-ingress/e2e-ait-f524d4ec-openshift-default-7cf9f69b46-jpkf2"
Sep 18 20:13:10.193627 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:10.193591    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c2246a4-6b66-49ff-a268-b34347959d01-workload-certs\") pod \"e2e-ait-f524d4ec-openshift-default-7cf9f69b46-jpkf2\" (UID: \"1c2246a4-6b66-49ff-a268-b34347959d01\") " pod="openshift-ingress/e2e-ait-f524d4ec-openshift-default-7cf9f69b46-jpkf2"
Sep 18 20:13:10.193761 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:10.193743    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c2246a4-6b66-49ff-a268-b34347959d01-credential-socket\") pod \"e2e-ait-f524d4ec-openshift-default-7cf9f69b46-jpkf2\" (UID: \"1c2246a4-6b66-49ff-a268-b34347959d01\") " pod="openshift-ingress/e2e-ait-f524d4ec-openshift-default-7cf9f69b46-jpkf2"
Sep 18 20:13:10.193831 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:10.193813    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c2246a4-6b66-49ff-a268-b34347959d01-istiod-ca-cert\") pod \"e2e-ait-f524d4ec-openshift-default-7cf9f69b46-jpkf2\" (UID: \"1c2246a4-6b66-49ff-a268-b34347959d01\") " pod="openshift-ingress/e2e-ait-f524d4ec-openshift-default-7cf9f69b46-jpkf2"
Sep 18 20:13:10.195657 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:10.195634    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c2246a4-6b66-49ff-a268-b34347959d01-istio-envoy\") pod \"e2e-ait-f524d4ec-openshift-default-7cf9f69b46-jpkf2\" (UID: \"1c2246a4-6b66-49ff-a268-b34347959d01\") " pod="openshift-ingress/e2e-ait-f524d4ec-openshift-default-7cf9f69b46-jpkf2"
Sep 18 20:13:10.195961 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:10.195943    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c2246a4-6b66-49ff-a268-b34347959d01-istio-podinfo\") pod \"e2e-ait-f524d4ec-openshift-default-7cf9f69b46-jpkf2\" (UID: \"1c2246a4-6b66-49ff-a268-b34347959d01\") " pod="openshift-ingress/e2e-ait-f524d4ec-openshift-default-7cf9f69b46-jpkf2"
Sep 18 20:13:10.210394 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:10.210363    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c2246a4-6b66-49ff-a268-b34347959d01-istio-token\") pod \"e2e-ait-f524d4ec-openshift-default-7cf9f69b46-jpkf2\" (UID: \"1c2246a4-6b66-49ff-a268-b34347959d01\") " pod="openshift-ingress/e2e-ait-f524d4ec-openshift-default-7cf9f69b46-jpkf2"
Sep 18 20:13:10.210536 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:10.210521    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkgsm\" (UniqueName: \"kubernetes.io/projected/1c2246a4-6b66-49ff-a268-b34347959d01-kube-api-access-kkgsm\") pod \"e2e-ait-f524d4ec-openshift-default-7cf9f69b46-jpkf2\" (UID: \"1c2246a4-6b66-49ff-a268-b34347959d01\") " pod="openshift-ingress/e2e-ait-f524d4ec-openshift-default-7cf9f69b46-jpkf2"
Sep 18 20:13:10.309891 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:10.309824    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f524d4ec-openshift-default-7cf9f69b46-jpkf2"
Sep 18 20:13:10.589844 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:10.589819    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f524d4ec-openshift-default-7cf9f69b46-jpkf2"]
Sep 18 20:13:10.591422 ip-10-0-140-178 kubenswrapper[2549]: W0918 20:13:10.591397    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c2246a4_6b66_49ff_a268_b34347959d01.slice/crio-f4c336d3d04c89b58a519b66206ca32a47b9f8ea01b59652e560e6271673e688 WatchSource:0}: Error finding container f4c336d3d04c89b58a519b66206ca32a47b9f8ea01b59652e560e6271673e688: Status 404 returned error can't find the container with id f4c336d3d04c89b58a519b66206ca32a47b9f8ea01b59652e560e6271673e688
Sep 18 20:13:10.593667 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:10.593633    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:13:10.593749 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:10.593700    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:13:10.593749 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:10.593728    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:13:10.620188 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:10.620167    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f524d4ec-openshift-default-7cf9f69b46-jpkf2" event={"ID":"1c2246a4-6b66-49ff-a268-b34347959d01","Type":"ContainerStarted","Data":"f4c336d3d04c89b58a519b66206ca32a47b9f8ea01b59652e560e6271673e688"}
Sep 18 20:13:11.630009 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:11.629930    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f524d4ec-openshift-default-7cf9f69b46-jpkf2" event={"ID":"1c2246a4-6b66-49ff-a268-b34347959d01","Type":"ContainerStarted","Data":"68bd4b1f4fbf4255bf7b7682c01be874dcbbf7f7ac84e99c22c3caeba34a5662"}
Sep 18 20:13:11.663947 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:11.663898    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-f524d4ec-openshift-default-7cf9f69b46-jpkf2" podStartSLOduration=2.663883135 podStartE2EDuration="2.663883135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:13:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:13:11.663181142 +0000 UTC m=+976.616922324" watchObservedRunningTime="2026-09-18 20:13:11.663883135 +0000 UTC m=+976.617624302"
Sep 18 20:13:12.310568 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:12.310533    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f524d4ec-openshift-default-7cf9f69b46-jpkf2"
Sep 18 20:13:12.315647 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:12.315626    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f524d4ec-openshift-default-7cf9f69b46-jpkf2"
Sep 18 20:13:12.637955 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:12.637927    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f524d4ec-openshift-default-7cf9f69b46-jpkf2"
Sep 18 20:13:12.638865 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:12.638845    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f524d4ec-openshift-default-7cf9f69b46-jpkf2"
Sep 18 20:13:14.238858 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:14.238827    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-5284dc/simulator-w0-5284dc-kserve-84dc7dc7f-mpwht"
Sep 18 20:13:14.239188 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:14.238877    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-5284dc/simulator-w0-5284dc-kserve-84dc7dc7f-mpwht"
Sep 18 20:13:14.251585 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:14.251563    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-5284dc/simulator-w0-5284dc-kserve-84dc7dc7f-mpwht"
Sep 18 20:13:14.471825 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:14.471750    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h"
Sep 18 20:13:14.536132 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:14.536105    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e079f67-e825-45ef-b75d-14dbf11bb9f3-credential-socket\") pod \"7e079f67-e825-45ef-b75d-14dbf11bb9f3\" (UID: \"7e079f67-e825-45ef-b75d-14dbf11bb9f3\") "
Sep 18 20:13:14.536132 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:14.536138    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e079f67-e825-45ef-b75d-14dbf11bb9f3-istio-token\") pod \"7e079f67-e825-45ef-b75d-14dbf11bb9f3\" (UID: \"7e079f67-e825-45ef-b75d-14dbf11bb9f3\") "
Sep 18 20:13:14.536326 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:14.536173    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e079f67-e825-45ef-b75d-14dbf11bb9f3-istio-data\") pod \"7e079f67-e825-45ef-b75d-14dbf11bb9f3\" (UID: \"7e079f67-e825-45ef-b75d-14dbf11bb9f3\") "
Sep 18 20:13:14.536326 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:14.536219    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e079f67-e825-45ef-b75d-14dbf11bb9f3-istio-podinfo\") pod \"7e079f67-e825-45ef-b75d-14dbf11bb9f3\" (UID: \"7e079f67-e825-45ef-b75d-14dbf11bb9f3\") "
Sep 18 20:13:14.536326 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:14.536244    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e079f67-e825-45ef-b75d-14dbf11bb9f3-workload-socket\") pod \"7e079f67-e825-45ef-b75d-14dbf11bb9f3\" (UID: \"7e079f67-e825-45ef-b75d-14dbf11bb9f3\") "
Sep 18 20:13:14.536326 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:14.536282    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e079f67-e825-45ef-b75d-14dbf11bb9f3-workload-certs\") pod \"7e079f67-e825-45ef-b75d-14dbf11bb9f3\" (UID: \"7e079f67-e825-45ef-b75d-14dbf11bb9f3\") "
Sep 18 20:13:14.536539 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:14.536361    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e079f67-e825-45ef-b75d-14dbf11bb9f3-istiod-ca-cert\") pod \"7e079f67-e825-45ef-b75d-14dbf11bb9f3\" (UID: \"7e079f67-e825-45ef-b75d-14dbf11bb9f3\") "
Sep 18 20:13:14.536539 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:14.536418    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e079f67-e825-45ef-b75d-14dbf11bb9f3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7e079f67-e825-45ef-b75d-14dbf11bb9f3" (UID: "7e079f67-e825-45ef-b75d-14dbf11bb9f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:13:14.536539 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:14.536441    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e079f67-e825-45ef-b75d-14dbf11bb9f3-istio-envoy\") pod \"7e079f67-e825-45ef-b75d-14dbf11bb9f3\" (UID: \"7e079f67-e825-45ef-b75d-14dbf11bb9f3\") "
Sep 18 20:13:14.536539 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:14.536470    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-76j42\" (UniqueName: \"kubernetes.io/projected/7e079f67-e825-45ef-b75d-14dbf11bb9f3-kube-api-access-76j42\") pod \"7e079f67-e825-45ef-b75d-14dbf11bb9f3\" (UID: \"7e079f67-e825-45ef-b75d-14dbf11bb9f3\") "
Sep 18 20:13:14.536759 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:14.536577    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e079f67-e825-45ef-b75d-14dbf11bb9f3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7e079f67-e825-45ef-b75d-14dbf11bb9f3" (UID: "7e079f67-e825-45ef-b75d-14dbf11bb9f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:13:14.536759 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:14.536584    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e079f67-e825-45ef-b75d-14dbf11bb9f3-istio-data" (OuterVolumeSpecName: "istio-data") pod "7e079f67-e825-45ef-b75d-14dbf11bb9f3" (UID: "7e079f67-e825-45ef-b75d-14dbf11bb9f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:13:14.536759 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:14.536660    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e079f67-e825-45ef-b75d-14dbf11bb9f3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7e079f67-e825-45ef-b75d-14dbf11bb9f3" (UID: "7e079f67-e825-45ef-b75d-14dbf11bb9f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:13:14.536898 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:14.536796    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e079f67-e825-45ef-b75d-14dbf11bb9f3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7e079f67-e825-45ef-b75d-14dbf11bb9f3" (UID: "7e079f67-e825-45ef-b75d-14dbf11bb9f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:13:14.536958 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:14.536908    2549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e079f67-e825-45ef-b75d-14dbf11bb9f3-workload-socket\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:13:14.536958 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:14.536928    2549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e079f67-e825-45ef-b75d-14dbf11bb9f3-workload-certs\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:13:14.536958 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:14.536943    2549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e079f67-e825-45ef-b75d-14dbf11bb9f3-istiod-ca-cert\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:13:14.537073 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:14.536956    2549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e079f67-e825-45ef-b75d-14dbf11bb9f3-credential-socket\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:13:14.537073 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:14.536970    2549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e079f67-e825-45ef-b75d-14dbf11bb9f3-istio-data\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:13:14.538447 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:14.538425    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e079f67-e825-45ef-b75d-14dbf11bb9f3-istio-token" (OuterVolumeSpecName: "istio-token") pod "7e079f67-e825-45ef-b75d-14dbf11bb9f3" (UID: "7e079f67-e825-45ef-b75d-14dbf11bb9f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:13:14.538654 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:14.538624    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e079f67-e825-45ef-b75d-14dbf11bb9f3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7e079f67-e825-45ef-b75d-14dbf11bb9f3" (UID: "7e079f67-e825-45ef-b75d-14dbf11bb9f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:13:14.538836 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:14.538815    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e079f67-e825-45ef-b75d-14dbf11bb9f3-kube-api-access-76j42" (OuterVolumeSpecName: "kube-api-access-76j42") pod "7e079f67-e825-45ef-b75d-14dbf11bb9f3" (UID: "7e079f67-e825-45ef-b75d-14dbf11bb9f3"). InnerVolumeSpecName "kube-api-access-76j42". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:13:14.538897 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:14.538815    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7e079f67-e825-45ef-b75d-14dbf11bb9f3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7e079f67-e825-45ef-b75d-14dbf11bb9f3" (UID: "7e079f67-e825-45ef-b75d-14dbf11bb9f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 20:13:14.637723 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:14.637690    2549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e079f67-e825-45ef-b75d-14dbf11bb9f3-istio-envoy\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:13:14.637723 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:14.637717    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76j42\" (UniqueName: \"kubernetes.io/projected/7e079f67-e825-45ef-b75d-14dbf11bb9f3-kube-api-access-76j42\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:13:14.637910 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:14.637732    2549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e079f67-e825-45ef-b75d-14dbf11bb9f3-istio-token\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:13:14.637910 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:14.637746    2549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e079f67-e825-45ef-b75d-14dbf11bb9f3-istio-podinfo\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:13:14.651482 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:14.651451    2549 generic.go:356] "Generic (PLEG): container finished" podID="7e079f67-e825-45ef-b75d-14dbf11bb9f3" containerID="b84ce33bbbe2d2a05750235d47f54cce628420814220a37eb343ba342b45413f" exitCode=0
Sep 18 20:13:14.651638 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:14.651512    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h"
Sep 18 20:13:14.651638 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:14.651541    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h" event={"ID":"7e079f67-e825-45ef-b75d-14dbf11bb9f3","Type":"ContainerDied","Data":"b84ce33bbbe2d2a05750235d47f54cce628420814220a37eb343ba342b45413f"}
Sep 18 20:13:14.651638 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:14.651604    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h" event={"ID":"7e079f67-e825-45ef-b75d-14dbf11bb9f3","Type":"ContainerDied","Data":"a9ec6e8b674b5ebd6d1ea3abb44ca9771c2be0c0ea7f7a9c21d2d5491ffdb20b"}
Sep 18 20:13:14.651803 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:14.651656    2549 scope.go:117] "RemoveContainer" containerID="b84ce33bbbe2d2a05750235d47f54cce628420814220a37eb343ba342b45413f"
Sep 18 20:13:14.665560 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:14.665526    2549 scope.go:117] "RemoveContainer" containerID="abbde195d3fb03b1a8d77cf2d4862caeeee16e9734858be7ade8479b5dd57e94"
Sep 18 20:13:14.666415 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:14.666376    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-5284dc/simulator-w0-5284dc-kserve-84dc7dc7f-mpwht"
Sep 18 20:13:14.674263 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:14.674245    2549 scope.go:117] "RemoveContainer" containerID="b84ce33bbbe2d2a05750235d47f54cce628420814220a37eb343ba342b45413f"
Sep 18 20:13:14.674560 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:13:14.674541    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b84ce33bbbe2d2a05750235d47f54cce628420814220a37eb343ba342b45413f\": container with ID starting with b84ce33bbbe2d2a05750235d47f54cce628420814220a37eb343ba342b45413f not found: ID does not exist" containerID="b84ce33bbbe2d2a05750235d47f54cce628420814220a37eb343ba342b45413f"
Sep 18 20:13:14.674626 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:14.674571    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b84ce33bbbe2d2a05750235d47f54cce628420814220a37eb343ba342b45413f"} err="failed to get container status \"b84ce33bbbe2d2a05750235d47f54cce628420814220a37eb343ba342b45413f\": rpc error: code = NotFound desc = could not find container \"b84ce33bbbe2d2a05750235d47f54cce628420814220a37eb343ba342b45413f\": container with ID starting with b84ce33bbbe2d2a05750235d47f54cce628420814220a37eb343ba342b45413f not found: ID does not exist"
Sep 18 20:13:14.674626 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:14.674588    2549 scope.go:117] "RemoveContainer" containerID="abbde195d3fb03b1a8d77cf2d4862caeeee16e9734858be7ade8479b5dd57e94"
Sep 18 20:13:14.674861 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:13:14.674834    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abbde195d3fb03b1a8d77cf2d4862caeeee16e9734858be7ade8479b5dd57e94\": container with ID starting with abbde195d3fb03b1a8d77cf2d4862caeeee16e9734858be7ade8479b5dd57e94 not found: ID does not exist" containerID="abbde195d3fb03b1a8d77cf2d4862caeeee16e9734858be7ade8479b5dd57e94"
Sep 18 20:13:14.674954 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:14.674869    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abbde195d3fb03b1a8d77cf2d4862caeeee16e9734858be7ade8479b5dd57e94"} err="failed to get container status \"abbde195d3fb03b1a8d77cf2d4862caeeee16e9734858be7ade8479b5dd57e94\": rpc error: code = NotFound desc = could not find container \"abbde195d3fb03b1a8d77cf2d4862caeeee16e9734858be7ade8479b5dd57e94\": container with ID starting with abbde195d3fb03b1a8d77cf2d4862caeeee16e9734858be7ade8479b5dd57e94 not found: ID does not exist"
Sep 18 20:13:14.678848 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:14.678826    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h"]
Sep 18 20:13:14.685824 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:14.682819    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-7bed33dc-gw-openshift-default-d5c7b4cd4-hxs6h"]
Sep 18 20:13:15.521291 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:15.521258    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e079f67-e825-45ef-b75d-14dbf11bb9f3" path="/var/lib/kubelet/pods/7e079f67-e825-45ef-b75d-14dbf11bb9f3/volumes"
Sep 18 20:13:22.913766 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:22.913686    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:13:34.771653 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:34.771620    2549 generic.go:356] "Generic (PLEG): container finished" podID="1c2246a4-6b66-49ff-a268-b34347959d01" containerID="68bd4b1f4fbf4255bf7b7682c01be874dcbbf7f7ac84e99c22c3caeba34a5662" exitCode=0
Sep 18 20:13:34.772076 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:34.771670    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f524d4ec-openshift-default-7cf9f69b46-jpkf2" event={"ID":"1c2246a4-6b66-49ff-a268-b34347959d01","Type":"ContainerDied","Data":"68bd4b1f4fbf4255bf7b7682c01be874dcbbf7f7ac84e99c22c3caeba34a5662"}
Sep 18 20:13:34.772076 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:34.772015    2549 scope.go:117] "RemoveContainer" containerID="68bd4b1f4fbf4255bf7b7682c01be874dcbbf7f7ac84e99c22c3caeba34a5662"
Sep 18 20:13:34.773113 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:34.773086    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:13:34.773181 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:34.773154    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:13:34.773219 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:34.773189    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:13:35.780521 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:35.780486    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f524d4ec-openshift-default-7cf9f69b46-jpkf2" event={"ID":"1c2246a4-6b66-49ff-a268-b34347959d01","Type":"ContainerStarted","Data":"c70a50ccd0cf0b41540c0a3d8c417b61d2c041d648f23e9e77716ae982643592"}
Sep 18 20:13:36.310740 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:36.310710    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f524d4ec-openshift-default-7cf9f69b46-jpkf2"
Sep 18 20:13:36.311002 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:36.310979    2549 patch_prober.go:28] interesting pod/e2e-ait-f524d4ec-openshift-default-7cf9f69b46-jpkf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Sep 18 20:13:36.311101 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:36.311037    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f524d4ec-openshift-default-7cf9f69b46-jpkf2" podUID="1c2246a4-6b66-49ff-a268-b34347959d01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Sep 18 20:13:36.941306 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:36.941275    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-4b633d/simulator-w3-4b633d-kserve-5f479956c4-hvpjd"]
Sep 18 20:13:36.941900 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:36.941879    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e079f67-e825-45ef-b75d-14dbf11bb9f3" containerName="istio-proxy"
Sep 18 20:13:36.942008 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:36.941903    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e079f67-e825-45ef-b75d-14dbf11bb9f3" containerName="istio-proxy"
Sep 18 20:13:36.942078 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:36.942043    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e079f67-e825-45ef-b75d-14dbf11bb9f3" containerName="istio-proxy"
Sep 18 20:13:36.942078 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:36.942058    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e079f67-e825-45ef-b75d-14dbf11bb9f3" containerName="istio-proxy"
Sep 18 20:13:36.944153 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:36.944126    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4b633d/simulator-w3-4b633d-kserve-5f479956c4-hvpjd"
Sep 18 20:13:36.951477 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:36.951432    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-4b633d\"/\"simulator-w3-4b633d-kserve-self-signed-certs\""
Sep 18 20:13:36.952784 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:36.951824    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-4b633d\"/\"kube-root-ca.crt\""
Sep 18 20:13:36.952784 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:36.951871    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-4b633d\"/\"openshift-service-ca.crt\""
Sep 18 20:13:36.952784 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:36.951935    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-4b633d\"/\"default-dockercfg-2vsff\""
Sep 18 20:13:36.961703 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:36.961672    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-4b633d/simulator-w3-4b633d-kserve-5f479956c4-hvpjd"]
Sep 18 20:13:37.076308 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:37.076256    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35274823-b198-4e3f-a8b8-d9ef1661e1b9-dshm\") pod \"simulator-w3-4b633d-kserve-5f479956c4-hvpjd\" (UID: \"35274823-b198-4e3f-a8b8-d9ef1661e1b9\") " pod="e2e-models-e2e-worker-w3-4b633d/simulator-w3-4b633d-kserve-5f479956c4-hvpjd"
Sep 18 20:13:37.076308 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:37.076313    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35274823-b198-4e3f-a8b8-d9ef1661e1b9-home\") pod \"simulator-w3-4b633d-kserve-5f479956c4-hvpjd\" (UID: \"35274823-b198-4e3f-a8b8-d9ef1661e1b9\") " pod="e2e-models-e2e-worker-w3-4b633d/simulator-w3-4b633d-kserve-5f479956c4-hvpjd"
Sep 18 20:13:37.076555 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:37.076412    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35274823-b198-4e3f-a8b8-d9ef1661e1b9-tmp-dir\") pod \"simulator-w3-4b633d-kserve-5f479956c4-hvpjd\" (UID: \"35274823-b198-4e3f-a8b8-d9ef1661e1b9\") " pod="e2e-models-e2e-worker-w3-4b633d/simulator-w3-4b633d-kserve-5f479956c4-hvpjd"
Sep 18 20:13:37.076555 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:37.076454    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35274823-b198-4e3f-a8b8-d9ef1661e1b9-model-cache\") pod \"simulator-w3-4b633d-kserve-5f479956c4-hvpjd\" (UID: \"35274823-b198-4e3f-a8b8-d9ef1661e1b9\") " pod="e2e-models-e2e-worker-w3-4b633d/simulator-w3-4b633d-kserve-5f479956c4-hvpjd"
Sep 18 20:13:37.076625 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:37.076556    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8cj8\" (UniqueName: \"kubernetes.io/projected/35274823-b198-4e3f-a8b8-d9ef1661e1b9-kube-api-access-c8cj8\") pod \"simulator-w3-4b633d-kserve-5f479956c4-hvpjd\" (UID: \"35274823-b198-4e3f-a8b8-d9ef1661e1b9\") " pod="e2e-models-e2e-worker-w3-4b633d/simulator-w3-4b633d-kserve-5f479956c4-hvpjd"
Sep 18 20:13:37.076625 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:37.076615    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35274823-b198-4e3f-a8b8-d9ef1661e1b9-tls-certs\") pod \"simulator-w3-4b633d-kserve-5f479956c4-hvpjd\" (UID: \"35274823-b198-4e3f-a8b8-d9ef1661e1b9\") " pod="e2e-models-e2e-worker-w3-4b633d/simulator-w3-4b633d-kserve-5f479956c4-hvpjd"
Sep 18 20:13:37.177305 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:37.177252    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8cj8\" (UniqueName: \"kubernetes.io/projected/35274823-b198-4e3f-a8b8-d9ef1661e1b9-kube-api-access-c8cj8\") pod \"simulator-w3-4b633d-kserve-5f479956c4-hvpjd\" (UID: \"35274823-b198-4e3f-a8b8-d9ef1661e1b9\") " pod="e2e-models-e2e-worker-w3-4b633d/simulator-w3-4b633d-kserve-5f479956c4-hvpjd"
Sep 18 20:13:37.177531 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:37.177341    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35274823-b198-4e3f-a8b8-d9ef1661e1b9-tls-certs\") pod \"simulator-w3-4b633d-kserve-5f479956c4-hvpjd\" (UID: \"35274823-b198-4e3f-a8b8-d9ef1661e1b9\") " pod="e2e-models-e2e-worker-w3-4b633d/simulator-w3-4b633d-kserve-5f479956c4-hvpjd"
Sep 18 20:13:37.177531 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:37.177437    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35274823-b198-4e3f-a8b8-d9ef1661e1b9-dshm\") pod \"simulator-w3-4b633d-kserve-5f479956c4-hvpjd\" (UID: \"35274823-b198-4e3f-a8b8-d9ef1661e1b9\") " pod="e2e-models-e2e-worker-w3-4b633d/simulator-w3-4b633d-kserve-5f479956c4-hvpjd"
Sep 18 20:13:37.177531 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:37.177470    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35274823-b198-4e3f-a8b8-d9ef1661e1b9-home\") pod \"simulator-w3-4b633d-kserve-5f479956c4-hvpjd\" (UID: \"35274823-b198-4e3f-a8b8-d9ef1661e1b9\") " pod="e2e-models-e2e-worker-w3-4b633d/simulator-w3-4b633d-kserve-5f479956c4-hvpjd"
Sep 18 20:13:37.177705 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:37.177552    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35274823-b198-4e3f-a8b8-d9ef1661e1b9-tmp-dir\") pod \"simulator-w3-4b633d-kserve-5f479956c4-hvpjd\" (UID: \"35274823-b198-4e3f-a8b8-d9ef1661e1b9\") " pod="e2e-models-e2e-worker-w3-4b633d/simulator-w3-4b633d-kserve-5f479956c4-hvpjd"
Sep 18 20:13:37.177705 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:37.177592    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35274823-b198-4e3f-a8b8-d9ef1661e1b9-model-cache\") pod \"simulator-w3-4b633d-kserve-5f479956c4-hvpjd\" (UID: \"35274823-b198-4e3f-a8b8-d9ef1661e1b9\") " pod="e2e-models-e2e-worker-w3-4b633d/simulator-w3-4b633d-kserve-5f479956c4-hvpjd"
Sep 18 20:13:37.178110 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:37.178082    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35274823-b198-4e3f-a8b8-d9ef1661e1b9-model-cache\") pod \"simulator-w3-4b633d-kserve-5f479956c4-hvpjd\" (UID: \"35274823-b198-4e3f-a8b8-d9ef1661e1b9\") " pod="e2e-models-e2e-worker-w3-4b633d/simulator-w3-4b633d-kserve-5f479956c4-hvpjd"
Sep 18 20:13:37.179169 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:37.179144    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35274823-b198-4e3f-a8b8-d9ef1661e1b9-home\") pod \"simulator-w3-4b633d-kserve-5f479956c4-hvpjd\" (UID: \"35274823-b198-4e3f-a8b8-d9ef1661e1b9\") " pod="e2e-models-e2e-worker-w3-4b633d/simulator-w3-4b633d-kserve-5f479956c4-hvpjd"
Sep 18 20:13:37.179370 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:37.179322    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35274823-b198-4e3f-a8b8-d9ef1661e1b9-tmp-dir\") pod \"simulator-w3-4b633d-kserve-5f479956c4-hvpjd\" (UID: \"35274823-b198-4e3f-a8b8-d9ef1661e1b9\") " pod="e2e-models-e2e-worker-w3-4b633d/simulator-w3-4b633d-kserve-5f479956c4-hvpjd"
Sep 18 20:13:37.193449 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:37.181954    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35274823-b198-4e3f-a8b8-d9ef1661e1b9-tls-certs\") pod \"simulator-w3-4b633d-kserve-5f479956c4-hvpjd\" (UID: \"35274823-b198-4e3f-a8b8-d9ef1661e1b9\") " pod="e2e-models-e2e-worker-w3-4b633d/simulator-w3-4b633d-kserve-5f479956c4-hvpjd"
Sep 18 20:13:37.193659 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:37.181988    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35274823-b198-4e3f-a8b8-d9ef1661e1b9-dshm\") pod \"simulator-w3-4b633d-kserve-5f479956c4-hvpjd\" (UID: \"35274823-b198-4e3f-a8b8-d9ef1661e1b9\") " pod="e2e-models-e2e-worker-w3-4b633d/simulator-w3-4b633d-kserve-5f479956c4-hvpjd"
Sep 18 20:13:37.193774 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:37.188592    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8cj8\" (UniqueName: \"kubernetes.io/projected/35274823-b198-4e3f-a8b8-d9ef1661e1b9-kube-api-access-c8cj8\") pod \"simulator-w3-4b633d-kserve-5f479956c4-hvpjd\" (UID: \"35274823-b198-4e3f-a8b8-d9ef1661e1b9\") " pod="e2e-models-e2e-worker-w3-4b633d/simulator-w3-4b633d-kserve-5f479956c4-hvpjd"
Sep 18 20:13:37.281852 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:37.281815    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4b633d/simulator-w3-4b633d-kserve-5f479956c4-hvpjd"
Sep 18 20:13:37.310402 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:37.310353    2549 patch_prober.go:28] interesting pod/e2e-ait-f524d4ec-openshift-default-7cf9f69b46-jpkf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Sep 18 20:13:37.310532 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:37.310438    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f524d4ec-openshift-default-7cf9f69b46-jpkf2" podUID="1c2246a4-6b66-49ff-a268-b34347959d01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Sep 18 20:13:37.556198 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:37.556105    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-4b633d/simulator-w3-4b633d-kserve-5f479956c4-hvpjd"]
Sep 18 20:13:37.580940 ip-10-0-140-178 kubenswrapper[2549]: W0918 20:13:37.580903    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod35274823_b198_4e3f_a8b8_d9ef1661e1b9.slice/crio-feda6500bc00145aaabd46de01d79d0d65fe1b54476ac5709cbb1e2b3a10b88c WatchSource:0}: Error finding container feda6500bc00145aaabd46de01d79d0d65fe1b54476ac5709cbb1e2b3a10b88c: Status 404 returned error can't find the container with id feda6500bc00145aaabd46de01d79d0d65fe1b54476ac5709cbb1e2b3a10b88c
Sep 18 20:13:37.796475 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:37.796433    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4b633d/simulator-w3-4b633d-kserve-5f479956c4-hvpjd" event={"ID":"35274823-b198-4e3f-a8b8-d9ef1661e1b9","Type":"ContainerStarted","Data":"d5d1bf40e890c0a5f47f57ee66301166664d04bbdc3ad3fe90c3d8ce533f0878"}
Sep 18 20:13:37.796662 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:37.796483    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4b633d/simulator-w3-4b633d-kserve-5f479956c4-hvpjd" event={"ID":"35274823-b198-4e3f-a8b8-d9ef1661e1b9","Type":"ContainerStarted","Data":"feda6500bc00145aaabd46de01d79d0d65fe1b54476ac5709cbb1e2b3a10b88c"}
Sep 18 20:13:37.829884 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:37.829833    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-4b633d/simulator-w3-4b633d-kserve-5f479956c4-hvpjd" podStartSLOduration=1.829816949 podStartE2EDuration="1.829816949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:13:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:13:37.829035505 +0000 UTC m=+1002.782776673" watchObservedRunningTime="2026-09-18 20:13:37.829816949 +0000 UTC m=+1002.783558120"
Sep 18 20:13:38.310684 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:38.310600    2549 patch_prober.go:28] interesting pod/e2e-ait-f524d4ec-openshift-default-7cf9f69b46-jpkf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Sep 18 20:13:38.310684 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:38.310663    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f524d4ec-openshift-default-7cf9f69b46-jpkf2" podUID="1c2246a4-6b66-49ff-a268-b34347959d01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Sep 18 20:13:39.311402 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:39.310952    2549 patch_prober.go:28] interesting pod/e2e-ait-f524d4ec-openshift-default-7cf9f69b46-jpkf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Sep 18 20:13:39.311402 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:39.311015    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f524d4ec-openshift-default-7cf9f69b46-jpkf2" podUID="1c2246a4-6b66-49ff-a268-b34347959d01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Sep 18 20:13:40.310996 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:40.310968    2549 patch_prober.go:28] interesting pod/e2e-ait-f524d4ec-openshift-default-7cf9f69b46-jpkf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Sep 18 20:13:40.311225 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:40.311203    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f524d4ec-openshift-default-7cf9f69b46-jpkf2" podUID="1c2246a4-6b66-49ff-a268-b34347959d01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Sep 18 20:13:41.314279 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:41.314250    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f524d4ec-openshift-default-7cf9f69b46-jpkf2"
Sep 18 20:13:41.314681 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:41.314661    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f524d4ec-openshift-default-7cf9f69b46-jpkf2"
Sep 18 20:13:41.315337 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:41.315314    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f524d4ec-openshift-default-7cf9f69b46-jpkf2"
Sep 18 20:13:43.116937 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:43.116902    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:13:43.892520 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:43.892490    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f524d4ec-b25v4"]
Sep 18 20:13:43.892932 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:43.892917    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e079f67-e825-45ef-b75d-14dbf11bb9f3" containerName="istio-proxy"
Sep 18 20:13:43.892932 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:43.892932    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e079f67-e825-45ef-b75d-14dbf11bb9f3" containerName="istio-proxy"
Sep 18 20:13:43.897724 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:43.897707    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f524d4ec-b25v4"
Sep 18 20:13:43.901889 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:43.901861    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bhssz\""
Sep 18 20:13:43.902141 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:43.902109    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 20:13:43.903895 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:43.903867    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f524d4ec-b25v4"]
Sep 18 20:13:43.943427 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:43.943370    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc6f06f2-cd92-4761-b2cf-1e260dd86a67-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f524d4ec-b25v4\" (UID: \"dc6f06f2-cd92-4761-b2cf-1e260dd86a67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f524d4ec-b25v4"
Sep 18 20:13:44.044576 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:44.044544    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc6f06f2-cd92-4761-b2cf-1e260dd86a67-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f524d4ec-b25v4\" (UID: \"dc6f06f2-cd92-4761-b2cf-1e260dd86a67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f524d4ec-b25v4"
Sep 18 20:13:44.045103 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:44.045083    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc6f06f2-cd92-4761-b2cf-1e260dd86a67-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f524d4ec-b25v4\" (UID: \"dc6f06f2-cd92-4761-b2cf-1e260dd86a67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f524d4ec-b25v4"
Sep 18 20:13:44.211269 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:44.211197    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f524d4ec-b25v4"
Sep 18 20:13:44.490346 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:44.490312    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f524d4ec-b25v4"]
Sep 18 20:13:44.492071 ip-10-0-140-178 kubenswrapper[2549]: W0918 20:13:44.492030    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc6f06f2_cd92_4761_b2cf_1e260dd86a67.slice/crio-820edb0cd2acb0bffc82af084c46f15468c505d0e4902c0c6b3d55bf7f65d4ed WatchSource:0}: Error finding container 820edb0cd2acb0bffc82af084c46f15468c505d0e4902c0c6b3d55bf7f65d4ed: Status 404 returned error can't find the container with id 820edb0cd2acb0bffc82af084c46f15468c505d0e4902c0c6b3d55bf7f65d4ed
Sep 18 20:13:44.840554 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:44.840518    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f524d4ec-b25v4" event={"ID":"dc6f06f2-cd92-4761-b2cf-1e260dd86a67","Type":"ContainerStarted","Data":"820edb0cd2acb0bffc82af084c46f15468c505d0e4902c0c6b3d55bf7f65d4ed"}
Sep 18 20:13:45.848737 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:45.848705    2549 generic.go:356] "Generic (PLEG): container finished" podID="dc6f06f2-cd92-4761-b2cf-1e260dd86a67" containerID="0a59a8974185c9390dd16ea7098df0e9bcfd4851c7988e899ed2a5ea5c2527a2" exitCode=0
Sep 18 20:13:45.849181 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:45.848799    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f524d4ec-b25v4" event={"ID":"dc6f06f2-cd92-4761-b2cf-1e260dd86a67","Type":"ContainerDied","Data":"0a59a8974185c9390dd16ea7098df0e9bcfd4851c7988e899ed2a5ea5c2527a2"}
Sep 18 20:13:46.988576 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:46.988548    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f524d4ec-b25v4"
Sep 18 20:13:47.075195 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:47.075165    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc6f06f2-cd92-4761-b2cf-1e260dd86a67-maas-api-service-ca\") pod \"dc6f06f2-cd92-4761-b2cf-1e260dd86a67\" (UID: \"dc6f06f2-cd92-4761-b2cf-1e260dd86a67\") "
Sep 18 20:13:47.075527 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:47.075504    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc6f06f2-cd92-4761-b2cf-1e260dd86a67-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "dc6f06f2-cd92-4761-b2cf-1e260dd86a67" (UID: "dc6f06f2-cd92-4761-b2cf-1e260dd86a67"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:13:47.176837 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:47.176761    2549 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc6f06f2-cd92-4761-b2cf-1e260dd86a67-maas-api-service-ca\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:13:47.282311 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:47.282278    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-4b633d/simulator-w3-4b633d-kserve-5f479956c4-hvpjd"
Sep 18 20:13:47.282499 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:47.282324    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4b633d/simulator-w3-4b633d-kserve-5f479956c4-hvpjd"
Sep 18 20:13:47.294712 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:47.294687    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-4b633d/simulator-w3-4b633d-kserve-5f479956c4-hvpjd"
Sep 18 20:13:47.863427 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:47.863365    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f524d4ec-b25v4" event={"ID":"dc6f06f2-cd92-4761-b2cf-1e260dd86a67","Type":"ContainerDied","Data":"820edb0cd2acb0bffc82af084c46f15468c505d0e4902c0c6b3d55bf7f65d4ed"}
Sep 18 20:13:47.863598 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:47.863440    2549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="820edb0cd2acb0bffc82af084c46f15468c505d0e4902c0c6b3d55bf7f65d4ed"
Sep 18 20:13:47.863598 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:47.863391    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f524d4ec-b25v4"
Sep 18 20:13:47.874793 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:47.874771    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4b633d/simulator-w3-4b633d-kserve-5f479956c4-hvpjd"
Sep 18 20:13:55.338923 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:55.338889    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f524d4ec-b25v4"]
Sep 18 20:13:55.343075 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:55.343047    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f524d4ec-b25v4"]
Sep 18 20:13:55.525056 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:55.525025    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc6f06f2-cd92-4761-b2cf-1e260dd86a67" path="/var/lib/kubelet/pods/dc6f06f2-cd92-4761-b2cf-1e260dd86a67/volumes"
Sep 18 20:13:56.518704 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:56.518673    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f524d4ec-openshift-default-7cf9f69b46-jpkf2"]
Sep 18 20:13:56.519063 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:56.518863    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-f524d4ec-openshift-default-7cf9f69b46-jpkf2" podUID="1c2246a4-6b66-49ff-a268-b34347959d01" containerName="istio-proxy" containerID="cri-o://c70a50ccd0cf0b41540c0a3d8c417b61d2c041d648f23e9e77716ae982643592" gracePeriod=30
Sep 18 20:13:57.239373 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:57.239340    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g"]
Sep 18 20:13:57.239789 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:57.239775    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc6f06f2-cd92-4761-b2cf-1e260dd86a67" containerName="revoke-keys"
Sep 18 20:13:57.239789 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:57.239790    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc6f06f2-cd92-4761-b2cf-1e260dd86a67" containerName="revoke-keys"
Sep 18 20:13:57.239905 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:57.239877    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc6f06f2-cd92-4761-b2cf-1e260dd86a67" containerName="revoke-keys"
Sep 18 20:13:57.244183 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:57.244165    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g"
Sep 18 20:13:57.247079 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:57.247056    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-fa77d0d6-openshift-default-dockercfg-zj894\""
Sep 18 20:13:57.258454 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:57.258430    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g"]
Sep 18 20:13:57.379835 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:57.379796    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf0d422b-5ebd-4b33-82cd-59b34033ed77-istiod-ca-cert\") pod \"e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g\" (UID: \"cf0d422b-5ebd-4b33-82cd-59b34033ed77\") " pod="openshift-ingress/e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g"
Sep 18 20:13:57.379835 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:57.379841    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf0d422b-5ebd-4b33-82cd-59b34033ed77-workload-socket\") pod \"e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g\" (UID: \"cf0d422b-5ebd-4b33-82cd-59b34033ed77\") " pod="openshift-ingress/e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g"
Sep 18 20:13:57.380054 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:57.379857    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sv26p\" (UniqueName: \"kubernetes.io/projected/cf0d422b-5ebd-4b33-82cd-59b34033ed77-kube-api-access-sv26p\") pod \"e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g\" (UID: \"cf0d422b-5ebd-4b33-82cd-59b34033ed77\") " pod="openshift-ingress/e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g"
Sep 18 20:13:57.380054 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:57.379916    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf0d422b-5ebd-4b33-82cd-59b34033ed77-istio-token\") pod \"e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g\" (UID: \"cf0d422b-5ebd-4b33-82cd-59b34033ed77\") " pod="openshift-ingress/e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g"
Sep 18 20:13:57.380054 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:57.380020    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf0d422b-5ebd-4b33-82cd-59b34033ed77-credential-socket\") pod \"e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g\" (UID: \"cf0d422b-5ebd-4b33-82cd-59b34033ed77\") " pod="openshift-ingress/e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g"
Sep 18 20:13:57.380054 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:57.380038    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf0d422b-5ebd-4b33-82cd-59b34033ed77-istio-envoy\") pod \"e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g\" (UID: \"cf0d422b-5ebd-4b33-82cd-59b34033ed77\") " pod="openshift-ingress/e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g"
Sep 18 20:13:57.380201 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:57.380064    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf0d422b-5ebd-4b33-82cd-59b34033ed77-istio-podinfo\") pod \"e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g\" (UID: \"cf0d422b-5ebd-4b33-82cd-59b34033ed77\") " pod="openshift-ingress/e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g"
Sep 18 20:13:57.380201 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:57.380086    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf0d422b-5ebd-4b33-82cd-59b34033ed77-workload-certs\") pod \"e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g\" (UID: \"cf0d422b-5ebd-4b33-82cd-59b34033ed77\") " pod="openshift-ingress/e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g"
Sep 18 20:13:57.380201 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:57.380111    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf0d422b-5ebd-4b33-82cd-59b34033ed77-istio-data\") pod \"e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g\" (UID: \"cf0d422b-5ebd-4b33-82cd-59b34033ed77\") " pod="openshift-ingress/e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g"
Sep 18 20:13:57.481339 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:57.481309    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf0d422b-5ebd-4b33-82cd-59b34033ed77-istiod-ca-cert\") pod \"e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g\" (UID: \"cf0d422b-5ebd-4b33-82cd-59b34033ed77\") " pod="openshift-ingress/e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g"
Sep 18 20:13:57.481514 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:57.481345    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf0d422b-5ebd-4b33-82cd-59b34033ed77-workload-socket\") pod \"e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g\" (UID: \"cf0d422b-5ebd-4b33-82cd-59b34033ed77\") " pod="openshift-ingress/e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g"
Sep 18 20:13:57.481514 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:57.481366    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv26p\" (UniqueName: \"kubernetes.io/projected/cf0d422b-5ebd-4b33-82cd-59b34033ed77-kube-api-access-sv26p\") pod \"e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g\" (UID: \"cf0d422b-5ebd-4b33-82cd-59b34033ed77\") " pod="openshift-ingress/e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g"
Sep 18 20:13:57.481514 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:57.481421    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf0d422b-5ebd-4b33-82cd-59b34033ed77-istio-token\") pod \"e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g\" (UID: \"cf0d422b-5ebd-4b33-82cd-59b34033ed77\") " pod="openshift-ingress/e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g"
Sep 18 20:13:57.481514 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:57.481504    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf0d422b-5ebd-4b33-82cd-59b34033ed77-credential-socket\") pod \"e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g\" (UID: \"cf0d422b-5ebd-4b33-82cd-59b34033ed77\") " pod="openshift-ingress/e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g"
Sep 18 20:13:57.481712 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:57.481529    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf0d422b-5ebd-4b33-82cd-59b34033ed77-istio-envoy\") pod \"e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g\" (UID: \"cf0d422b-5ebd-4b33-82cd-59b34033ed77\") " pod="openshift-ingress/e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g"
Sep 18 20:13:57.481712 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:57.481563    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf0d422b-5ebd-4b33-82cd-59b34033ed77-istio-podinfo\") pod \"e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g\" (UID: \"cf0d422b-5ebd-4b33-82cd-59b34033ed77\") " pod="openshift-ingress/e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g"
Sep 18 20:13:57.481712 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:57.481643    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf0d422b-5ebd-4b33-82cd-59b34033ed77-workload-certs\") pod \"e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g\" (UID: \"cf0d422b-5ebd-4b33-82cd-59b34033ed77\") " pod="openshift-ingress/e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g"
Sep 18 20:13:57.481712 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:57.481702    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf0d422b-5ebd-4b33-82cd-59b34033ed77-istio-data\") pod \"e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g\" (UID: \"cf0d422b-5ebd-4b33-82cd-59b34033ed77\") " pod="openshift-ingress/e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g"
Sep 18 20:13:57.481925 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:57.481882    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf0d422b-5ebd-4b33-82cd-59b34033ed77-workload-socket\") pod \"e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g\" (UID: \"cf0d422b-5ebd-4b33-82cd-59b34033ed77\") " pod="openshift-ingress/e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g"
Sep 18 20:13:57.481992 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:57.481960    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf0d422b-5ebd-4b33-82cd-59b34033ed77-workload-certs\") pod \"e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g\" (UID: \"cf0d422b-5ebd-4b33-82cd-59b34033ed77\") " pod="openshift-ingress/e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g"
Sep 18 20:13:57.482089 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:57.482067    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf0d422b-5ebd-4b33-82cd-59b34033ed77-istio-data\") pod \"e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g\" (UID: \"cf0d422b-5ebd-4b33-82cd-59b34033ed77\") " pod="openshift-ingress/e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g"
Sep 18 20:13:57.482197 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:57.482176    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf0d422b-5ebd-4b33-82cd-59b34033ed77-istiod-ca-cert\") pod \"e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g\" (UID: \"cf0d422b-5ebd-4b33-82cd-59b34033ed77\") " pod="openshift-ingress/e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g"
Sep 18 20:13:57.482404 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:57.482361    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf0d422b-5ebd-4b33-82cd-59b34033ed77-credential-socket\") pod \"e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g\" (UID: \"cf0d422b-5ebd-4b33-82cd-59b34033ed77\") " pod="openshift-ingress/e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g"
Sep 18 20:13:57.483818 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:57.483799    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf0d422b-5ebd-4b33-82cd-59b34033ed77-istio-envoy\") pod \"e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g\" (UID: \"cf0d422b-5ebd-4b33-82cd-59b34033ed77\") " pod="openshift-ingress/e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g"
Sep 18 20:13:57.484040 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:57.484024    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf0d422b-5ebd-4b33-82cd-59b34033ed77-istio-podinfo\") pod \"e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g\" (UID: \"cf0d422b-5ebd-4b33-82cd-59b34033ed77\") " pod="openshift-ingress/e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g"
Sep 18 20:13:57.490188 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:57.490134    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf0d422b-5ebd-4b33-82cd-59b34033ed77-istio-token\") pod \"e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g\" (UID: \"cf0d422b-5ebd-4b33-82cd-59b34033ed77\") " pod="openshift-ingress/e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g"
Sep 18 20:13:57.490545 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:57.490530    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv26p\" (UniqueName: \"kubernetes.io/projected/cf0d422b-5ebd-4b33-82cd-59b34033ed77-kube-api-access-sv26p\") pod \"e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g\" (UID: \"cf0d422b-5ebd-4b33-82cd-59b34033ed77\") " pod="openshift-ingress/e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g"
Sep 18 20:13:57.556316 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:57.556284    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g"
Sep 18 20:13:57.842163 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:57.842133    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g"]
Sep 18 20:13:57.844145 ip-10-0-140-178 kubenswrapper[2549]: W0918 20:13:57.844114    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf0d422b_5ebd_4b33_82cd_59b34033ed77.slice/crio-5be64867f193701da8dc82cba13fa30287ee1968393745591de913d30142d952 WatchSource:0}: Error finding container 5be64867f193701da8dc82cba13fa30287ee1968393745591de913d30142d952: Status 404 returned error can't find the container with id 5be64867f193701da8dc82cba13fa30287ee1968393745591de913d30142d952
Sep 18 20:13:57.846949 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:57.846887    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:13:57.847087 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:57.846999    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:13:57.847087 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:57.847046    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:13:57.927402 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:57.927353    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g" event={"ID":"cf0d422b-5ebd-4b33-82cd-59b34033ed77","Type":"ContainerStarted","Data":"23b8c06a5e9bcc967066fa0d129874542639d24a99543680bdaade932b0f43b4"}
Sep 18 20:13:57.927547 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:57.927445    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g" event={"ID":"cf0d422b-5ebd-4b33-82cd-59b34033ed77","Type":"ContainerStarted","Data":"5be64867f193701da8dc82cba13fa30287ee1968393745591de913d30142d952"}
Sep 18 20:13:58.557060 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:58.557019    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g"
Sep 18 20:13:58.562627 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:58.562602    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g"
Sep 18 20:13:58.586799 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:58.586750    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g" podStartSLOduration=1.586734774 podStartE2EDuration="1.586734774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:13:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:13:57.949982209 +0000 UTC m=+1022.903723378" watchObservedRunningTime="2026-09-18 20:13:58.586734774 +0000 UTC m=+1023.540475941"
Sep 18 20:13:58.933144 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:58.933111    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g"
Sep 18 20:13:58.934000 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:13:58.933977    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g"
Sep 18 20:14:01.786152 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:01.786091    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f524d4ec-openshift-default-7cf9f69b46-jpkf2"
Sep 18 20:14:01.929545 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:01.929506    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c2246a4-6b66-49ff-a268-b34347959d01-istiod-ca-cert\") pod \"1c2246a4-6b66-49ff-a268-b34347959d01\" (UID: \"1c2246a4-6b66-49ff-a268-b34347959d01\") "
Sep 18 20:14:01.929735 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:01.929563    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kkgsm\" (UniqueName: \"kubernetes.io/projected/1c2246a4-6b66-49ff-a268-b34347959d01-kube-api-access-kkgsm\") pod \"1c2246a4-6b66-49ff-a268-b34347959d01\" (UID: \"1c2246a4-6b66-49ff-a268-b34347959d01\") "
Sep 18 20:14:01.929735 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:01.929595    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c2246a4-6b66-49ff-a268-b34347959d01-istio-podinfo\") pod \"1c2246a4-6b66-49ff-a268-b34347959d01\" (UID: \"1c2246a4-6b66-49ff-a268-b34347959d01\") "
Sep 18 20:14:01.929735 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:01.929634    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c2246a4-6b66-49ff-a268-b34347959d01-credential-socket\") pod \"1c2246a4-6b66-49ff-a268-b34347959d01\" (UID: \"1c2246a4-6b66-49ff-a268-b34347959d01\") "
Sep 18 20:14:01.929735 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:01.929677    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c2246a4-6b66-49ff-a268-b34347959d01-istio-data\") pod \"1c2246a4-6b66-49ff-a268-b34347959d01\" (UID: \"1c2246a4-6b66-49ff-a268-b34347959d01\") "
Sep 18 20:14:01.929735 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:01.929710    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c2246a4-6b66-49ff-a268-b34347959d01-workload-certs\") pod \"1c2246a4-6b66-49ff-a268-b34347959d01\" (UID: \"1c2246a4-6b66-49ff-a268-b34347959d01\") "
Sep 18 20:14:01.930013 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:01.929749    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c2246a4-6b66-49ff-a268-b34347959d01-workload-socket\") pod \"1c2246a4-6b66-49ff-a268-b34347959d01\" (UID: \"1c2246a4-6b66-49ff-a268-b34347959d01\") "
Sep 18 20:14:01.930013 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:01.929837    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c2246a4-6b66-49ff-a268-b34347959d01-istio-token\") pod \"1c2246a4-6b66-49ff-a268-b34347959d01\" (UID: \"1c2246a4-6b66-49ff-a268-b34347959d01\") "
Sep 18 20:14:01.930013 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:01.929902    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c2246a4-6b66-49ff-a268-b34347959d01-istio-envoy\") pod \"1c2246a4-6b66-49ff-a268-b34347959d01\" (UID: \"1c2246a4-6b66-49ff-a268-b34347959d01\") "
Sep 18 20:14:01.930013 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:01.929935    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c2246a4-6b66-49ff-a268-b34347959d01-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1c2246a4-6b66-49ff-a268-b34347959d01" (UID: "1c2246a4-6b66-49ff-a268-b34347959d01"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:14:01.930013 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:01.929957    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c2246a4-6b66-49ff-a268-b34347959d01-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1c2246a4-6b66-49ff-a268-b34347959d01" (UID: "1c2246a4-6b66-49ff-a268-b34347959d01"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:14:01.930013 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:01.929967    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c2246a4-6b66-49ff-a268-b34347959d01-istio-data" (OuterVolumeSpecName: "istio-data") pod "1c2246a4-6b66-49ff-a268-b34347959d01" (UID: "1c2246a4-6b66-49ff-a268-b34347959d01"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:14:01.930330 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:01.930024    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c2246a4-6b66-49ff-a268-b34347959d01-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1c2246a4-6b66-49ff-a268-b34347959d01" (UID: "1c2246a4-6b66-49ff-a268-b34347959d01"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:14:01.930330 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:01.930106    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c2246a4-6b66-49ff-a268-b34347959d01-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1c2246a4-6b66-49ff-a268-b34347959d01" (UID: "1c2246a4-6b66-49ff-a268-b34347959d01"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:14:01.930330 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:01.930291    2549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c2246a4-6b66-49ff-a268-b34347959d01-istiod-ca-cert\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:14:01.930330 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:01.930314    2549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c2246a4-6b66-49ff-a268-b34347959d01-credential-socket\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:14:01.930330 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:01.930330    2549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c2246a4-6b66-49ff-a268-b34347959d01-istio-data\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:14:01.930652 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:01.930343    2549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c2246a4-6b66-49ff-a268-b34347959d01-workload-certs\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:14:01.930652 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:01.930356    2549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c2246a4-6b66-49ff-a268-b34347959d01-workload-socket\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:14:01.931889 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:01.931858    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1c2246a4-6b66-49ff-a268-b34347959d01-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1c2246a4-6b66-49ff-a268-b34347959d01" (UID: "1c2246a4-6b66-49ff-a268-b34347959d01"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 20:14:01.932015 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:01.931931    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c2246a4-6b66-49ff-a268-b34347959d01-kube-api-access-kkgsm" (OuterVolumeSpecName: "kube-api-access-kkgsm") pod "1c2246a4-6b66-49ff-a268-b34347959d01" (UID: "1c2246a4-6b66-49ff-a268-b34347959d01"). InnerVolumeSpecName "kube-api-access-kkgsm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:14:01.932077 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:01.932011    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c2246a4-6b66-49ff-a268-b34347959d01-istio-token" (OuterVolumeSpecName: "istio-token") pod "1c2246a4-6b66-49ff-a268-b34347959d01" (UID: "1c2246a4-6b66-49ff-a268-b34347959d01"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:14:01.932128 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:01.932083    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c2246a4-6b66-49ff-a268-b34347959d01-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1c2246a4-6b66-49ff-a268-b34347959d01" (UID: "1c2246a4-6b66-49ff-a268-b34347959d01"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:14:01.954476 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:01.954448    2549 generic.go:356] "Generic (PLEG): container finished" podID="1c2246a4-6b66-49ff-a268-b34347959d01" containerID="c70a50ccd0cf0b41540c0a3d8c417b61d2c041d648f23e9e77716ae982643592" exitCode=0
Sep 18 20:14:01.954589 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:01.954495    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f524d4ec-openshift-default-7cf9f69b46-jpkf2"
Sep 18 20:14:01.954589 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:01.954499    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f524d4ec-openshift-default-7cf9f69b46-jpkf2" event={"ID":"1c2246a4-6b66-49ff-a268-b34347959d01","Type":"ContainerDied","Data":"c70a50ccd0cf0b41540c0a3d8c417b61d2c041d648f23e9e77716ae982643592"}
Sep 18 20:14:01.954589 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:01.954560    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f524d4ec-openshift-default-7cf9f69b46-jpkf2" event={"ID":"1c2246a4-6b66-49ff-a268-b34347959d01","Type":"ContainerDied","Data":"f4c336d3d04c89b58a519b66206ca32a47b9f8ea01b59652e560e6271673e688"}
Sep 18 20:14:01.954589 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:01.954586    2549 scope.go:117] "RemoveContainer" containerID="c70a50ccd0cf0b41540c0a3d8c417b61d2c041d648f23e9e77716ae982643592"
Sep 18 20:14:01.964703 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:01.964685    2549 scope.go:117] "RemoveContainer" containerID="68bd4b1f4fbf4255bf7b7682c01be874dcbbf7f7ac84e99c22c3caeba34a5662"
Sep 18 20:14:01.972884 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:01.972867    2549 scope.go:117] "RemoveContainer" containerID="c70a50ccd0cf0b41540c0a3d8c417b61d2c041d648f23e9e77716ae982643592"
Sep 18 20:14:01.973135 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:14:01.973116    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c70a50ccd0cf0b41540c0a3d8c417b61d2c041d648f23e9e77716ae982643592\": container with ID starting with c70a50ccd0cf0b41540c0a3d8c417b61d2c041d648f23e9e77716ae982643592 not found: ID does not exist" containerID="c70a50ccd0cf0b41540c0a3d8c417b61d2c041d648f23e9e77716ae982643592"
Sep 18 20:14:01.973185 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:01.973146    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c70a50ccd0cf0b41540c0a3d8c417b61d2c041d648f23e9e77716ae982643592"} err="failed to get container status \"c70a50ccd0cf0b41540c0a3d8c417b61d2c041d648f23e9e77716ae982643592\": rpc error: code = NotFound desc = could not find container \"c70a50ccd0cf0b41540c0a3d8c417b61d2c041d648f23e9e77716ae982643592\": container with ID starting with c70a50ccd0cf0b41540c0a3d8c417b61d2c041d648f23e9e77716ae982643592 not found: ID does not exist"
Sep 18 20:14:01.973185 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:01.973165    2549 scope.go:117] "RemoveContainer" containerID="68bd4b1f4fbf4255bf7b7682c01be874dcbbf7f7ac84e99c22c3caeba34a5662"
Sep 18 20:14:01.973422 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:14:01.973405    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68bd4b1f4fbf4255bf7b7682c01be874dcbbf7f7ac84e99c22c3caeba34a5662\": container with ID starting with 68bd4b1f4fbf4255bf7b7682c01be874dcbbf7f7ac84e99c22c3caeba34a5662 not found: ID does not exist" containerID="68bd4b1f4fbf4255bf7b7682c01be874dcbbf7f7ac84e99c22c3caeba34a5662"
Sep 18 20:14:01.973486 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:01.973432    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68bd4b1f4fbf4255bf7b7682c01be874dcbbf7f7ac84e99c22c3caeba34a5662"} err="failed to get container status \"68bd4b1f4fbf4255bf7b7682c01be874dcbbf7f7ac84e99c22c3caeba34a5662\": rpc error: code = NotFound desc = could not find container \"68bd4b1f4fbf4255bf7b7682c01be874dcbbf7f7ac84e99c22c3caeba34a5662\": container with ID starting with 68bd4b1f4fbf4255bf7b7682c01be874dcbbf7f7ac84e99c22c3caeba34a5662 not found: ID does not exist"
Sep 18 20:14:01.978732 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:01.978709    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f524d4ec-openshift-default-7cf9f69b46-jpkf2"]
Sep 18 20:14:01.982569 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:01.982549    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-f524d4ec-openshift-default-7cf9f69b46-jpkf2"]
Sep 18 20:14:02.031176 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:02.031156    2549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c2246a4-6b66-49ff-a268-b34347959d01-istio-envoy\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:14:02.031176 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:02.031176    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkgsm\" (UniqueName: \"kubernetes.io/projected/1c2246a4-6b66-49ff-a268-b34347959d01-kube-api-access-kkgsm\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:14:02.031307 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:02.031186    2549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c2246a4-6b66-49ff-a268-b34347959d01-istio-podinfo\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:14:02.031307 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:02.031196    2549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c2246a4-6b66-49ff-a268-b34347959d01-istio-token\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:14:03.523123 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:03.523090    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c2246a4-6b66-49ff-a268-b34347959d01" path="/var/lib/kubelet/pods/1c2246a4-6b66-49ff-a268-b34347959d01/volumes"
Sep 18 20:14:03.523493 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:03.523372    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:14:22.356795 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:22.356754    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:14:23.098701 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:23.098671    2549 generic.go:356] "Generic (PLEG): container finished" podID="cf0d422b-5ebd-4b33-82cd-59b34033ed77" containerID="23b8c06a5e9bcc967066fa0d129874542639d24a99543680bdaade932b0f43b4" exitCode=0
Sep 18 20:14:23.098871 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:23.098738    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g" event={"ID":"cf0d422b-5ebd-4b33-82cd-59b34033ed77","Type":"ContainerDied","Data":"23b8c06a5e9bcc967066fa0d129874542639d24a99543680bdaade932b0f43b4"}
Sep 18 20:14:23.099153 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:23.099137    2549 scope.go:117] "RemoveContainer" containerID="23b8c06a5e9bcc967066fa0d129874542639d24a99543680bdaade932b0f43b4"
Sep 18 20:14:23.100225 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:23.100200    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:14:23.100323 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:23.100280    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:14:23.100409 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:23.100327    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:14:24.106501 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:24.106461    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g" event={"ID":"cf0d422b-5ebd-4b33-82cd-59b34033ed77","Type":"ContainerStarted","Data":"8771c116bb8f01cdab714df594bcbe544682365819471bc90db1026e7620a67f"}
Sep 18 20:14:24.556798 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:24.556698    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g"
Sep 18 20:14:24.557895 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:24.557861    2549 patch_prober.go:28] interesting pod/e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Sep 18 20:14:24.558037 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:24.557920    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g" podUID="cf0d422b-5ebd-4b33-82cd-59b34033ed77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Sep 18 20:14:25.557330 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:25.557292    2549 patch_prober.go:28] interesting pod/e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Sep 18 20:14:25.557934 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:25.557904    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g" podUID="cf0d422b-5ebd-4b33-82cd-59b34033ed77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Sep 18 20:14:25.996803 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:25.996766    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d0d77c-5zcj9"]
Sep 18 20:14:25.997188 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:25.997176    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c2246a4-6b66-49ff-a268-b34347959d01" containerName="istio-proxy"
Sep 18 20:14:25.997234 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:25.997189    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c2246a4-6b66-49ff-a268-b34347959d01" containerName="istio-proxy"
Sep 18 20:14:25.997234 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:25.997217    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c2246a4-6b66-49ff-a268-b34347959d01" containerName="istio-proxy"
Sep 18 20:14:25.997234 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:25.997222    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c2246a4-6b66-49ff-a268-b34347959d01" containerName="istio-proxy"
Sep 18 20:14:25.997327 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:25.997288    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c2246a4-6b66-49ff-a268-b34347959d01" containerName="istio-proxy"
Sep 18 20:14:25.999465 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:25.999444    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d0d77c-5zcj9"
Sep 18 20:14:26.003600 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:26.003579    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bhssz\""
Sep 18 20:14:26.003737 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:26.003624    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 20:14:26.012891 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:26.012864    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d0d77c-5zcj9"]
Sep 18 20:14:26.084002 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:26.083967    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0228e63f-85ab-45a3-b90c-1873c907e4b0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-d0d77c-5zcj9\" (UID: \"0228e63f-85ab-45a3-b90c-1873c907e4b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d0d77c-5zcj9"
Sep 18 20:14:26.185343 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:26.185307    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0228e63f-85ab-45a3-b90c-1873c907e4b0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-d0d77c-5zcj9\" (UID: \"0228e63f-85ab-45a3-b90c-1873c907e4b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d0d77c-5zcj9"
Sep 18 20:14:26.185932 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:26.185913    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0228e63f-85ab-45a3-b90c-1873c907e4b0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-d0d77c-5zcj9\" (UID: \"0228e63f-85ab-45a3-b90c-1873c907e4b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d0d77c-5zcj9"
Sep 18 20:14:26.310237 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:26.310153    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d0d77c-5zcj9"
Sep 18 20:14:26.561194 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:26.560701    2549 patch_prober.go:28] interesting pod/e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Sep 18 20:14:26.561194 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:26.560762    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g" podUID="cf0d422b-5ebd-4b33-82cd-59b34033ed77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Sep 18 20:14:26.615064 ip-10-0-140-178 kubenswrapper[2549]: W0918 20:14:26.615022    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0228e63f_85ab_45a3_b90c_1873c907e4b0.slice/crio-22924c144b21ac02215c8480183c1a847c0a6a86621a75b3c3c84c98a3d792f7 WatchSource:0}: Error finding container 22924c144b21ac02215c8480183c1a847c0a6a86621a75b3c3c84c98a3d792f7: Status 404 returned error can't find the container with id 22924c144b21ac02215c8480183c1a847c0a6a86621a75b3c3c84c98a3d792f7
Sep 18 20:14:26.616561 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:26.616395    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d0d77c-5zcj9"]
Sep 18 20:14:27.125491 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:27.125453    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d0d77c-5zcj9" event={"ID":"0228e63f-85ab-45a3-b90c-1873c907e4b0","Type":"ContainerStarted","Data":"55935c5afe1d093bfff9339a6a352a8ccb9bdf4210724272d2f359e696ddf0f8"}
Sep 18 20:14:27.125696 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:27.125511    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d0d77c-5zcj9" event={"ID":"0228e63f-85ab-45a3-b90c-1873c907e4b0","Type":"ContainerStarted","Data":"22924c144b21ac02215c8480183c1a847c0a6a86621a75b3c3c84c98a3d792f7"}
Sep 18 20:14:27.162765 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:27.162703    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d0d77c-5zcj9" podStartSLOduration=2.162682019 podStartE2EDuration="2.162682019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:14:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:14:27.15647629 +0000 UTC m=+1052.110217457" watchObservedRunningTime="2026-09-18 20:14:27.162682019 +0000 UTC m=+1052.116423188"
Sep 18 20:14:27.560827 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:27.560731    2549 patch_prober.go:28] interesting pod/e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Sep 18 20:14:27.560827 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:27.560799    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g" podUID="cf0d422b-5ebd-4b33-82cd-59b34033ed77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Sep 18 20:14:28.133124 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:28.132928    2549 generic.go:356] "Generic (PLEG): container finished" podID="0228e63f-85ab-45a3-b90c-1873c907e4b0" containerID="55935c5afe1d093bfff9339a6a352a8ccb9bdf4210724272d2f359e696ddf0f8" exitCode=0
Sep 18 20:14:28.133124 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:28.133016    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d0d77c-5zcj9" event={"ID":"0228e63f-85ab-45a3-b90c-1873c907e4b0","Type":"ContainerDied","Data":"55935c5afe1d093bfff9339a6a352a8ccb9bdf4210724272d2f359e696ddf0f8"}
Sep 18 20:14:28.557692 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:28.557589    2549 patch_prober.go:28] interesting pod/e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Sep 18 20:14:28.557692 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:28.557659    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g" podUID="cf0d422b-5ebd-4b33-82cd-59b34033ed77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Sep 18 20:14:28.933476 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:28.933443    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g"
Sep 18 20:14:29.293338 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:29.293307    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d0d77c-5zcj9"
Sep 18 20:14:29.346431 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:29.346372    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fa77d0d6-vqxbc"]
Sep 18 20:14:29.347272 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:29.347246    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0228e63f-85ab-45a3-b90c-1873c907e4b0" containerName="revoke-keys"
Sep 18 20:14:29.347461 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:29.347449    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="0228e63f-85ab-45a3-b90c-1873c907e4b0" containerName="revoke-keys"
Sep 18 20:14:29.347688 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:29.347675    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="0228e63f-85ab-45a3-b90c-1873c907e4b0" containerName="revoke-keys"
Sep 18 20:14:29.347777 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:29.347766    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c2246a4-6b66-49ff-a268-b34347959d01" containerName="istio-proxy"
Sep 18 20:14:29.349921 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:29.349902    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fa77d0d6-vqxbc"
Sep 18 20:14:29.372280 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:29.372254    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fa77d0d6-vqxbc"]
Sep 18 20:14:29.420248 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:29.420216    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0228e63f-85ab-45a3-b90c-1873c907e4b0-maas-api-service-ca\") pod \"0228e63f-85ab-45a3-b90c-1873c907e4b0\" (UID: \"0228e63f-85ab-45a3-b90c-1873c907e4b0\") "
Sep 18 20:14:29.420542 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:29.420519    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f8cfc64-b80f-4cde-917e-b90f027eedbe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-fa77d0d6-vqxbc\" (UID: \"1f8cfc64-b80f-4cde-917e-b90f027eedbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fa77d0d6-vqxbc"
Sep 18 20:14:29.420617 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:29.420549    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0228e63f-85ab-45a3-b90c-1873c907e4b0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0228e63f-85ab-45a3-b90c-1873c907e4b0" (UID: "0228e63f-85ab-45a3-b90c-1873c907e4b0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:14:29.420715 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:29.420701    2549 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0228e63f-85ab-45a3-b90c-1873c907e4b0-maas-api-service-ca\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:14:29.522117 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:29.522028    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f8cfc64-b80f-4cde-917e-b90f027eedbe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-fa77d0d6-vqxbc\" (UID: \"1f8cfc64-b80f-4cde-917e-b90f027eedbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fa77d0d6-vqxbc"
Sep 18 20:14:29.522637 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:29.522617    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f8cfc64-b80f-4cde-917e-b90f027eedbe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-fa77d0d6-vqxbc\" (UID: \"1f8cfc64-b80f-4cde-917e-b90f027eedbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fa77d0d6-vqxbc"
Sep 18 20:14:29.561175 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:29.561149    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g"
Sep 18 20:14:29.562126 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:29.562107    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g"
Sep 18 20:14:29.677613 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:29.677585    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fa77d0d6-vqxbc"
Sep 18 20:14:29.960731 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:29.960701    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fa77d0d6-vqxbc"]
Sep 18 20:14:29.962329 ip-10-0-140-178 kubenswrapper[2549]: W0918 20:14:29.962299    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f8cfc64_b80f_4cde_917e_b90f027eedbe.slice/crio-46dd0bba7a5421c6c6fe292cf69d278c031d12485ef8d352b2686a86aae4c9eb WatchSource:0}: Error finding container 46dd0bba7a5421c6c6fe292cf69d278c031d12485ef8d352b2686a86aae4c9eb: Status 404 returned error can't find the container with id 46dd0bba7a5421c6c6fe292cf69d278c031d12485ef8d352b2686a86aae4c9eb
Sep 18 20:14:30.149057 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:30.149029    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d0d77c-5zcj9"
Sep 18 20:14:30.149643 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:30.149611    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d0d77c-5zcj9" event={"ID":"0228e63f-85ab-45a3-b90c-1873c907e4b0","Type":"ContainerDied","Data":"22924c144b21ac02215c8480183c1a847c0a6a86621a75b3c3c84c98a3d792f7"}
Sep 18 20:14:30.149845 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:30.149831    2549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="22924c144b21ac02215c8480183c1a847c0a6a86621a75b3c3c84c98a3d792f7"
Sep 18 20:14:30.151748 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:30.151721    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fa77d0d6-vqxbc" event={"ID":"1f8cfc64-b80f-4cde-917e-b90f027eedbe","Type":"ContainerStarted","Data":"46dd0bba7a5421c6c6fe292cf69d278c031d12485ef8d352b2686a86aae4c9eb"}
Sep 18 20:14:31.160688 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:31.160573    2549 generic.go:356] "Generic (PLEG): container finished" podID="1f8cfc64-b80f-4cde-917e-b90f027eedbe" containerID="bfc8cdec651459455f5c62880403f0ec6e1570072514ae2d7cf4a468e398fb5a" exitCode=0
Sep 18 20:14:31.160688 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:31.160662    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fa77d0d6-vqxbc" event={"ID":"1f8cfc64-b80f-4cde-917e-b90f027eedbe","Type":"ContainerDied","Data":"bfc8cdec651459455f5c62880403f0ec6e1570072514ae2d7cf4a468e398fb5a"}
Sep 18 20:14:32.313239 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:32.313213    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fa77d0d6-vqxbc"
Sep 18 20:14:32.453556 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:32.453473    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f8cfc64-b80f-4cde-917e-b90f027eedbe-maas-api-service-ca\") pod \"1f8cfc64-b80f-4cde-917e-b90f027eedbe\" (UID: \"1f8cfc64-b80f-4cde-917e-b90f027eedbe\") "
Sep 18 20:14:32.453825 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:32.453800    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f8cfc64-b80f-4cde-917e-b90f027eedbe-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1f8cfc64-b80f-4cde-917e-b90f027eedbe" (UID: "1f8cfc64-b80f-4cde-917e-b90f027eedbe"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:14:32.554695 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:32.554659    2549 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f8cfc64-b80f-4cde-917e-b90f027eedbe-maas-api-service-ca\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:14:33.173514 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:33.173485    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fa77d0d6-vqxbc"
Sep 18 20:14:33.173682 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:33.173524    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fa77d0d6-vqxbc" event={"ID":"1f8cfc64-b80f-4cde-917e-b90f027eedbe","Type":"ContainerDied","Data":"46dd0bba7a5421c6c6fe292cf69d278c031d12485ef8d352b2686a86aae4c9eb"}
Sep 18 20:14:33.173682 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:33.173568    2549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="46dd0bba7a5421c6c6fe292cf69d278c031d12485ef8d352b2686a86aae4c9eb"
Sep 18 20:14:37.972510 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:37.972458    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d0d77c-5zcj9"]
Sep 18 20:14:37.975619 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:37.975575    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d0d77c-5zcj9"]
Sep 18 20:14:39.522479 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:39.522438    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0228e63f-85ab-45a3-b90c-1873c907e4b0" path="/var/lib/kubelet/pods/0228e63f-85ab-45a3-b90c-1873c907e4b0/volumes"
Sep 18 20:14:42.931334 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:42.931297    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:14:44.226501 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:44.226463    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fa77d0d6-vqxbc"]
Sep 18 20:14:44.232721 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:44.232694    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fa77d0d6-vqxbc"]
Sep 18 20:14:44.409343 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:44.409313    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g"]
Sep 18 20:14:44.409576 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:44.409555    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g" podUID="cf0d422b-5ebd-4b33-82cd-59b34033ed77" containerName="istio-proxy" containerID="cri-o://8771c116bb8f01cdab714df594bcbe544682365819471bc90db1026e7620a67f" gracePeriod=30
Sep 18 20:14:44.563558 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:44.563481    2549 patch_prober.go:28] interesting pod/e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 18 20:14:44.563558 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:44.563536    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g" podUID="cf0d422b-5ebd-4b33-82cd-59b34033ed77" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:14:45.524013 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:45.523984    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f8cfc64-b80f-4cde-917e-b90f027eedbe" path="/var/lib/kubelet/pods/1f8cfc64-b80f-4cde-917e-b90f027eedbe/volumes"
Sep 18 20:14:48.977130 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:48.977097    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-4b633d-bxp6m"]
Sep 18 20:14:48.977796 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:48.977776    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f8cfc64-b80f-4cde-917e-b90f027eedbe" containerName="revoke-keys"
Sep 18 20:14:48.977846 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:48.977800    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f8cfc64-b80f-4cde-917e-b90f027eedbe" containerName="revoke-keys"
Sep 18 20:14:48.977909 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:48.977898    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f8cfc64-b80f-4cde-917e-b90f027eedbe" containerName="revoke-keys"
Sep 18 20:14:48.980700 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:48.980683    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-4b633d-bxp6m"
Sep 18 20:14:48.983842 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:48.983819    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 20:14:48.983947 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:48.983892    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bhssz\""
Sep 18 20:14:48.990248 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:48.990225    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-4b633d-bxp6m"]
Sep 18 20:14:49.018990 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:49.018969    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24856e6c-c20d-471e-85cb-9b7a110e5a8a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-4b633d-bxp6m\" (UID: \"24856e6c-c20d-471e-85cb-9b7a110e5a8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-4b633d-bxp6m"
Sep 18 20:14:49.119639 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:49.119610    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24856e6c-c20d-471e-85cb-9b7a110e5a8a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-4b633d-bxp6m\" (UID: \"24856e6c-c20d-471e-85cb-9b7a110e5a8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-4b633d-bxp6m"
Sep 18 20:14:49.120182 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:49.120164    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24856e6c-c20d-471e-85cb-9b7a110e5a8a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-4b633d-bxp6m\" (UID: \"24856e6c-c20d-471e-85cb-9b7a110e5a8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-4b633d-bxp6m"
Sep 18 20:14:49.292591 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:49.292514    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-4b633d-bxp6m"
Sep 18 20:14:49.564776 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:49.564746    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-4b633d-bxp6m"]
Sep 18 20:14:49.566411 ip-10-0-140-178 kubenswrapper[2549]: W0918 20:14:49.566366    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24856e6c_c20d_471e_85cb_9b7a110e5a8a.slice/crio-955129b982de46875024e22c51f68db192390147510291decd251a7792836e11 WatchSource:0}: Error finding container 955129b982de46875024e22c51f68db192390147510291decd251a7792836e11: Status 404 returned error can't find the container with id 955129b982de46875024e22c51f68db192390147510291decd251a7792836e11
Sep 18 20:14:49.645249 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:49.645226    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g"
Sep 18 20:14:49.724725 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:49.724693    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf0d422b-5ebd-4b33-82cd-59b34033ed77-istio-data\") pod \"cf0d422b-5ebd-4b33-82cd-59b34033ed77\" (UID: \"cf0d422b-5ebd-4b33-82cd-59b34033ed77\") "
Sep 18 20:14:49.724903 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:49.724739    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf0d422b-5ebd-4b33-82cd-59b34033ed77-istio-podinfo\") pod \"cf0d422b-5ebd-4b33-82cd-59b34033ed77\" (UID: \"cf0d422b-5ebd-4b33-82cd-59b34033ed77\") "
Sep 18 20:14:49.724903 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:49.724763    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf0d422b-5ebd-4b33-82cd-59b34033ed77-workload-socket\") pod \"cf0d422b-5ebd-4b33-82cd-59b34033ed77\" (UID: \"cf0d422b-5ebd-4b33-82cd-59b34033ed77\") "
Sep 18 20:14:49.724903 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:49.724814    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sv26p\" (UniqueName: \"kubernetes.io/projected/cf0d422b-5ebd-4b33-82cd-59b34033ed77-kube-api-access-sv26p\") pod \"cf0d422b-5ebd-4b33-82cd-59b34033ed77\" (UID: \"cf0d422b-5ebd-4b33-82cd-59b34033ed77\") "
Sep 18 20:14:49.724903 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:49.724831    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf0d422b-5ebd-4b33-82cd-59b34033ed77-istio-envoy\") pod \"cf0d422b-5ebd-4b33-82cd-59b34033ed77\" (UID: \"cf0d422b-5ebd-4b33-82cd-59b34033ed77\") "
Sep 18 20:14:49.724903 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:49.724856    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf0d422b-5ebd-4b33-82cd-59b34033ed77-istiod-ca-cert\") pod \"cf0d422b-5ebd-4b33-82cd-59b34033ed77\" (UID: \"cf0d422b-5ebd-4b33-82cd-59b34033ed77\") "
Sep 18 20:14:49.724903 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:49.724881    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf0d422b-5ebd-4b33-82cd-59b34033ed77-istio-token\") pod \"cf0d422b-5ebd-4b33-82cd-59b34033ed77\" (UID: \"cf0d422b-5ebd-4b33-82cd-59b34033ed77\") "
Sep 18 20:14:49.724903 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:49.724903    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf0d422b-5ebd-4b33-82cd-59b34033ed77-credential-socket\") pod \"cf0d422b-5ebd-4b33-82cd-59b34033ed77\" (UID: \"cf0d422b-5ebd-4b33-82cd-59b34033ed77\") "
Sep 18 20:14:49.725280 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:49.724965    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf0d422b-5ebd-4b33-82cd-59b34033ed77-istio-data" (OuterVolumeSpecName: "istio-data") pod "cf0d422b-5ebd-4b33-82cd-59b34033ed77" (UID: "cf0d422b-5ebd-4b33-82cd-59b34033ed77"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:14:49.725280 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:49.724985    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf0d422b-5ebd-4b33-82cd-59b34033ed77-workload-certs\") pod \"cf0d422b-5ebd-4b33-82cd-59b34033ed77\" (UID: \"cf0d422b-5ebd-4b33-82cd-59b34033ed77\") "
Sep 18 20:14:49.725280 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:49.725216    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf0d422b-5ebd-4b33-82cd-59b34033ed77-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cf0d422b-5ebd-4b33-82cd-59b34033ed77" (UID: "cf0d422b-5ebd-4b33-82cd-59b34033ed77"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:14:49.725280 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:49.725231    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf0d422b-5ebd-4b33-82cd-59b34033ed77-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cf0d422b-5ebd-4b33-82cd-59b34033ed77" (UID: "cf0d422b-5ebd-4b33-82cd-59b34033ed77"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:14:49.725533 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:49.725375    2549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf0d422b-5ebd-4b33-82cd-59b34033ed77-istio-data\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:14:49.725533 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:49.725412    2549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf0d422b-5ebd-4b33-82cd-59b34033ed77-workload-socket\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:14:49.725533 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:49.725446    2549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf0d422b-5ebd-4b33-82cd-59b34033ed77-credential-socket\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:14:49.725533 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:49.725451    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf0d422b-5ebd-4b33-82cd-59b34033ed77-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cf0d422b-5ebd-4b33-82cd-59b34033ed77" (UID: "cf0d422b-5ebd-4b33-82cd-59b34033ed77"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:14:49.725533 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:49.725510    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf0d422b-5ebd-4b33-82cd-59b34033ed77-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cf0d422b-5ebd-4b33-82cd-59b34033ed77" (UID: "cf0d422b-5ebd-4b33-82cd-59b34033ed77"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:14:49.727043 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:49.727017    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cf0d422b-5ebd-4b33-82cd-59b34033ed77-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cf0d422b-5ebd-4b33-82cd-59b34033ed77" (UID: "cf0d422b-5ebd-4b33-82cd-59b34033ed77"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 20:14:49.727217 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:49.727202    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf0d422b-5ebd-4b33-82cd-59b34033ed77-kube-api-access-sv26p" (OuterVolumeSpecName: "kube-api-access-sv26p") pod "cf0d422b-5ebd-4b33-82cd-59b34033ed77" (UID: "cf0d422b-5ebd-4b33-82cd-59b34033ed77"). InnerVolumeSpecName "kube-api-access-sv26p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:14:49.727436 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:49.727409    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf0d422b-5ebd-4b33-82cd-59b34033ed77-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cf0d422b-5ebd-4b33-82cd-59b34033ed77" (UID: "cf0d422b-5ebd-4b33-82cd-59b34033ed77"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:14:49.727436 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:49.727417    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf0d422b-5ebd-4b33-82cd-59b34033ed77-istio-token" (OuterVolumeSpecName: "istio-token") pod "cf0d422b-5ebd-4b33-82cd-59b34033ed77" (UID: "cf0d422b-5ebd-4b33-82cd-59b34033ed77"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:14:49.826681 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:49.826655    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sv26p\" (UniqueName: \"kubernetes.io/projected/cf0d422b-5ebd-4b33-82cd-59b34033ed77-kube-api-access-sv26p\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:14:49.826845 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:49.826684    2549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf0d422b-5ebd-4b33-82cd-59b34033ed77-istio-envoy\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:14:49.826845 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:49.826693    2549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf0d422b-5ebd-4b33-82cd-59b34033ed77-istiod-ca-cert\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:14:49.826845 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:49.826701    2549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf0d422b-5ebd-4b33-82cd-59b34033ed77-istio-token\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:14:49.826845 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:49.826708    2549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf0d422b-5ebd-4b33-82cd-59b34033ed77-workload-certs\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:14:49.826845 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:49.826716    2549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf0d422b-5ebd-4b33-82cd-59b34033ed77-istio-podinfo\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:14:50.288277 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:50.288244    2549 generic.go:356] "Generic (PLEG): container finished" podID="cf0d422b-5ebd-4b33-82cd-59b34033ed77" containerID="8771c116bb8f01cdab714df594bcbe544682365819471bc90db1026e7620a67f" exitCode=0
Sep 18 20:14:50.288761 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:50.288294    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g"
Sep 18 20:14:50.288761 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:50.288296    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g" event={"ID":"cf0d422b-5ebd-4b33-82cd-59b34033ed77","Type":"ContainerDied","Data":"8771c116bb8f01cdab714df594bcbe544682365819471bc90db1026e7620a67f"}
Sep 18 20:14:50.288761 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:50.288361    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g" event={"ID":"cf0d422b-5ebd-4b33-82cd-59b34033ed77","Type":"ContainerDied","Data":"5be64867f193701da8dc82cba13fa30287ee1968393745591de913d30142d952"}
Sep 18 20:14:50.288761 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:50.288419    2549 scope.go:117] "RemoveContainer" containerID="8771c116bb8f01cdab714df594bcbe544682365819471bc90db1026e7620a67f"
Sep 18 20:14:50.290131 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:50.290107    2549 generic.go:356] "Generic (PLEG): container finished" podID="24856e6c-c20d-471e-85cb-9b7a110e5a8a" containerID="7f0d32084081d5cf007495b2dee0cf5872c59d36bc3b62343022b04af604a75a" exitCode=0
Sep 18 20:14:50.290285 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:50.290134    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-4b633d-bxp6m" event={"ID":"24856e6c-c20d-471e-85cb-9b7a110e5a8a","Type":"ContainerDied","Data":"7f0d32084081d5cf007495b2dee0cf5872c59d36bc3b62343022b04af604a75a"}
Sep 18 20:14:50.290285 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:50.290175    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-4b633d-bxp6m" event={"ID":"24856e6c-c20d-471e-85cb-9b7a110e5a8a","Type":"ContainerStarted","Data":"955129b982de46875024e22c51f68db192390147510291decd251a7792836e11"}
Sep 18 20:14:50.298290 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:50.298104    2549 scope.go:117] "RemoveContainer" containerID="23b8c06a5e9bcc967066fa0d129874542639d24a99543680bdaade932b0f43b4"
Sep 18 20:14:50.317471 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:50.317445    2549 scope.go:117] "RemoveContainer" containerID="8771c116bb8f01cdab714df594bcbe544682365819471bc90db1026e7620a67f"
Sep 18 20:14:50.317738 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:14:50.317716    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8771c116bb8f01cdab714df594bcbe544682365819471bc90db1026e7620a67f\": container with ID starting with 8771c116bb8f01cdab714df594bcbe544682365819471bc90db1026e7620a67f not found: ID does not exist" containerID="8771c116bb8f01cdab714df594bcbe544682365819471bc90db1026e7620a67f"
Sep 18 20:14:50.317808 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:50.317748    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8771c116bb8f01cdab714df594bcbe544682365819471bc90db1026e7620a67f"} err="failed to get container status \"8771c116bb8f01cdab714df594bcbe544682365819471bc90db1026e7620a67f\": rpc error: code = NotFound desc = could not find container \"8771c116bb8f01cdab714df594bcbe544682365819471bc90db1026e7620a67f\": container with ID starting with 8771c116bb8f01cdab714df594bcbe544682365819471bc90db1026e7620a67f not found: ID does not exist"
Sep 18 20:14:50.317808 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:50.317764    2549 scope.go:117] "RemoveContainer" containerID="23b8c06a5e9bcc967066fa0d129874542639d24a99543680bdaade932b0f43b4"
Sep 18 20:14:50.318036 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:14:50.318018    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23b8c06a5e9bcc967066fa0d129874542639d24a99543680bdaade932b0f43b4\": container with ID starting with 23b8c06a5e9bcc967066fa0d129874542639d24a99543680bdaade932b0f43b4 not found: ID does not exist" containerID="23b8c06a5e9bcc967066fa0d129874542639d24a99543680bdaade932b0f43b4"
Sep 18 20:14:50.318081 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:50.318043    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23b8c06a5e9bcc967066fa0d129874542639d24a99543680bdaade932b0f43b4"} err="failed to get container status \"23b8c06a5e9bcc967066fa0d129874542639d24a99543680bdaade932b0f43b4\": rpc error: code = NotFound desc = could not find container \"23b8c06a5e9bcc967066fa0d129874542639d24a99543680bdaade932b0f43b4\": container with ID starting with 23b8c06a5e9bcc967066fa0d129874542639d24a99543680bdaade932b0f43b4 not found: ID does not exist"
Sep 18 20:14:50.351507 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:50.351480    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g"]
Sep 18 20:14:50.355197 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:50.355178    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-fa77d0d6-openshift-default-749c9d66cb-bdr8g"]
Sep 18 20:14:51.437135 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:51.437107    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-4b633d-bxp6m"
Sep 18 20:14:51.522908 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:51.522878    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf0d422b-5ebd-4b33-82cd-59b34033ed77" path="/var/lib/kubelet/pods/cf0d422b-5ebd-4b33-82cd-59b34033ed77/volumes"
Sep 18 20:14:51.543632 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:51.543610    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24856e6c-c20d-471e-85cb-9b7a110e5a8a-maas-api-service-ca\") pod \"24856e6c-c20d-471e-85cb-9b7a110e5a8a\" (UID: \"24856e6c-c20d-471e-85cb-9b7a110e5a8a\") "
Sep 18 20:14:51.543923 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:51.543901    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24856e6c-c20d-471e-85cb-9b7a110e5a8a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "24856e6c-c20d-471e-85cb-9b7a110e5a8a" (UID: "24856e6c-c20d-471e-85cb-9b7a110e5a8a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:14:51.644868 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:51.644844    2549 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24856e6c-c20d-471e-85cb-9b7a110e5a8a-maas-api-service-ca\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:14:52.311486 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:52.311446    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-4b633d-bxp6m" event={"ID":"24856e6c-c20d-471e-85cb-9b7a110e5a8a","Type":"ContainerDied","Data":"955129b982de46875024e22c51f68db192390147510291decd251a7792836e11"}
Sep 18 20:14:52.311655 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:52.311502    2549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="955129b982de46875024e22c51f68db192390147510291decd251a7792836e11"
Sep 18 20:14:52.311655 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:52.311508    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-4b633d-bxp6m"
Sep 18 20:14:53.850431 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:14:53.850376    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-4b633d/simulator-w3-4b633d-kserve-5f479956c4-hvpjd"]
Sep 18 20:15:00.018349 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:00.018307    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-4b633d-bxp6m"]
Sep 18 20:15:00.031607 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:00.031578    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-4b633d-bxp6m"]
Sep 18 20:15:00.176447 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:00.176412    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3a8052-29829375-62q2m"]
Sep 18 20:15:00.176883 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:00.176867    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf0d422b-5ebd-4b33-82cd-59b34033ed77" containerName="istio-proxy"
Sep 18 20:15:00.176933 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:00.176886    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf0d422b-5ebd-4b33-82cd-59b34033ed77" containerName="istio-proxy"
Sep 18 20:15:00.176933 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:00.176906    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24856e6c-c20d-471e-85cb-9b7a110e5a8a" containerName="revoke-keys"
Sep 18 20:15:00.176933 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:00.176912    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="24856e6c-c20d-471e-85cb-9b7a110e5a8a" containerName="revoke-keys"
Sep 18 20:15:00.177044 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:00.176995    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf0d422b-5ebd-4b33-82cd-59b34033ed77" containerName="istio-proxy"
Sep 18 20:15:00.177044 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:00.177003    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="24856e6c-c20d-471e-85cb-9b7a110e5a8a" containerName="revoke-keys"
Sep 18 20:15:00.181322 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:00.181302    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3a8052-29829375-62q2m"
Sep 18 20:15:00.186725 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:00.186704    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bhssz\""
Sep 18 20:15:00.191223 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:00.191203    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3a8052-29829375-62q2m"
Sep 18 20:15:00.193658 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:00.193627    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29829375-4wmm9"]
Sep 18 20:15:00.194130 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:00.194115    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf0d422b-5ebd-4b33-82cd-59b34033ed77" containerName="istio-proxy"
Sep 18 20:15:00.194172 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:00.194132    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf0d422b-5ebd-4b33-82cd-59b34033ed77" containerName="istio-proxy"
Sep 18 20:15:00.194218 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:00.194208    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf0d422b-5ebd-4b33-82cd-59b34033ed77" containerName="istio-proxy"
Sep 18 20:15:00.197273 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:00.197254    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829375-4wmm9"
Sep 18 20:15:00.205964 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:00.204285    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3a8052-29829375-62q2m"]
Sep 18 20:15:00.207914 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:00.207886    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29829375-4wmm9"]
Sep 18 20:15:00.234655 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:00.234631    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829375-4wmm9"
Sep 18 20:15:00.491192 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:00.491167    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3a8052-29829375-62q2m"]
Sep 18 20:15:00.492322 ip-10-0-140-178 kubenswrapper[2549]: W0918 20:15:00.492291    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77597449_e2a2_4288_a23d_3610b1ac4595.slice/crio-10295e993074b6a8d53d52ff9a0473d7586ed352820a58d933846d2ca8469cd5 WatchSource:0}: Error finding container 10295e993074b6a8d53d52ff9a0473d7586ed352820a58d933846d2ca8469cd5: Status 404 returned error can't find the container with id 10295e993074b6a8d53d52ff9a0473d7586ed352820a58d933846d2ca8469cd5
Sep 18 20:15:00.552093 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:00.552066    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29829375-4wmm9"]
Sep 18 20:15:00.553649 ip-10-0-140-178 kubenswrapper[2549]: W0918 20:15:00.553624    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21ca971a_459c_49d8_a299_85be11490dc3.slice/crio-228fb7b40052fe4c254a5787e43a80a1f619ea391d94d5e287955cd1d71a76d3 WatchSource:0}: Error finding container 228fb7b40052fe4c254a5787e43a80a1f619ea391d94d5e287955cd1d71a76d3: Status 404 returned error can't find the container with id 228fb7b40052fe4c254a5787e43a80a1f619ea391d94d5e287955cd1d71a76d3
Sep 18 20:15:01.237242 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:01.237167    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:15:01.368933 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:01.368892    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3a8052-29829375-62q2m" event={"ID":"77597449-e2a2-4288-a23d-3610b1ac4595","Type":"ContainerStarted","Data":"bc4b5659567b5458a457c0200342c3abf07ca2cf691d5533901aaf3726d96f7d"}
Sep 18 20:15:01.369108 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:01.368953    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3a8052-29829375-62q2m" event={"ID":"77597449-e2a2-4288-a23d-3610b1ac4595","Type":"ContainerStarted","Data":"10295e993074b6a8d53d52ff9a0473d7586ed352820a58d933846d2ca8469cd5"}
Sep 18 20:15:01.370472 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:01.370448    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829375-4wmm9" event={"ID":"21ca971a-459c-49d8-a299-85be11490dc3","Type":"ContainerStarted","Data":"8a61d854af9c9749153bb8afc9e10284a5bb72a77fa0946860d8c224c7bf0a5a"}
Sep 18 20:15:01.370603 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:01.370488    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829375-4wmm9" event={"ID":"21ca971a-459c-49d8-a299-85be11490dc3","Type":"ContainerStarted","Data":"228fb7b40052fe4c254a5787e43a80a1f619ea391d94d5e287955cd1d71a76d3"}
Sep 18 20:15:01.393795 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:01.393753    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3a8052-29829375-62q2m" podStartSLOduration=1.393739021 podStartE2EDuration="1.393739021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:15:01.392338071 +0000 UTC m=+1086.346079239" watchObservedRunningTime="2026-09-18 20:15:01.393739021 +0000 UTC m=+1086.347480188"
Sep 18 20:15:01.435128 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:01.435086    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829375-4wmm9" podStartSLOduration=1.435070974 podStartE2EDuration="1.435070974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:15:01.429669784 +0000 UTC m=+1086.383410951" watchObservedRunningTime="2026-09-18 20:15:01.435070974 +0000 UTC m=+1086.388812141"
Sep 18 20:15:01.521579 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:01.521512    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24856e6c-c20d-471e-85cb-9b7a110e5a8a" path="/var/lib/kubelet/pods/24856e6c-c20d-471e-85cb-9b7a110e5a8a/volumes"
Sep 18 20:15:02.379172 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:02.379133    2549 generic.go:356] "Generic (PLEG): container finished" podID="77597449-e2a2-4288-a23d-3610b1ac4595" containerID="bc4b5659567b5458a457c0200342c3abf07ca2cf691d5533901aaf3726d96f7d" exitCode=0
Sep 18 20:15:02.379590 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:02.379207    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3a8052-29829375-62q2m" event={"ID":"77597449-e2a2-4288-a23d-3610b1ac4595","Type":"ContainerDied","Data":"bc4b5659567b5458a457c0200342c3abf07ca2cf691d5533901aaf3726d96f7d"}
Sep 18 20:15:02.380987 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:02.380916    2549 generic.go:356] "Generic (PLEG): container finished" podID="21ca971a-459c-49d8-a299-85be11490dc3" containerID="8a61d854af9c9749153bb8afc9e10284a5bb72a77fa0946860d8c224c7bf0a5a" exitCode=0
Sep 18 20:15:02.381127 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:02.380995    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829375-4wmm9" event={"ID":"21ca971a-459c-49d8-a299-85be11490dc3","Type":"ContainerDied","Data":"8a61d854af9c9749153bb8afc9e10284a5bb72a77fa0946860d8c224c7bf0a5a"}
Sep 18 20:15:03.544808 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:03.544782    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829375-4wmm9"
Sep 18 20:15:03.548083 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:03.548060    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3a8052-29829375-62q2m"
Sep 18 20:15:04.394737 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:04.394709    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3a8052-29829375-62q2m"
Sep 18 20:15:04.394896 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:04.394745    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3a8052-29829375-62q2m" event={"ID":"77597449-e2a2-4288-a23d-3610b1ac4595","Type":"ContainerDied","Data":"10295e993074b6a8d53d52ff9a0473d7586ed352820a58d933846d2ca8469cd5"}
Sep 18 20:15:04.394896 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:04.394804    2549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10295e993074b6a8d53d52ff9a0473d7586ed352820a58d933846d2ca8469cd5"
Sep 18 20:15:04.396112 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:04.396087    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829375-4wmm9"
Sep 18 20:15:04.396112 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:04.396098    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829375-4wmm9" event={"ID":"21ca971a-459c-49d8-a299-85be11490dc3","Type":"ContainerDied","Data":"228fb7b40052fe4c254a5787e43a80a1f619ea391d94d5e287955cd1d71a76d3"}
Sep 18 20:15:04.396269 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:04.396137    2549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="228fb7b40052fe4c254a5787e43a80a1f619ea391d94d5e287955cd1d71a76d3"
Sep 18 20:15:08.688569 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:08.688538    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48"]
Sep 18 20:15:08.688965 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:08.688745    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48" podUID="e773b934-ce8f-43ce-899c-a2c425da0c33" containerName="istio-proxy" containerID="cri-o://3bedc9afbef4d4d01605b9660cc4639b3e70f994b85f0df2e97591fb2b004d29" gracePeriod=30
Sep 18 20:15:08.870715 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:08.870675    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-4b633d/simulator-w3-4b633d-kserve-5f479956c4-hvpjd" podUID="35274823-b198-4e3f-a8b8-d9ef1661e1b9" containerName="main" containerID="cri-o://d5d1bf40e890c0a5f47f57ee66301166664d04bbdc3ad3fe90c3d8ce533f0878" gracePeriod=45
Sep 18 20:15:09.125344 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:09.125317    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4b633d/simulator-w3-4b633d-kserve-5f479956c4-hvpjd"
Sep 18 20:15:09.211052 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:09.211021    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35274823-b198-4e3f-a8b8-d9ef1661e1b9-dshm\") pod \"35274823-b198-4e3f-a8b8-d9ef1661e1b9\" (UID: \"35274823-b198-4e3f-a8b8-d9ef1661e1b9\") "
Sep 18 20:15:09.211237 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:09.211071    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35274823-b198-4e3f-a8b8-d9ef1661e1b9-model-cache\") pod \"35274823-b198-4e3f-a8b8-d9ef1661e1b9\" (UID: \"35274823-b198-4e3f-a8b8-d9ef1661e1b9\") "
Sep 18 20:15:09.211237 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:09.211113    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35274823-b198-4e3f-a8b8-d9ef1661e1b9-home\") pod \"35274823-b198-4e3f-a8b8-d9ef1661e1b9\" (UID: \"35274823-b198-4e3f-a8b8-d9ef1661e1b9\") "
Sep 18 20:15:09.211237 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:09.211191    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c8cj8\" (UniqueName: \"kubernetes.io/projected/35274823-b198-4e3f-a8b8-d9ef1661e1b9-kube-api-access-c8cj8\") pod \"35274823-b198-4e3f-a8b8-d9ef1661e1b9\" (UID: \"35274823-b198-4e3f-a8b8-d9ef1661e1b9\") "
Sep 18 20:15:09.211237 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:09.211216    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35274823-b198-4e3f-a8b8-d9ef1661e1b9-tmp-dir\") pod \"35274823-b198-4e3f-a8b8-d9ef1661e1b9\" (UID: \"35274823-b198-4e3f-a8b8-d9ef1661e1b9\") "
Sep 18 20:15:09.211521 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:09.211253    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35274823-b198-4e3f-a8b8-d9ef1661e1b9-tls-certs\") pod \"35274823-b198-4e3f-a8b8-d9ef1661e1b9\" (UID: \"35274823-b198-4e3f-a8b8-d9ef1661e1b9\") "
Sep 18 20:15:09.211521 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:09.211404    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35274823-b198-4e3f-a8b8-d9ef1661e1b9-model-cache" (OuterVolumeSpecName: "model-cache") pod "35274823-b198-4e3f-a8b8-d9ef1661e1b9" (UID: "35274823-b198-4e3f-a8b8-d9ef1661e1b9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:15:09.211521 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:09.211431    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35274823-b198-4e3f-a8b8-d9ef1661e1b9-home" (OuterVolumeSpecName: "home") pod "35274823-b198-4e3f-a8b8-d9ef1661e1b9" (UID: "35274823-b198-4e3f-a8b8-d9ef1661e1b9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:15:09.211659 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:09.211524    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35274823-b198-4e3f-a8b8-d9ef1661e1b9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "35274823-b198-4e3f-a8b8-d9ef1661e1b9" (UID: "35274823-b198-4e3f-a8b8-d9ef1661e1b9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:15:09.211746 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:09.211720    2549 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35274823-b198-4e3f-a8b8-d9ef1661e1b9-model-cache\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:15:09.211746 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:09.211746    2549 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35274823-b198-4e3f-a8b8-d9ef1661e1b9-home\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:15:09.211932 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:09.211758    2549 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35274823-b198-4e3f-a8b8-d9ef1661e1b9-tmp-dir\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:15:09.213480 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:09.213454    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35274823-b198-4e3f-a8b8-d9ef1661e1b9-dshm" (OuterVolumeSpecName: "dshm") pod "35274823-b198-4e3f-a8b8-d9ef1661e1b9" (UID: "35274823-b198-4e3f-a8b8-d9ef1661e1b9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:15:09.213585 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:09.213455    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35274823-b198-4e3f-a8b8-d9ef1661e1b9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "35274823-b198-4e3f-a8b8-d9ef1661e1b9" (UID: "35274823-b198-4e3f-a8b8-d9ef1661e1b9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:15:09.213585 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:09.213539    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35274823-b198-4e3f-a8b8-d9ef1661e1b9-kube-api-access-c8cj8" (OuterVolumeSpecName: "kube-api-access-c8cj8") pod "35274823-b198-4e3f-a8b8-d9ef1661e1b9" (UID: "35274823-b198-4e3f-a8b8-d9ef1661e1b9"). InnerVolumeSpecName "kube-api-access-c8cj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:15:09.312453 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:09.312344    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8cj8\" (UniqueName: \"kubernetes.io/projected/35274823-b198-4e3f-a8b8-d9ef1661e1b9-kube-api-access-c8cj8\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:15:09.312453 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:09.312375    2549 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35274823-b198-4e3f-a8b8-d9ef1661e1b9-tls-certs\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:15:09.312453 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:09.312415    2549 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35274823-b198-4e3f-a8b8-d9ef1661e1b9-dshm\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:15:09.426882 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:09.426846    2549 generic.go:356] "Generic (PLEG): container finished" podID="35274823-b198-4e3f-a8b8-d9ef1661e1b9" containerID="d5d1bf40e890c0a5f47f57ee66301166664d04bbdc3ad3fe90c3d8ce533f0878" exitCode=0
Sep 18 20:15:09.427054 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:09.426887    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4b633d/simulator-w3-4b633d-kserve-5f479956c4-hvpjd"
Sep 18 20:15:09.427054 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:09.426935    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4b633d/simulator-w3-4b633d-kserve-5f479956c4-hvpjd" event={"ID":"35274823-b198-4e3f-a8b8-d9ef1661e1b9","Type":"ContainerDied","Data":"d5d1bf40e890c0a5f47f57ee66301166664d04bbdc3ad3fe90c3d8ce533f0878"}
Sep 18 20:15:09.427054 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:09.426975    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4b633d/simulator-w3-4b633d-kserve-5f479956c4-hvpjd" event={"ID":"35274823-b198-4e3f-a8b8-d9ef1661e1b9","Type":"ContainerDied","Data":"feda6500bc00145aaabd46de01d79d0d65fe1b54476ac5709cbb1e2b3a10b88c"}
Sep 18 20:15:09.427054 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:09.426996    2549 scope.go:117] "RemoveContainer" containerID="d5d1bf40e890c0a5f47f57ee66301166664d04bbdc3ad3fe90c3d8ce533f0878"
Sep 18 20:15:09.437939 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:09.437922    2549 scope.go:117] "RemoveContainer" containerID="d5d1bf40e890c0a5f47f57ee66301166664d04bbdc3ad3fe90c3d8ce533f0878"
Sep 18 20:15:09.438210 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:15:09.438193    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5d1bf40e890c0a5f47f57ee66301166664d04bbdc3ad3fe90c3d8ce533f0878\": container with ID starting with d5d1bf40e890c0a5f47f57ee66301166664d04bbdc3ad3fe90c3d8ce533f0878 not found: ID does not exist" containerID="d5d1bf40e890c0a5f47f57ee66301166664d04bbdc3ad3fe90c3d8ce533f0878"
Sep 18 20:15:09.438250 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:09.438218    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5d1bf40e890c0a5f47f57ee66301166664d04bbdc3ad3fe90c3d8ce533f0878"} err="failed to get container status \"d5d1bf40e890c0a5f47f57ee66301166664d04bbdc3ad3fe90c3d8ce533f0878\": rpc error: code = NotFound desc = could not find container \"d5d1bf40e890c0a5f47f57ee66301166664d04bbdc3ad3fe90c3d8ce533f0878\": container with ID starting with d5d1bf40e890c0a5f47f57ee66301166664d04bbdc3ad3fe90c3d8ce533f0878 not found: ID does not exist"
Sep 18 20:15:09.453107 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:09.453077    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-4b633d/simulator-w3-4b633d-kserve-5f479956c4-hvpjd"]
Sep 18 20:15:09.458847 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:09.458822    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-4b633d/simulator-w3-4b633d-kserve-5f479956c4-hvpjd"]
Sep 18 20:15:09.521219 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:09.521194    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35274823-b198-4e3f-a8b8-d9ef1661e1b9" path="/var/lib/kubelet/pods/35274823-b198-4e3f-a8b8-d9ef1661e1b9/volumes"
Sep 18 20:15:09.691028 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:09.690990    2549 patch_prober.go:28] interesting pod/e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 18 20:15:09.691402 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:09.691048    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48" podUID="e773b934-ce8f-43ce-899c-a2c425da0c33" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:15:13.724947 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:13.724918    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:15:13.946471 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:13.946442    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48"
Sep 18 20:15:14.059065 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:14.058992    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e773b934-ce8f-43ce-899c-a2c425da0c33-istio-podinfo\") pod \"e773b934-ce8f-43ce-899c-a2c425da0c33\" (UID: \"e773b934-ce8f-43ce-899c-a2c425da0c33\") "
Sep 18 20:15:14.059065 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:14.059050    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e773b934-ce8f-43ce-899c-a2c425da0c33-istiod-ca-cert\") pod \"e773b934-ce8f-43ce-899c-a2c425da0c33\" (UID: \"e773b934-ce8f-43ce-899c-a2c425da0c33\") "
Sep 18 20:15:14.059295 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:14.059078    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e773b934-ce8f-43ce-899c-a2c425da0c33-istio-envoy\") pod \"e773b934-ce8f-43ce-899c-a2c425da0c33\" (UID: \"e773b934-ce8f-43ce-899c-a2c425da0c33\") "
Sep 18 20:15:14.059295 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:14.059098    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e773b934-ce8f-43ce-899c-a2c425da0c33-credential-socket\") pod \"e773b934-ce8f-43ce-899c-a2c425da0c33\" (UID: \"e773b934-ce8f-43ce-899c-a2c425da0c33\") "
Sep 18 20:15:14.059295 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:14.059114    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e773b934-ce8f-43ce-899c-a2c425da0c33-istio-token\") pod \"e773b934-ce8f-43ce-899c-a2c425da0c33\" (UID: \"e773b934-ce8f-43ce-899c-a2c425da0c33\") "
Sep 18 20:15:14.059295 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:14.059131    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-klnpm\" (UniqueName: \"kubernetes.io/projected/e773b934-ce8f-43ce-899c-a2c425da0c33-kube-api-access-klnpm\") pod \"e773b934-ce8f-43ce-899c-a2c425da0c33\" (UID: \"e773b934-ce8f-43ce-899c-a2c425da0c33\") "
Sep 18 20:15:14.059295 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:14.059154    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e773b934-ce8f-43ce-899c-a2c425da0c33-istio-data\") pod \"e773b934-ce8f-43ce-899c-a2c425da0c33\" (UID: \"e773b934-ce8f-43ce-899c-a2c425da0c33\") "
Sep 18 20:15:14.059295 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:14.059181    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e773b934-ce8f-43ce-899c-a2c425da0c33-workload-certs\") pod \"e773b934-ce8f-43ce-899c-a2c425da0c33\" (UID: \"e773b934-ce8f-43ce-899c-a2c425da0c33\") "
Sep 18 20:15:14.059295 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:14.059255    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e773b934-ce8f-43ce-899c-a2c425da0c33-workload-socket\") pod \"e773b934-ce8f-43ce-899c-a2c425da0c33\" (UID: \"e773b934-ce8f-43ce-899c-a2c425da0c33\") "
Sep 18 20:15:14.059677 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:14.059618    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e773b934-ce8f-43ce-899c-a2c425da0c33-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e773b934-ce8f-43ce-899c-a2c425da0c33" (UID: "e773b934-ce8f-43ce-899c-a2c425da0c33"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:15:14.059732 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:14.059716    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e773b934-ce8f-43ce-899c-a2c425da0c33-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e773b934-ce8f-43ce-899c-a2c425da0c33" (UID: "e773b934-ce8f-43ce-899c-a2c425da0c33"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:15:14.059831 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:14.059809    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e773b934-ce8f-43ce-899c-a2c425da0c33-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e773b934-ce8f-43ce-899c-a2c425da0c33" (UID: "e773b934-ce8f-43ce-899c-a2c425da0c33"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:15:14.059831 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:14.059819    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e773b934-ce8f-43ce-899c-a2c425da0c33-istio-data" (OuterVolumeSpecName: "istio-data") pod "e773b934-ce8f-43ce-899c-a2c425da0c33" (UID: "e773b934-ce8f-43ce-899c-a2c425da0c33"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:15:14.059958 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:14.059854    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e773b934-ce8f-43ce-899c-a2c425da0c33-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e773b934-ce8f-43ce-899c-a2c425da0c33" (UID: "e773b934-ce8f-43ce-899c-a2c425da0c33"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:15:14.061374 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:14.061344    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e773b934-ce8f-43ce-899c-a2c425da0c33-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e773b934-ce8f-43ce-899c-a2c425da0c33" (UID: "e773b934-ce8f-43ce-899c-a2c425da0c33"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 20:15:14.061496 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:14.061403    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e773b934-ce8f-43ce-899c-a2c425da0c33-istio-token" (OuterVolumeSpecName: "istio-token") pod "e773b934-ce8f-43ce-899c-a2c425da0c33" (UID: "e773b934-ce8f-43ce-899c-a2c425da0c33"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:15:14.061558 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:14.061541    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e773b934-ce8f-43ce-899c-a2c425da0c33-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e773b934-ce8f-43ce-899c-a2c425da0c33" (UID: "e773b934-ce8f-43ce-899c-a2c425da0c33"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:15:14.061706 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:14.061684    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e773b934-ce8f-43ce-899c-a2c425da0c33-kube-api-access-klnpm" (OuterVolumeSpecName: "kube-api-access-klnpm") pod "e773b934-ce8f-43ce-899c-a2c425da0c33" (UID: "e773b934-ce8f-43ce-899c-a2c425da0c33"). InnerVolumeSpecName "kube-api-access-klnpm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:15:14.161113 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:14.161082    2549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e773b934-ce8f-43ce-899c-a2c425da0c33-workload-socket\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:15:14.161113 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:14.161109    2549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e773b934-ce8f-43ce-899c-a2c425da0c33-istio-podinfo\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:15:14.161113 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:14.161118    2549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e773b934-ce8f-43ce-899c-a2c425da0c33-istiod-ca-cert\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:15:14.161306 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:14.161126    2549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e773b934-ce8f-43ce-899c-a2c425da0c33-istio-envoy\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:15:14.161306 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:14.161136    2549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e773b934-ce8f-43ce-899c-a2c425da0c33-credential-socket\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:15:14.161306 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:14.161145    2549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e773b934-ce8f-43ce-899c-a2c425da0c33-istio-token\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:15:14.161306 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:14.161153    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klnpm\" (UniqueName: \"kubernetes.io/projected/e773b934-ce8f-43ce-899c-a2c425da0c33-kube-api-access-klnpm\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:15:14.161306 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:14.161161    2549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e773b934-ce8f-43ce-899c-a2c425da0c33-istio-data\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:15:14.161306 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:14.161169    2549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e773b934-ce8f-43ce-899c-a2c425da0c33-workload-certs\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:15:14.462166 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:14.462135    2549 generic.go:356] "Generic (PLEG): container finished" podID="e773b934-ce8f-43ce-899c-a2c425da0c33" containerID="3bedc9afbef4d4d01605b9660cc4639b3e70f994b85f0df2e97591fb2b004d29" exitCode=0
Sep 18 20:15:14.462366 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:14.462195    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48"
Sep 18 20:15:14.462366 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:14.462218    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48" event={"ID":"e773b934-ce8f-43ce-899c-a2c425da0c33","Type":"ContainerDied","Data":"3bedc9afbef4d4d01605b9660cc4639b3e70f994b85f0df2e97591fb2b004d29"}
Sep 18 20:15:14.462366 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:14.462283    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48" event={"ID":"e773b934-ce8f-43ce-899c-a2c425da0c33","Type":"ContainerDied","Data":"ebd67750f59425ef2ee0ab8c5f438e1e15eb663215c62867f1b7a6fa2c404611"}
Sep 18 20:15:14.462366 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:14.462314    2549 scope.go:117] "RemoveContainer" containerID="3bedc9afbef4d4d01605b9660cc4639b3e70f994b85f0df2e97591fb2b004d29"
Sep 18 20:15:14.472081 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:14.472063    2549 scope.go:117] "RemoveContainer" containerID="0ecd23622c0958500be173983ee355784faaaf0494e238debce8c4acd460d3c1"
Sep 18 20:15:14.480179 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:14.480163    2549 scope.go:117] "RemoveContainer" containerID="3bedc9afbef4d4d01605b9660cc4639b3e70f994b85f0df2e97591fb2b004d29"
Sep 18 20:15:14.480474 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:15:14.480457    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bedc9afbef4d4d01605b9660cc4639b3e70f994b85f0df2e97591fb2b004d29\": container with ID starting with 3bedc9afbef4d4d01605b9660cc4639b3e70f994b85f0df2e97591fb2b004d29 not found: ID does not exist" containerID="3bedc9afbef4d4d01605b9660cc4639b3e70f994b85f0df2e97591fb2b004d29"
Sep 18 20:15:14.480552 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:14.480485    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bedc9afbef4d4d01605b9660cc4639b3e70f994b85f0df2e97591fb2b004d29"} err="failed to get container status \"3bedc9afbef4d4d01605b9660cc4639b3e70f994b85f0df2e97591fb2b004d29\": rpc error: code = NotFound desc = could not find container \"3bedc9afbef4d4d01605b9660cc4639b3e70f994b85f0df2e97591fb2b004d29\": container with ID starting with 3bedc9afbef4d4d01605b9660cc4639b3e70f994b85f0df2e97591fb2b004d29 not found: ID does not exist"
Sep 18 20:15:14.480552 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:14.480508    2549 scope.go:117] "RemoveContainer" containerID="0ecd23622c0958500be173983ee355784faaaf0494e238debce8c4acd460d3c1"
Sep 18 20:15:14.480764 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:15:14.480746    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ecd23622c0958500be173983ee355784faaaf0494e238debce8c4acd460d3c1\": container with ID starting with 0ecd23622c0958500be173983ee355784faaaf0494e238debce8c4acd460d3c1 not found: ID does not exist" containerID="0ecd23622c0958500be173983ee355784faaaf0494e238debce8c4acd460d3c1"
Sep 18 20:15:14.480819 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:14.480771    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ecd23622c0958500be173983ee355784faaaf0494e238debce8c4acd460d3c1"} err="failed to get container status \"0ecd23622c0958500be173983ee355784faaaf0494e238debce8c4acd460d3c1\": rpc error: code = NotFound desc = could not find container \"0ecd23622c0958500be173983ee355784faaaf0494e238debce8c4acd460d3c1\": container with ID starting with 0ecd23622c0958500be173983ee355784faaaf0494e238debce8c4acd460d3c1 not found: ID does not exist"
Sep 18 20:15:14.524831 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:14.524797    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48"]
Sep 18 20:15:14.527008 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:14.526986    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-4b633d-openshift-default-fcd56d6f4-fxn48"]
Sep 18 20:15:15.521081 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:15.521048    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e773b934-ce8f-43ce-899c-a2c425da0c33" path="/var/lib/kubelet/pods/e773b934-ce8f-43ce-899c-a2c425da0c33/volumes"
Sep 18 20:15:20.796995 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:20.796961    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr"]
Sep 18 20:15:20.797464 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:20.797217    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr" podUID="eeadd113-df76-4c65-83e6-c99756b64684" containerName="istio-proxy" containerID="cri-o://65716d550e61a8d07528a1dd04ef02b529c9a28ed9563d00d6d568cb3a6402d5" gracePeriod=30
Sep 18 20:15:21.552747 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:21.552714    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm"]
Sep 18 20:15:21.553112 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:21.553099    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77597449-e2a2-4288-a23d-3610b1ac4595" containerName="cleanup"
Sep 18 20:15:21.553112 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:21.553112    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="77597449-e2a2-4288-a23d-3610b1ac4595" containerName="cleanup"
Sep 18 20:15:21.553233 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:21.553123    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e773b934-ce8f-43ce-899c-a2c425da0c33" containerName="istio-proxy"
Sep 18 20:15:21.553233 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:21.553129    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="e773b934-ce8f-43ce-899c-a2c425da0c33" containerName="istio-proxy"
Sep 18 20:15:21.553233 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:21.553148    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21ca971a-459c-49d8-a299-85be11490dc3" containerName="cleanup"
Sep 18 20:15:21.553233 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:21.553154    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="21ca971a-459c-49d8-a299-85be11490dc3" containerName="cleanup"
Sep 18 20:15:21.553233 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:21.553161    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e773b934-ce8f-43ce-899c-a2c425da0c33" containerName="istio-proxy"
Sep 18 20:15:21.553233 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:21.553166    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="e773b934-ce8f-43ce-899c-a2c425da0c33" containerName="istio-proxy"
Sep 18 20:15:21.553233 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:21.553177    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35274823-b198-4e3f-a8b8-d9ef1661e1b9" containerName="main"
Sep 18 20:15:21.553233 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:21.553182    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="35274823-b198-4e3f-a8b8-d9ef1661e1b9" containerName="main"
Sep 18 20:15:21.553233 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:21.553237    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="35274823-b198-4e3f-a8b8-d9ef1661e1b9" containerName="main"
Sep 18 20:15:21.553708 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:21.553245    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="77597449-e2a2-4288-a23d-3610b1ac4595" containerName="cleanup"
Sep 18 20:15:21.553708 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:21.553250    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="21ca971a-459c-49d8-a299-85be11490dc3" containerName="cleanup"
Sep 18 20:15:21.553708 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:21.553257    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="e773b934-ce8f-43ce-899c-a2c425da0c33" containerName="istio-proxy"
Sep 18 20:15:21.553708 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:21.553264    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="e773b934-ce8f-43ce-899c-a2c425da0c33" containerName="istio-proxy"
Sep 18 20:15:21.557950 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:21.557929    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm"
Sep 18 20:15:21.561224 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:21.561206    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-46f024-openshift-default-dockercfg-7hh5h\""
Sep 18 20:15:21.571452 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:21.571427    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm"]
Sep 18 20:15:21.736228 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:21.736192    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6958d670-0930-40a9-b953-db65c1a71a78-istio-envoy\") pod \"e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm\" (UID: \"6958d670-0930-40a9-b953-db65c1a71a78\") " pod="openshift-ingress/e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm"
Sep 18 20:15:21.736415 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:21.736314    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6958d670-0930-40a9-b953-db65c1a71a78-istiod-ca-cert\") pod \"e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm\" (UID: \"6958d670-0930-40a9-b953-db65c1a71a78\") " pod="openshift-ingress/e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm"
Sep 18 20:15:21.736415 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:21.736350    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6958d670-0930-40a9-b953-db65c1a71a78-istio-token\") pod \"e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm\" (UID: \"6958d670-0930-40a9-b953-db65c1a71a78\") " pod="openshift-ingress/e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm"
Sep 18 20:15:21.736499 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:21.736439    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6958d670-0930-40a9-b953-db65c1a71a78-workload-socket\") pod \"e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm\" (UID: \"6958d670-0930-40a9-b953-db65c1a71a78\") " pod="openshift-ingress/e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm"
Sep 18 20:15:21.736499 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:21.736483    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6958d670-0930-40a9-b953-db65c1a71a78-istio-data\") pod \"e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm\" (UID: \"6958d670-0930-40a9-b953-db65c1a71a78\") " pod="openshift-ingress/e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm"
Sep 18 20:15:21.736560 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:21.736510    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6958d670-0930-40a9-b953-db65c1a71a78-credential-socket\") pod \"e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm\" (UID: \"6958d670-0930-40a9-b953-db65c1a71a78\") " pod="openshift-ingress/e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm"
Sep 18 20:15:21.736560 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:21.736533    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trrvl\" (UniqueName: \"kubernetes.io/projected/6958d670-0930-40a9-b953-db65c1a71a78-kube-api-access-trrvl\") pod \"e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm\" (UID: \"6958d670-0930-40a9-b953-db65c1a71a78\") " pod="openshift-ingress/e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm"
Sep 18 20:15:21.736653 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:21.736631    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6958d670-0930-40a9-b953-db65c1a71a78-istio-podinfo\") pod \"e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm\" (UID: \"6958d670-0930-40a9-b953-db65c1a71a78\") " pod="openshift-ingress/e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm"
Sep 18 20:15:21.736701 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:21.736690    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6958d670-0930-40a9-b953-db65c1a71a78-workload-certs\") pod \"e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm\" (UID: \"6958d670-0930-40a9-b953-db65c1a71a78\") " pod="openshift-ingress/e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm"
Sep 18 20:15:21.838213 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:21.838179    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6958d670-0930-40a9-b953-db65c1a71a78-workload-socket\") pod \"e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm\" (UID: \"6958d670-0930-40a9-b953-db65c1a71a78\") " pod="openshift-ingress/e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm"
Sep 18 20:15:21.838626 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:21.838231    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6958d670-0930-40a9-b953-db65c1a71a78-istio-data\") pod \"e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm\" (UID: \"6958d670-0930-40a9-b953-db65c1a71a78\") " pod="openshift-ingress/e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm"
Sep 18 20:15:21.838626 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:21.838248    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6958d670-0930-40a9-b953-db65c1a71a78-credential-socket\") pod \"e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm\" (UID: \"6958d670-0930-40a9-b953-db65c1a71a78\") " pod="openshift-ingress/e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm"
Sep 18 20:15:21.838626 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:21.838372    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trrvl\" (UniqueName: \"kubernetes.io/projected/6958d670-0930-40a9-b953-db65c1a71a78-kube-api-access-trrvl\") pod \"e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm\" (UID: \"6958d670-0930-40a9-b953-db65c1a71a78\") " pod="openshift-ingress/e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm"
Sep 18 20:15:21.838626 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:21.838494    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6958d670-0930-40a9-b953-db65c1a71a78-istio-podinfo\") pod \"e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm\" (UID: \"6958d670-0930-40a9-b953-db65c1a71a78\") " pod="openshift-ingress/e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm"
Sep 18 20:15:21.838626 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:21.838555    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6958d670-0930-40a9-b953-db65c1a71a78-workload-certs\") pod \"e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm\" (UID: \"6958d670-0930-40a9-b953-db65c1a71a78\") " pod="openshift-ingress/e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm"
Sep 18 20:15:21.838626 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:21.838609    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6958d670-0930-40a9-b953-db65c1a71a78-istio-envoy\") pod \"e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm\" (UID: \"6958d670-0930-40a9-b953-db65c1a71a78\") " pod="openshift-ingress/e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm"
Sep 18 20:15:21.838967 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:21.838630    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6958d670-0930-40a9-b953-db65c1a71a78-workload-socket\") pod \"e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm\" (UID: \"6958d670-0930-40a9-b953-db65c1a71a78\") " pod="openshift-ingress/e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm"
Sep 18 20:15:21.838967 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:21.838654    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6958d670-0930-40a9-b953-db65c1a71a78-istiod-ca-cert\") pod \"e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm\" (UID: \"6958d670-0930-40a9-b953-db65c1a71a78\") " pod="openshift-ingress/e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm"
Sep 18 20:15:21.838967 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:21.838668    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6958d670-0930-40a9-b953-db65c1a71a78-credential-socket\") pod \"e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm\" (UID: \"6958d670-0930-40a9-b953-db65c1a71a78\") " pod="openshift-ingress/e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm"
Sep 18 20:15:21.838967 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:21.838709    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6958d670-0930-40a9-b953-db65c1a71a78-istio-data\") pod \"e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm\" (UID: \"6958d670-0930-40a9-b953-db65c1a71a78\") " pod="openshift-ingress/e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm"
Sep 18 20:15:21.838967 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:21.838721    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6958d670-0930-40a9-b953-db65c1a71a78-istio-token\") pod \"e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm\" (UID: \"6958d670-0930-40a9-b953-db65c1a71a78\") " pod="openshift-ingress/e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm"
Sep 18 20:15:21.838967 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:21.838864    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6958d670-0930-40a9-b953-db65c1a71a78-workload-certs\") pod \"e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm\" (UID: \"6958d670-0930-40a9-b953-db65c1a71a78\") " pod="openshift-ingress/e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm"
Sep 18 20:15:21.839340 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:21.839318    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6958d670-0930-40a9-b953-db65c1a71a78-istiod-ca-cert\") pod \"e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm\" (UID: \"6958d670-0930-40a9-b953-db65c1a71a78\") " pod="openshift-ingress/e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm"
Sep 18 20:15:21.840946 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:21.840927    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6958d670-0930-40a9-b953-db65c1a71a78-istio-envoy\") pod \"e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm\" (UID: \"6958d670-0930-40a9-b953-db65c1a71a78\") " pod="openshift-ingress/e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm"
Sep 18 20:15:21.841143 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:21.841125    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6958d670-0930-40a9-b953-db65c1a71a78-istio-podinfo\") pod \"e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm\" (UID: \"6958d670-0930-40a9-b953-db65c1a71a78\") " pod="openshift-ingress/e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm"
Sep 18 20:15:21.849243 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:21.849218    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6958d670-0930-40a9-b953-db65c1a71a78-istio-token\") pod \"e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm\" (UID: \"6958d670-0930-40a9-b953-db65c1a71a78\") " pod="openshift-ingress/e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm"
Sep 18 20:15:21.849684 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:21.849659    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trrvl\" (UniqueName: \"kubernetes.io/projected/6958d670-0930-40a9-b953-db65c1a71a78-kube-api-access-trrvl\") pod \"e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm\" (UID: \"6958d670-0930-40a9-b953-db65c1a71a78\") " pod="openshift-ingress/e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm"
Sep 18 20:15:21.868493 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:21.868472    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm"
Sep 18 20:15:22.158931 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:22.158892    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm"]
Sep 18 20:15:22.159678 ip-10-0-140-178 kubenswrapper[2549]: W0918 20:15:22.159651    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6958d670_0930_40a9_b953_db65c1a71a78.slice/crio-481619bb9c11f576920a3a91a7c9aac0a568c41cd756faa9b6d4d6ee1b83bad9 WatchSource:0}: Error finding container 481619bb9c11f576920a3a91a7c9aac0a568c41cd756faa9b6d4d6ee1b83bad9: Status 404 returned error can't find the container with id 481619bb9c11f576920a3a91a7c9aac0a568c41cd756faa9b6d4d6ee1b83bad9
Sep 18 20:15:22.161904 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:22.161866    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:15:22.161990 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:22.161961    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:15:22.162038 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:22.162020    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:15:22.484406 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:22.484312    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:15:22.515432 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:22.515392    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm" event={"ID":"6958d670-0930-40a9-b953-db65c1a71a78","Type":"ContainerStarted","Data":"876b8779c9367133d411bacd35055fcfb2b5269eebb6c9323f97a71ef94d90ce"}
Sep 18 20:15:22.515589 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:22.515457    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm" event={"ID":"6958d670-0930-40a9-b953-db65c1a71a78","Type":"ContainerStarted","Data":"481619bb9c11f576920a3a91a7c9aac0a568c41cd756faa9b6d4d6ee1b83bad9"}
Sep 18 20:15:22.869468 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:22.869440    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm"
Sep 18 20:15:23.874699 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:23.874669    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm"
Sep 18 20:15:23.939236 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:23.939188    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm" podStartSLOduration=2.939170179 podStartE2EDuration="2.939170179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:15:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:15:22.581360119 +0000 UTC m=+1107.535101287" watchObservedRunningTime="2026-09-18 20:15:23.939170179 +0000 UTC m=+1108.892911347"
Sep 18 20:15:24.527807 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:24.527778    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm"
Sep 18 20:15:24.528731 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:24.528711    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm"
Sep 18 20:15:26.052150 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:26.052125    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr"
Sep 18 20:15:26.183641 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:26.183614    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eeadd113-df76-4c65-83e6-c99756b64684-istio-token\") pod \"eeadd113-df76-4c65-83e6-c99756b64684\" (UID: \"eeadd113-df76-4c65-83e6-c99756b64684\") "
Sep 18 20:15:26.183800 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:26.183653    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeadd113-df76-4c65-83e6-c99756b64684-workload-socket\") pod \"eeadd113-df76-4c65-83e6-c99756b64684\" (UID: \"eeadd113-df76-4c65-83e6-c99756b64684\") "
Sep 18 20:15:26.183800 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:26.183693    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eeadd113-df76-4c65-83e6-c99756b64684-istio-envoy\") pod \"eeadd113-df76-4c65-83e6-c99756b64684\" (UID: \"eeadd113-df76-4c65-83e6-c99756b64684\") "
Sep 18 20:15:26.183800 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:26.183732    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l6gg9\" (UniqueName: \"kubernetes.io/projected/eeadd113-df76-4c65-83e6-c99756b64684-kube-api-access-l6gg9\") pod \"eeadd113-df76-4c65-83e6-c99756b64684\" (UID: \"eeadd113-df76-4c65-83e6-c99756b64684\") "
Sep 18 20:15:26.183800 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:26.183772    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eeadd113-df76-4c65-83e6-c99756b64684-istio-data\") pod \"eeadd113-df76-4c65-83e6-c99756b64684\" (UID: \"eeadd113-df76-4c65-83e6-c99756b64684\") "
Sep 18 20:15:26.183800 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:26.183795    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eeadd113-df76-4c65-83e6-c99756b64684-istiod-ca-cert\") pod \"eeadd113-df76-4c65-83e6-c99756b64684\" (UID: \"eeadd113-df76-4c65-83e6-c99756b64684\") "
Sep 18 20:15:26.184077 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:26.183837    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeadd113-df76-4c65-83e6-c99756b64684-credential-socket\") pod \"eeadd113-df76-4c65-83e6-c99756b64684\" (UID: \"eeadd113-df76-4c65-83e6-c99756b64684\") "
Sep 18 20:15:26.184077 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:26.183878    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eeadd113-df76-4c65-83e6-c99756b64684-istio-podinfo\") pod \"eeadd113-df76-4c65-83e6-c99756b64684\" (UID: \"eeadd113-df76-4c65-83e6-c99756b64684\") "
Sep 18 20:15:26.184077 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:26.183938    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eeadd113-df76-4c65-83e6-c99756b64684-workload-certs\") pod \"eeadd113-df76-4c65-83e6-c99756b64684\" (UID: \"eeadd113-df76-4c65-83e6-c99756b64684\") "
Sep 18 20:15:26.184077 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:26.184020    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeadd113-df76-4c65-83e6-c99756b64684-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "eeadd113-df76-4c65-83e6-c99756b64684" (UID: "eeadd113-df76-4c65-83e6-c99756b64684"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:15:26.184294 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:26.184090    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeadd113-df76-4c65-83e6-c99756b64684-istio-data" (OuterVolumeSpecName: "istio-data") pod "eeadd113-df76-4c65-83e6-c99756b64684" (UID: "eeadd113-df76-4c65-83e6-c99756b64684"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:15:26.184294 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:26.184181    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eeadd113-df76-4c65-83e6-c99756b64684-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "eeadd113-df76-4c65-83e6-c99756b64684" (UID: "eeadd113-df76-4c65-83e6-c99756b64684"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:15:26.184294 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:26.184240    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeadd113-df76-4c65-83e6-c99756b64684-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "eeadd113-df76-4c65-83e6-c99756b64684" (UID: "eeadd113-df76-4c65-83e6-c99756b64684"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:15:26.184294 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:26.184255    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeadd113-df76-4c65-83e6-c99756b64684-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "eeadd113-df76-4c65-83e6-c99756b64684" (UID: "eeadd113-df76-4c65-83e6-c99756b64684"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:15:26.184528 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:26.184364    2549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eeadd113-df76-4c65-83e6-c99756b64684-istio-data\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:15:26.184528 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:26.184406    2549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eeadd113-df76-4c65-83e6-c99756b64684-istiod-ca-cert\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:15:26.184528 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:26.184421    2549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeadd113-df76-4c65-83e6-c99756b64684-credential-socket\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:15:26.184528 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:26.184434    2549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eeadd113-df76-4c65-83e6-c99756b64684-workload-certs\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:15:26.184528 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:26.184447    2549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeadd113-df76-4c65-83e6-c99756b64684-workload-socket\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:15:26.186065 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:26.186039    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeadd113-df76-4c65-83e6-c99756b64684-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "eeadd113-df76-4c65-83e6-c99756b64684" (UID: "eeadd113-df76-4c65-83e6-c99756b64684"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:15:26.186065 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:26.186062    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eeadd113-df76-4c65-83e6-c99756b64684-istio-token" (OuterVolumeSpecName: "istio-token") pod "eeadd113-df76-4c65-83e6-c99756b64684" (UID: "eeadd113-df76-4c65-83e6-c99756b64684"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:15:26.186223 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:26.186047    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eeadd113-df76-4c65-83e6-c99756b64684-kube-api-access-l6gg9" (OuterVolumeSpecName: "kube-api-access-l6gg9") pod "eeadd113-df76-4c65-83e6-c99756b64684" (UID: "eeadd113-df76-4c65-83e6-c99756b64684"). InnerVolumeSpecName "kube-api-access-l6gg9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:15:26.186223 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:26.186109    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eeadd113-df76-4c65-83e6-c99756b64684-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "eeadd113-df76-4c65-83e6-c99756b64684" (UID: "eeadd113-df76-4c65-83e6-c99756b64684"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 20:15:26.285293 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:26.285249    2549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eeadd113-df76-4c65-83e6-c99756b64684-istio-token\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:15:26.285293 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:26.285288    2549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eeadd113-df76-4c65-83e6-c99756b64684-istio-envoy\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:15:26.285293 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:26.285297    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6gg9\" (UniqueName: \"kubernetes.io/projected/eeadd113-df76-4c65-83e6-c99756b64684-kube-api-access-l6gg9\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:15:26.285293 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:26.285307    2549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eeadd113-df76-4c65-83e6-c99756b64684-istio-podinfo\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:15:26.541240 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:26.541146    2549 generic.go:356] "Generic (PLEG): container finished" podID="eeadd113-df76-4c65-83e6-c99756b64684" containerID="65716d550e61a8d07528a1dd04ef02b529c9a28ed9563d00d6d568cb3a6402d5" exitCode=0
Sep 18 20:15:26.541240 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:26.541199    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr"
Sep 18 20:15:26.541240 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:26.541230    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr" event={"ID":"eeadd113-df76-4c65-83e6-c99756b64684","Type":"ContainerDied","Data":"65716d550e61a8d07528a1dd04ef02b529c9a28ed9563d00d6d568cb3a6402d5"}
Sep 18 20:15:26.541547 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:26.541285    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr" event={"ID":"eeadd113-df76-4c65-83e6-c99756b64684","Type":"ContainerDied","Data":"97f080c4979f7b0698bcc411ecc2dceddae3a91c247f6fa923b5bdefcf7ee05d"}
Sep 18 20:15:26.541547 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:26.541314    2549 scope.go:117] "RemoveContainer" containerID="65716d550e61a8d07528a1dd04ef02b529c9a28ed9563d00d6d568cb3a6402d5"
Sep 18 20:15:26.559888 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:26.559865    2549 scope.go:117] "RemoveContainer" containerID="105f0cea652267717428f2c9330d0b08b62f15c33a01d78affa90cab0d87152e"
Sep 18 20:15:26.568780 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:26.568761    2549 scope.go:117] "RemoveContainer" containerID="65716d550e61a8d07528a1dd04ef02b529c9a28ed9563d00d6d568cb3a6402d5"
Sep 18 20:15:26.569023 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:15:26.569002    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65716d550e61a8d07528a1dd04ef02b529c9a28ed9563d00d6d568cb3a6402d5\": container with ID starting with 65716d550e61a8d07528a1dd04ef02b529c9a28ed9563d00d6d568cb3a6402d5 not found: ID does not exist" containerID="65716d550e61a8d07528a1dd04ef02b529c9a28ed9563d00d6d568cb3a6402d5"
Sep 18 20:15:26.569077 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:26.569031    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65716d550e61a8d07528a1dd04ef02b529c9a28ed9563d00d6d568cb3a6402d5"} err="failed to get container status \"65716d550e61a8d07528a1dd04ef02b529c9a28ed9563d00d6d568cb3a6402d5\": rpc error: code = NotFound desc = could not find container \"65716d550e61a8d07528a1dd04ef02b529c9a28ed9563d00d6d568cb3a6402d5\": container with ID starting with 65716d550e61a8d07528a1dd04ef02b529c9a28ed9563d00d6d568cb3a6402d5 not found: ID does not exist"
Sep 18 20:15:26.569077 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:26.569047    2549 scope.go:117] "RemoveContainer" containerID="105f0cea652267717428f2c9330d0b08b62f15c33a01d78affa90cab0d87152e"
Sep 18 20:15:26.569293 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:15:26.569274    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"105f0cea652267717428f2c9330d0b08b62f15c33a01d78affa90cab0d87152e\": container with ID starting with 105f0cea652267717428f2c9330d0b08b62f15c33a01d78affa90cab0d87152e not found: ID does not exist" containerID="105f0cea652267717428f2c9330d0b08b62f15c33a01d78affa90cab0d87152e"
Sep 18 20:15:26.569334 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:26.569298    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"105f0cea652267717428f2c9330d0b08b62f15c33a01d78affa90cab0d87152e"} err="failed to get container status \"105f0cea652267717428f2c9330d0b08b62f15c33a01d78affa90cab0d87152e\": rpc error: code = NotFound desc = could not find container \"105f0cea652267717428f2c9330d0b08b62f15c33a01d78affa90cab0d87152e\": container with ID starting with 105f0cea652267717428f2c9330d0b08b62f15c33a01d78affa90cab0d87152e not found: ID does not exist"
Sep 18 20:15:26.581543 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:26.581522    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr"]
Sep 18 20:15:26.588522 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:26.588500    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-d0d77c-openshift-default-5b67d98676-5p2nr"]
Sep 18 20:15:27.521518 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:27.521484    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eeadd113-df76-4c65-83e6-c99756b64684" path="/var/lib/kubelet/pods/eeadd113-df76-4c65-83e6-c99756b64684/volumes"
Sep 18 20:15:35.846119 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:35.846079    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:15:38.509205 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:38.509165    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-c34fad/test-model-c34fad-kserve-64f6869cf5-96zdt"]
Sep 18 20:15:38.509767 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:38.509746    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eeadd113-df76-4c65-83e6-c99756b64684" containerName="istio-proxy"
Sep 18 20:15:38.509858 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:38.509770    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="eeadd113-df76-4c65-83e6-c99756b64684" containerName="istio-proxy"
Sep 18 20:15:38.509858 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:38.509791    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eeadd113-df76-4c65-83e6-c99756b64684" containerName="istio-proxy"
Sep 18 20:15:38.509858 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:38.509800    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="eeadd113-df76-4c65-83e6-c99756b64684" containerName="istio-proxy"
Sep 18 20:15:38.510016 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:38.509916    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="eeadd113-df76-4c65-83e6-c99756b64684" containerName="istio-proxy"
Sep 18 20:15:38.514747 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:38.514726    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c34fad/test-model-c34fad-kserve-64f6869cf5-96zdt"
Sep 18 20:15:38.517907 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:38.517888    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-c34fad\"/\"openshift-service-ca.crt\""
Sep 18 20:15:38.519476 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:38.519444    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-c34fad\"/\"kube-root-ca.crt\""
Sep 18 20:15:38.519583 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:38.519533    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-c34fad\"/\"default-dockercfg-72qqp\""
Sep 18 20:15:38.519654 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:38.519588    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-c34fad\"/\"test-model-c34fad-kserve-self-signed-certs\""
Sep 18 20:15:38.523152 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:38.523130    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-c34fad/test-model-c34fad-kserve-64f6869cf5-96zdt"]
Sep 18 20:15:38.599073 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:38.599041    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b50ade65-c67e-43c3-8e63-7d60bcb96d0c-dshm\") pod \"test-model-c34fad-kserve-64f6869cf5-96zdt\" (UID: \"b50ade65-c67e-43c3-8e63-7d60bcb96d0c\") " pod="ai-tenant-e2e-inf-b-c34fad/test-model-c34fad-kserve-64f6869cf5-96zdt"
Sep 18 20:15:38.599245 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:38.599092    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8f2w\" (UniqueName: \"kubernetes.io/projected/b50ade65-c67e-43c3-8e63-7d60bcb96d0c-kube-api-access-x8f2w\") pod \"test-model-c34fad-kserve-64f6869cf5-96zdt\" (UID: \"b50ade65-c67e-43c3-8e63-7d60bcb96d0c\") " pod="ai-tenant-e2e-inf-b-c34fad/test-model-c34fad-kserve-64f6869cf5-96zdt"
Sep 18 20:15:38.599245 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:38.599119    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b50ade65-c67e-43c3-8e63-7d60bcb96d0c-tmp-dir\") pod \"test-model-c34fad-kserve-64f6869cf5-96zdt\" (UID: \"b50ade65-c67e-43c3-8e63-7d60bcb96d0c\") " pod="ai-tenant-e2e-inf-b-c34fad/test-model-c34fad-kserve-64f6869cf5-96zdt"
Sep 18 20:15:38.599245 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:38.599202    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b50ade65-c67e-43c3-8e63-7d60bcb96d0c-home\") pod \"test-model-c34fad-kserve-64f6869cf5-96zdt\" (UID: \"b50ade65-c67e-43c3-8e63-7d60bcb96d0c\") " pod="ai-tenant-e2e-inf-b-c34fad/test-model-c34fad-kserve-64f6869cf5-96zdt"
Sep 18 20:15:38.599348 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:38.599256    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b50ade65-c67e-43c3-8e63-7d60bcb96d0c-model-cache\") pod \"test-model-c34fad-kserve-64f6869cf5-96zdt\" (UID: \"b50ade65-c67e-43c3-8e63-7d60bcb96d0c\") " pod="ai-tenant-e2e-inf-b-c34fad/test-model-c34fad-kserve-64f6869cf5-96zdt"
Sep 18 20:15:38.599348 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:38.599277    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b50ade65-c67e-43c3-8e63-7d60bcb96d0c-tls-certs\") pod \"test-model-c34fad-kserve-64f6869cf5-96zdt\" (UID: \"b50ade65-c67e-43c3-8e63-7d60bcb96d0c\") " pod="ai-tenant-e2e-inf-b-c34fad/test-model-c34fad-kserve-64f6869cf5-96zdt"
Sep 18 20:15:38.699785 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:38.699754    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b50ade65-c67e-43c3-8e63-7d60bcb96d0c-dshm\") pod \"test-model-c34fad-kserve-64f6869cf5-96zdt\" (UID: \"b50ade65-c67e-43c3-8e63-7d60bcb96d0c\") " pod="ai-tenant-e2e-inf-b-c34fad/test-model-c34fad-kserve-64f6869cf5-96zdt"
Sep 18 20:15:38.699949 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:38.699803    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8f2w\" (UniqueName: \"kubernetes.io/projected/b50ade65-c67e-43c3-8e63-7d60bcb96d0c-kube-api-access-x8f2w\") pod \"test-model-c34fad-kserve-64f6869cf5-96zdt\" (UID: \"b50ade65-c67e-43c3-8e63-7d60bcb96d0c\") " pod="ai-tenant-e2e-inf-b-c34fad/test-model-c34fad-kserve-64f6869cf5-96zdt"
Sep 18 20:15:38.699949 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:38.699833    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b50ade65-c67e-43c3-8e63-7d60bcb96d0c-tmp-dir\") pod \"test-model-c34fad-kserve-64f6869cf5-96zdt\" (UID: \"b50ade65-c67e-43c3-8e63-7d60bcb96d0c\") " pod="ai-tenant-e2e-inf-b-c34fad/test-model-c34fad-kserve-64f6869cf5-96zdt"
Sep 18 20:15:38.699949 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:38.699901    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b50ade65-c67e-43c3-8e63-7d60bcb96d0c-home\") pod \"test-model-c34fad-kserve-64f6869cf5-96zdt\" (UID: \"b50ade65-c67e-43c3-8e63-7d60bcb96d0c\") " pod="ai-tenant-e2e-inf-b-c34fad/test-model-c34fad-kserve-64f6869cf5-96zdt"
Sep 18 20:15:38.700144 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:38.699952    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b50ade65-c67e-43c3-8e63-7d60bcb96d0c-model-cache\") pod \"test-model-c34fad-kserve-64f6869cf5-96zdt\" (UID: \"b50ade65-c67e-43c3-8e63-7d60bcb96d0c\") " pod="ai-tenant-e2e-inf-b-c34fad/test-model-c34fad-kserve-64f6869cf5-96zdt"
Sep 18 20:15:38.700144 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:38.699977    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b50ade65-c67e-43c3-8e63-7d60bcb96d0c-tls-certs\") pod \"test-model-c34fad-kserve-64f6869cf5-96zdt\" (UID: \"b50ade65-c67e-43c3-8e63-7d60bcb96d0c\") " pod="ai-tenant-e2e-inf-b-c34fad/test-model-c34fad-kserve-64f6869cf5-96zdt"
Sep 18 20:15:38.700366 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:38.700336    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b50ade65-c67e-43c3-8e63-7d60bcb96d0c-model-cache\") pod \"test-model-c34fad-kserve-64f6869cf5-96zdt\" (UID: \"b50ade65-c67e-43c3-8e63-7d60bcb96d0c\") " pod="ai-tenant-e2e-inf-b-c34fad/test-model-c34fad-kserve-64f6869cf5-96zdt"
Sep 18 20:15:38.700366 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:38.700359    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b50ade65-c67e-43c3-8e63-7d60bcb96d0c-home\") pod \"test-model-c34fad-kserve-64f6869cf5-96zdt\" (UID: \"b50ade65-c67e-43c3-8e63-7d60bcb96d0c\") " pod="ai-tenant-e2e-inf-b-c34fad/test-model-c34fad-kserve-64f6869cf5-96zdt"
Sep 18 20:15:38.700575 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:38.700556    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b50ade65-c67e-43c3-8e63-7d60bcb96d0c-tmp-dir\") pod \"test-model-c34fad-kserve-64f6869cf5-96zdt\" (UID: \"b50ade65-c67e-43c3-8e63-7d60bcb96d0c\") " pod="ai-tenant-e2e-inf-b-c34fad/test-model-c34fad-kserve-64f6869cf5-96zdt"
Sep 18 20:15:38.702176 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:38.702153    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b50ade65-c67e-43c3-8e63-7d60bcb96d0c-dshm\") pod \"test-model-c34fad-kserve-64f6869cf5-96zdt\" (UID: \"b50ade65-c67e-43c3-8e63-7d60bcb96d0c\") " pod="ai-tenant-e2e-inf-b-c34fad/test-model-c34fad-kserve-64f6869cf5-96zdt"
Sep 18 20:15:38.702442 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:38.702426    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b50ade65-c67e-43c3-8e63-7d60bcb96d0c-tls-certs\") pod \"test-model-c34fad-kserve-64f6869cf5-96zdt\" (UID: \"b50ade65-c67e-43c3-8e63-7d60bcb96d0c\") " pod="ai-tenant-e2e-inf-b-c34fad/test-model-c34fad-kserve-64f6869cf5-96zdt"
Sep 18 20:15:38.709664 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:38.709636    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8f2w\" (UniqueName: \"kubernetes.io/projected/b50ade65-c67e-43c3-8e63-7d60bcb96d0c-kube-api-access-x8f2w\") pod \"test-model-c34fad-kserve-64f6869cf5-96zdt\" (UID: \"b50ade65-c67e-43c3-8e63-7d60bcb96d0c\") " pod="ai-tenant-e2e-inf-b-c34fad/test-model-c34fad-kserve-64f6869cf5-96zdt"
Sep 18 20:15:38.824855 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:38.824825    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c34fad/test-model-c34fad-kserve-64f6869cf5-96zdt"
Sep 18 20:15:39.128656 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:39.128632    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-c34fad/test-model-c34fad-kserve-64f6869cf5-96zdt"]
Sep 18 20:15:39.130178 ip-10-0-140-178 kubenswrapper[2549]: W0918 20:15:39.130151    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb50ade65_c67e_43c3_8e63_7d60bcb96d0c.slice/crio-4bccc5ba8f9269c0f46db41f84e96d582e629acae03eb44d27eb202c6ecf4124 WatchSource:0}: Error finding container 4bccc5ba8f9269c0f46db41f84e96d582e629acae03eb44d27eb202c6ecf4124: Status 404 returned error can't find the container with id 4bccc5ba8f9269c0f46db41f84e96d582e629acae03eb44d27eb202c6ecf4124
Sep 18 20:15:39.621414 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:39.621357    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c34fad/test-model-c34fad-kserve-64f6869cf5-96zdt" event={"ID":"b50ade65-c67e-43c3-8e63-7d60bcb96d0c","Type":"ContainerStarted","Data":"d4ae7dfc34d4b89dd89864886c6582853b71ac97d087344c1af5c5e19695145d"}
Sep 18 20:15:39.621896 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:39.621435    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c34fad/test-model-c34fad-kserve-64f6869cf5-96zdt" event={"ID":"b50ade65-c67e-43c3-8e63-7d60bcb96d0c","Type":"ContainerStarted","Data":"4bccc5ba8f9269c0f46db41f84e96d582e629acae03eb44d27eb202c6ecf4124"}
Sep 18 20:15:39.649092 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:39.649033    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-c34fad/test-model-c34fad-kserve-64f6869cf5-96zdt" podStartSLOduration=1.648998739 podStartE2EDuration="1.648998739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:15:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:15:39.644497618 +0000 UTC m=+1124.598238787" watchObservedRunningTime="2026-09-18 20:15:39.648998739 +0000 UTC m=+1124.602739908"
Sep 18 20:15:46.672309 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:46.672228    2549 generic.go:356] "Generic (PLEG): container finished" podID="6958d670-0930-40a9-b953-db65c1a71a78" containerID="876b8779c9367133d411bacd35055fcfb2b5269eebb6c9323f97a71ef94d90ce" exitCode=0
Sep 18 20:15:46.672309 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:46.672281    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm" event={"ID":"6958d670-0930-40a9-b953-db65c1a71a78","Type":"ContainerDied","Data":"876b8779c9367133d411bacd35055fcfb2b5269eebb6c9323f97a71ef94d90ce"}
Sep 18 20:15:46.672839 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:46.672732    2549 scope.go:117] "RemoveContainer" containerID="876b8779c9367133d411bacd35055fcfb2b5269eebb6c9323f97a71ef94d90ce"
Sep 18 20:15:46.674102 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:46.674060    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:15:46.674188 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:46.674154    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:15:46.674230 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:46.674189    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:15:47.680881 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:47.680844    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm" event={"ID":"6958d670-0930-40a9-b953-db65c1a71a78","Type":"ContainerStarted","Data":"ac1a1ce6600aa1e8418c76972edbbea830f2b179d88c9a02b88d8789a80a9f92"}
Sep 18 20:15:47.868889 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:47.868844    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm"
Sep 18 20:15:47.869093 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:47.868979    2549 patch_prober.go:28] interesting pod/e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Sep 18 20:15:47.869093 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:47.869022    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm" podUID="6958d670-0930-40a9-b953-db65c1a71a78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Sep 18 20:15:48.825359 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:48.825320    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-c34fad/test-model-c34fad-kserve-64f6869cf5-96zdt"
Sep 18 20:15:48.825773 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:48.825369    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-c34fad/test-model-c34fad-kserve-64f6869cf5-96zdt"
Sep 18 20:15:48.837982 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:48.837958    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-c34fad/test-model-c34fad-kserve-64f6869cf5-96zdt"
Sep 18 20:15:48.869876 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:48.869827    2549 patch_prober.go:28] interesting pod/e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Sep 18 20:15:48.870070 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:48.869892    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm" podUID="6958d670-0930-40a9-b953-db65c1a71a78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Sep 18 20:15:49.710430 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:49.710399    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-c34fad/test-model-c34fad-kserve-64f6869cf5-96zdt"
Sep 18 20:15:49.869841 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:49.869806    2549 patch_prober.go:28] interesting pod/e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Sep 18 20:15:49.870447 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:49.870414    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm" podUID="6958d670-0930-40a9-b953-db65c1a71a78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Sep 18 20:15:50.869356 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:50.869314    2549 patch_prober.go:28] interesting pod/e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Sep 18 20:15:50.869575 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:50.869409    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm" podUID="6958d670-0930-40a9-b953-db65c1a71a78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Sep 18 20:15:51.869794 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:51.869754    2549 patch_prober.go:28] interesting pod/e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Sep 18 20:15:51.870431 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:51.870373    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm" podUID="6958d670-0930-40a9-b953-db65c1a71a78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Sep 18 20:15:52.869375 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:52.868882    2549 patch_prober.go:28] interesting pod/e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Sep 18 20:15:52.869375 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:52.868970    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm" podUID="6958d670-0930-40a9-b953-db65c1a71a78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Sep 18 20:15:53.873017 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:53.872989    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm"
Sep 18 20:15:53.873414 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:53.873266    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm"
Sep 18 20:15:53.873873 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:53.873853    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm"
Sep 18 20:15:55.061470 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:15:55.061436    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:16:08.825275 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:08.825215    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:16:13.734770 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:13.734732    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-46f024-cxt5r"]
Sep 18 20:16:13.735249 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:13.735234    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="eeadd113-df76-4c65-83e6-c99756b64684" containerName="istio-proxy"
Sep 18 20:16:13.739600 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:13.739584    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-46f024-cxt5r"
Sep 18 20:16:13.742604 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:13.742579    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 20:16:13.742745 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:13.742669    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bhssz\""
Sep 18 20:16:13.748529 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:13.748504    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-46f024-cxt5r"]
Sep 18 20:16:13.761932 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:13.761908    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/493b6c1f-0cf2-4679-8e30-1c6301dbaa1b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-46f024-cxt5r\" (UID: \"493b6c1f-0cf2-4679-8e30-1c6301dbaa1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-46f024-cxt5r"
Sep 18 20:16:13.862791 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:13.862758    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/493b6c1f-0cf2-4679-8e30-1c6301dbaa1b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-46f024-cxt5r\" (UID: \"493b6c1f-0cf2-4679-8e30-1c6301dbaa1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-46f024-cxt5r"
Sep 18 20:16:13.863342 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:13.863324    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/493b6c1f-0cf2-4679-8e30-1c6301dbaa1b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-46f024-cxt5r\" (UID: \"493b6c1f-0cf2-4679-8e30-1c6301dbaa1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-46f024-cxt5r"
Sep 18 20:16:14.052516 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:14.052437    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-46f024-cxt5r"
Sep 18 20:16:14.331349 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:14.331318    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-46f024-cxt5r"]
Sep 18 20:16:14.331599 ip-10-0-140-178 kubenswrapper[2549]: W0918 20:16:14.331577    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod493b6c1f_0cf2_4679_8e30_1c6301dbaa1b.slice/crio-7ada95caa6c6b2f6ecf8b98ababcd7926afce92e0af24347f3c0be31a0c73a7c WatchSource:0}: Error finding container 7ada95caa6c6b2f6ecf8b98ababcd7926afce92e0af24347f3c0be31a0c73a7c: Status 404 returned error can't find the container with id 7ada95caa6c6b2f6ecf8b98ababcd7926afce92e0af24347f3c0be31a0c73a7c
Sep 18 20:16:14.842868 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:14.842799    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-46f024-cxt5r" event={"ID":"493b6c1f-0cf2-4679-8e30-1c6301dbaa1b","Type":"ContainerStarted","Data":"480fff18c6b4d1ed2e127d546c0f286a113ca41b678bb4fd5117ba181fbcae78"}
Sep 18 20:16:14.842868 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:14.842857    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-46f024-cxt5r" event={"ID":"493b6c1f-0cf2-4679-8e30-1c6301dbaa1b","Type":"ContainerStarted","Data":"7ada95caa6c6b2f6ecf8b98ababcd7926afce92e0af24347f3c0be31a0c73a7c"}
Sep 18 20:16:14.873045 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:14.872995    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-46f024-cxt5r" podStartSLOduration=1.8729793689999998 podStartE2EDuration="1.872979369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:16:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:16:14.870805045 +0000 UTC m=+1159.824546212" watchObservedRunningTime="2026-09-18 20:16:14.872979369 +0000 UTC m=+1159.826720528"
Sep 18 20:16:15.850347 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:15.850311    2549 generic.go:356] "Generic (PLEG): container finished" podID="493b6c1f-0cf2-4679-8e30-1c6301dbaa1b" containerID="480fff18c6b4d1ed2e127d546c0f286a113ca41b678bb4fd5117ba181fbcae78" exitCode=0
Sep 18 20:16:15.850747 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:15.850412    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-46f024-cxt5r" event={"ID":"493b6c1f-0cf2-4679-8e30-1c6301dbaa1b","Type":"ContainerDied","Data":"480fff18c6b4d1ed2e127d546c0f286a113ca41b678bb4fd5117ba181fbcae78"}
Sep 18 20:16:16.994585 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:16.994557    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-46f024-cxt5r"
Sep 18 20:16:17.094337 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:17.094300    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/493b6c1f-0cf2-4679-8e30-1c6301dbaa1b-maas-api-service-ca\") pod \"493b6c1f-0cf2-4679-8e30-1c6301dbaa1b\" (UID: \"493b6c1f-0cf2-4679-8e30-1c6301dbaa1b\") "
Sep 18 20:16:17.094672 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:17.094647    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/493b6c1f-0cf2-4679-8e30-1c6301dbaa1b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "493b6c1f-0cf2-4679-8e30-1c6301dbaa1b" (UID: "493b6c1f-0cf2-4679-8e30-1c6301dbaa1b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:16:17.196002 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:17.195917    2549 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/493b6c1f-0cf2-4679-8e30-1c6301dbaa1b-maas-api-service-ca\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:16:17.863597 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:17.863569    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-46f024-cxt5r"
Sep 18 20:16:17.863807 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:17.863568    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-46f024-cxt5r" event={"ID":"493b6c1f-0cf2-4679-8e30-1c6301dbaa1b","Type":"ContainerDied","Data":"7ada95caa6c6b2f6ecf8b98ababcd7926afce92e0af24347f3c0be31a0c73a7c"}
Sep 18 20:16:17.863807 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:17.863693    2549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ada95caa6c6b2f6ecf8b98ababcd7926afce92e0af24347f3c0be31a0c73a7c"
Sep 18 20:16:25.550817 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:25.550782    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-46f024-cxt5r"]
Sep 18 20:16:25.554481 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:25.554455    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-46f024-cxt5r"]
Sep 18 20:16:27.521278 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:27.521249    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="493b6c1f-0cf2-4679-8e30-1c6301dbaa1b" path="/var/lib/kubelet/pods/493b6c1f-0cf2-4679-8e30-1c6301dbaa1b/volumes"
Sep 18 20:16:30.662656 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:30.662615    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df"]
Sep 18 20:16:30.663055 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:30.663035    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="493b6c1f-0cf2-4679-8e30-1c6301dbaa1b" containerName="revoke-keys"
Sep 18 20:16:30.663055 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:30.663047    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="493b6c1f-0cf2-4679-8e30-1c6301dbaa1b" containerName="revoke-keys"
Sep 18 20:16:30.663139 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:30.663120    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="493b6c1f-0cf2-4679-8e30-1c6301dbaa1b" containerName="revoke-keys"
Sep 18 20:16:30.666058 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:30.666042    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df"
Sep 18 20:16:30.670035 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:16:30.670006    2549 status_manager.go:1018] "Failed to get status for pod" err="pods \"e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df\" is forbidden: User \"system:node:ip-10-0-140-178.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-140-178.ec2.internal' and this object" podUID="964d92ac-6e8a-4b90-ad38-47cfd743b76c" pod="openshift-ingress/e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df"
Sep 18 20:16:30.670158 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:16:30.670037    2549 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"e2e-mt-aba3750c-openshift-default-dockercfg-kfb4f\" is forbidden: User \"system:node:ip-10-0-140-178.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-140-178.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"e2e-mt-aba3750c-openshift-default-dockercfg-kfb4f\"" type="*v1.Secret"
Sep 18 20:16:30.681195 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:30.681169    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df"]
Sep 18 20:16:30.719801 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:30.719760    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/964d92ac-6e8a-4b90-ad38-47cfd743b76c-istio-podinfo\") pod \"e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df\" (UID: \"964d92ac-6e8a-4b90-ad38-47cfd743b76c\") " pod="openshift-ingress/e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df"
Sep 18 20:16:30.719801 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:30.719797    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/964d92ac-6e8a-4b90-ad38-47cfd743b76c-workload-certs\") pod \"e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df\" (UID: \"964d92ac-6e8a-4b90-ad38-47cfd743b76c\") " pod="openshift-ingress/e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df"
Sep 18 20:16:30.719992 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:30.719836    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/964d92ac-6e8a-4b90-ad38-47cfd743b76c-istio-data\") pod \"e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df\" (UID: \"964d92ac-6e8a-4b90-ad38-47cfd743b76c\") " pod="openshift-ingress/e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df"
Sep 18 20:16:30.719992 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:30.719866    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/964d92ac-6e8a-4b90-ad38-47cfd743b76c-credential-socket\") pod \"e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df\" (UID: \"964d92ac-6e8a-4b90-ad38-47cfd743b76c\") " pod="openshift-ingress/e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df"
Sep 18 20:16:30.719992 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:30.719882    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8wwh\" (UniqueName: \"kubernetes.io/projected/964d92ac-6e8a-4b90-ad38-47cfd743b76c-kube-api-access-n8wwh\") pod \"e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df\" (UID: \"964d92ac-6e8a-4b90-ad38-47cfd743b76c\") " pod="openshift-ingress/e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df"
Sep 18 20:16:30.719992 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:30.719963    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/964d92ac-6e8a-4b90-ad38-47cfd743b76c-istio-envoy\") pod \"e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df\" (UID: \"964d92ac-6e8a-4b90-ad38-47cfd743b76c\") " pod="openshift-ingress/e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df"
Sep 18 20:16:30.719992 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:30.719980    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/964d92ac-6e8a-4b90-ad38-47cfd743b76c-istio-token\") pod \"e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df\" (UID: \"964d92ac-6e8a-4b90-ad38-47cfd743b76c\") " pod="openshift-ingress/e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df"
Sep 18 20:16:30.720189 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:30.720004    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/964d92ac-6e8a-4b90-ad38-47cfd743b76c-istiod-ca-cert\") pod \"e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df\" (UID: \"964d92ac-6e8a-4b90-ad38-47cfd743b76c\") " pod="openshift-ingress/e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df"
Sep 18 20:16:30.720189 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:30.720024    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/964d92ac-6e8a-4b90-ad38-47cfd743b76c-workload-socket\") pod \"e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df\" (UID: \"964d92ac-6e8a-4b90-ad38-47cfd743b76c\") " pod="openshift-ingress/e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df"
Sep 18 20:16:30.821203 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:30.821165    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/964d92ac-6e8a-4b90-ad38-47cfd743b76c-workload-certs\") pod \"e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df\" (UID: \"964d92ac-6e8a-4b90-ad38-47cfd743b76c\") " pod="openshift-ingress/e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df"
Sep 18 20:16:30.821412 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:30.821235    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/964d92ac-6e8a-4b90-ad38-47cfd743b76c-istio-data\") pod \"e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df\" (UID: \"964d92ac-6e8a-4b90-ad38-47cfd743b76c\") " pod="openshift-ingress/e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df"
Sep 18 20:16:30.821412 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:30.821267    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/964d92ac-6e8a-4b90-ad38-47cfd743b76c-credential-socket\") pod \"e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df\" (UID: \"964d92ac-6e8a-4b90-ad38-47cfd743b76c\") " pod="openshift-ingress/e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df"
Sep 18 20:16:30.821412 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:30.821290    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8wwh\" (UniqueName: \"kubernetes.io/projected/964d92ac-6e8a-4b90-ad38-47cfd743b76c-kube-api-access-n8wwh\") pod \"e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df\" (UID: \"964d92ac-6e8a-4b90-ad38-47cfd743b76c\") " pod="openshift-ingress/e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df"
Sep 18 20:16:30.821412 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:30.821376    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/964d92ac-6e8a-4b90-ad38-47cfd743b76c-istio-envoy\") pod \"e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df\" (UID: \"964d92ac-6e8a-4b90-ad38-47cfd743b76c\") " pod="openshift-ingress/e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df"
Sep 18 20:16:30.821663 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:30.821419    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/964d92ac-6e8a-4b90-ad38-47cfd743b76c-istio-token\") pod \"e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df\" (UID: \"964d92ac-6e8a-4b90-ad38-47cfd743b76c\") " pod="openshift-ingress/e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df"
Sep 18 20:16:30.821663 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:30.821462    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/964d92ac-6e8a-4b90-ad38-47cfd743b76c-istiod-ca-cert\") pod \"e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df\" (UID: \"964d92ac-6e8a-4b90-ad38-47cfd743b76c\") " pod="openshift-ingress/e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df"
Sep 18 20:16:30.821663 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:30.821495    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/964d92ac-6e8a-4b90-ad38-47cfd743b76c-workload-socket\") pod \"e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df\" (UID: \"964d92ac-6e8a-4b90-ad38-47cfd743b76c\") " pod="openshift-ingress/e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df"
Sep 18 20:16:30.821663 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:30.821556    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/964d92ac-6e8a-4b90-ad38-47cfd743b76c-istio-podinfo\") pod \"e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df\" (UID: \"964d92ac-6e8a-4b90-ad38-47cfd743b76c\") " pod="openshift-ingress/e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df"
Sep 18 20:16:30.821663 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:30.821641    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/964d92ac-6e8a-4b90-ad38-47cfd743b76c-workload-certs\") pod \"e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df\" (UID: \"964d92ac-6e8a-4b90-ad38-47cfd743b76c\") " pod="openshift-ingress/e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df"
Sep 18 20:16:30.821926 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:30.821675    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/964d92ac-6e8a-4b90-ad38-47cfd743b76c-credential-socket\") pod \"e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df\" (UID: \"964d92ac-6e8a-4b90-ad38-47cfd743b76c\") " pod="openshift-ingress/e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df"
Sep 18 20:16:30.821994 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:30.821952    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/964d92ac-6e8a-4b90-ad38-47cfd743b76c-workload-socket\") pod \"e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df\" (UID: \"964d92ac-6e8a-4b90-ad38-47cfd743b76c\") " pod="openshift-ingress/e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df"
Sep 18 20:16:30.822046 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:30.821987    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/964d92ac-6e8a-4b90-ad38-47cfd743b76c-istio-data\") pod \"e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df\" (UID: \"964d92ac-6e8a-4b90-ad38-47cfd743b76c\") " pod="openshift-ingress/e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df"
Sep 18 20:16:30.822291 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:30.822274    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/964d92ac-6e8a-4b90-ad38-47cfd743b76c-istiod-ca-cert\") pod \"e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df\" (UID: \"964d92ac-6e8a-4b90-ad38-47cfd743b76c\") " pod="openshift-ingress/e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df"
Sep 18 20:16:30.823743 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:30.823725    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/964d92ac-6e8a-4b90-ad38-47cfd743b76c-istio-envoy\") pod \"e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df\" (UID: \"964d92ac-6e8a-4b90-ad38-47cfd743b76c\") " pod="openshift-ingress/e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df"
Sep 18 20:16:30.823878 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:30.823862    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/964d92ac-6e8a-4b90-ad38-47cfd743b76c-istio-podinfo\") pod \"e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df\" (UID: \"964d92ac-6e8a-4b90-ad38-47cfd743b76c\") " pod="openshift-ingress/e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df"
Sep 18 20:16:30.833174 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:30.833141    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/964d92ac-6e8a-4b90-ad38-47cfd743b76c-istio-token\") pod \"e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df\" (UID: \"964d92ac-6e8a-4b90-ad38-47cfd743b76c\") " pod="openshift-ingress/e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df"
Sep 18 20:16:30.834481 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:30.834461    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8wwh\" (UniqueName: \"kubernetes.io/projected/964d92ac-6e8a-4b90-ad38-47cfd743b76c-kube-api-access-n8wwh\") pod \"e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df\" (UID: \"964d92ac-6e8a-4b90-ad38-47cfd743b76c\") " pod="openshift-ingress/e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df"
Sep 18 20:16:31.507246 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:31.507203    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-aba3750c-openshift-default-dockercfg-kfb4f\""
Sep 18 20:16:31.517059 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:31.517021    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df"
Sep 18 20:16:31.872907 ip-10-0-140-178 kubenswrapper[2549]: W0918 20:16:31.872870    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod964d92ac_6e8a_4b90_ad38_47cfd743b76c.slice/crio-d98efadb3722733446518451e50839a313bbf600514e2f3f483314b538597ce2 WatchSource:0}: Error finding container d98efadb3722733446518451e50839a313bbf600514e2f3f483314b538597ce2: Status 404 returned error can't find the container with id d98efadb3722733446518451e50839a313bbf600514e2f3f483314b538597ce2
Sep 18 20:16:31.875639 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:31.875569    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:16:31.875766 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:31.875691    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:16:31.875766 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:31.875732    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:16:31.875844 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:31.875800    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df"]
Sep 18 20:16:31.878609 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:31.878569    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:16:31.954541 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:31.954505    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df" event={"ID":"964d92ac-6e8a-4b90-ad38-47cfd743b76c","Type":"ContainerStarted","Data":"d98efadb3722733446518451e50839a313bbf600514e2f3f483314b538597ce2"}
Sep 18 20:16:32.961828 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:32.961787    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df" event={"ID":"964d92ac-6e8a-4b90-ad38-47cfd743b76c","Type":"ContainerStarted","Data":"fba5c5d0da6af1816b9890f3f679f6c793b1d659d130304fcaa10c8497ccef4d"}
Sep 18 20:16:33.021261 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:33.021196    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df" podStartSLOduration=3.021177238 podStartE2EDuration="3.021177238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:16:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:16:33.017249269 +0000 UTC m=+1177.970990436" watchObservedRunningTime="2026-09-18 20:16:33.021177238 +0000 UTC m=+1177.974918406"
Sep 18 20:16:33.522300 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:33.522267    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df"
Sep 18 20:16:33.523351 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:33.523325    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df"
Sep 18 20:16:33.968046 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:33.968006    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df"
Sep 18 20:16:33.969717 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:33.969691    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df"
Sep 18 20:16:35.218090 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:35.218057    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm"]
Sep 18 20:16:35.218488 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:35.218266    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm" podUID="6958d670-0930-40a9-b953-db65c1a71a78" containerName="istio-proxy" containerID="cri-o://ac1a1ce6600aa1e8418c76972edbbea830f2b179d88c9a02b88d8789a80a9f92" gracePeriod=30
Sep 18 20:16:36.122227 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:36.122197    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-a733e2-openshift-default-66b6684854-xdvs6"]
Sep 18 20:16:36.125826 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:36.125808    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-a733e2-openshift-default-66b6684854-xdvs6"
Sep 18 20:16:36.131183 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:36.131158    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-a733e2-openshift-default-dockercfg-xhpz4\""
Sep 18 20:16:36.146491 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:36.146462    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-a733e2-openshift-default-66b6684854-xdvs6"]
Sep 18 20:16:36.285278 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:36.285245    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56800167-e66c-4b44-af4a-9404d63c662b-workload-socket\") pod \"e2e-noten-a733e2-openshift-default-66b6684854-xdvs6\" (UID: \"56800167-e66c-4b44-af4a-9404d63c662b\") " pod="openshift-ingress/e2e-noten-a733e2-openshift-default-66b6684854-xdvs6"
Sep 18 20:16:36.285702 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:36.285289    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56800167-e66c-4b44-af4a-9404d63c662b-istio-token\") pod \"e2e-noten-a733e2-openshift-default-66b6684854-xdvs6\" (UID: \"56800167-e66c-4b44-af4a-9404d63c662b\") " pod="openshift-ingress/e2e-noten-a733e2-openshift-default-66b6684854-xdvs6"
Sep 18 20:16:36.285702 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:36.285323    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btsr7\" (UniqueName: \"kubernetes.io/projected/56800167-e66c-4b44-af4a-9404d63c662b-kube-api-access-btsr7\") pod \"e2e-noten-a733e2-openshift-default-66b6684854-xdvs6\" (UID: \"56800167-e66c-4b44-af4a-9404d63c662b\") " pod="openshift-ingress/e2e-noten-a733e2-openshift-default-66b6684854-xdvs6"
Sep 18 20:16:36.285702 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:36.285421    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56800167-e66c-4b44-af4a-9404d63c662b-istiod-ca-cert\") pod \"e2e-noten-a733e2-openshift-default-66b6684854-xdvs6\" (UID: \"56800167-e66c-4b44-af4a-9404d63c662b\") " pod="openshift-ingress/e2e-noten-a733e2-openshift-default-66b6684854-xdvs6"
Sep 18 20:16:36.285702 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:36.285487    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56800167-e66c-4b44-af4a-9404d63c662b-istio-envoy\") pod \"e2e-noten-a733e2-openshift-default-66b6684854-xdvs6\" (UID: \"56800167-e66c-4b44-af4a-9404d63c662b\") " pod="openshift-ingress/e2e-noten-a733e2-openshift-default-66b6684854-xdvs6"
Sep 18 20:16:36.285702 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:36.285529    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56800167-e66c-4b44-af4a-9404d63c662b-istio-data\") pod \"e2e-noten-a733e2-openshift-default-66b6684854-xdvs6\" (UID: \"56800167-e66c-4b44-af4a-9404d63c662b\") " pod="openshift-ingress/e2e-noten-a733e2-openshift-default-66b6684854-xdvs6"
Sep 18 20:16:36.285702 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:36.285544    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56800167-e66c-4b44-af4a-9404d63c662b-istio-podinfo\") pod \"e2e-noten-a733e2-openshift-default-66b6684854-xdvs6\" (UID: \"56800167-e66c-4b44-af4a-9404d63c662b\") " pod="openshift-ingress/e2e-noten-a733e2-openshift-default-66b6684854-xdvs6"
Sep 18 20:16:36.285702 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:36.285566    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56800167-e66c-4b44-af4a-9404d63c662b-credential-socket\") pod \"e2e-noten-a733e2-openshift-default-66b6684854-xdvs6\" (UID: \"56800167-e66c-4b44-af4a-9404d63c662b\") " pod="openshift-ingress/e2e-noten-a733e2-openshift-default-66b6684854-xdvs6"
Sep 18 20:16:36.285702 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:36.285591    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56800167-e66c-4b44-af4a-9404d63c662b-workload-certs\") pod \"e2e-noten-a733e2-openshift-default-66b6684854-xdvs6\" (UID: \"56800167-e66c-4b44-af4a-9404d63c662b\") " pod="openshift-ingress/e2e-noten-a733e2-openshift-default-66b6684854-xdvs6"
Sep 18 20:16:36.386579 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:36.386496    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btsr7\" (UniqueName: \"kubernetes.io/projected/56800167-e66c-4b44-af4a-9404d63c662b-kube-api-access-btsr7\") pod \"e2e-noten-a733e2-openshift-default-66b6684854-xdvs6\" (UID: \"56800167-e66c-4b44-af4a-9404d63c662b\") " pod="openshift-ingress/e2e-noten-a733e2-openshift-default-66b6684854-xdvs6"
Sep 18 20:16:36.386579 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:36.386535    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56800167-e66c-4b44-af4a-9404d63c662b-istiod-ca-cert\") pod \"e2e-noten-a733e2-openshift-default-66b6684854-xdvs6\" (UID: \"56800167-e66c-4b44-af4a-9404d63c662b\") " pod="openshift-ingress/e2e-noten-a733e2-openshift-default-66b6684854-xdvs6"
Sep 18 20:16:36.386579 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:36.386557    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56800167-e66c-4b44-af4a-9404d63c662b-istio-envoy\") pod \"e2e-noten-a733e2-openshift-default-66b6684854-xdvs6\" (UID: \"56800167-e66c-4b44-af4a-9404d63c662b\") " pod="openshift-ingress/e2e-noten-a733e2-openshift-default-66b6684854-xdvs6"
Sep 18 20:16:36.386858 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:36.386592    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56800167-e66c-4b44-af4a-9404d63c662b-istio-data\") pod \"e2e-noten-a733e2-openshift-default-66b6684854-xdvs6\" (UID: \"56800167-e66c-4b44-af4a-9404d63c662b\") " pod="openshift-ingress/e2e-noten-a733e2-openshift-default-66b6684854-xdvs6"
Sep 18 20:16:36.386858 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:36.386680    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56800167-e66c-4b44-af4a-9404d63c662b-istio-podinfo\") pod \"e2e-noten-a733e2-openshift-default-66b6684854-xdvs6\" (UID: \"56800167-e66c-4b44-af4a-9404d63c662b\") " pod="openshift-ingress/e2e-noten-a733e2-openshift-default-66b6684854-xdvs6"
Sep 18 20:16:36.386858 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:36.386730    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56800167-e66c-4b44-af4a-9404d63c662b-credential-socket\") pod \"e2e-noten-a733e2-openshift-default-66b6684854-xdvs6\" (UID: \"56800167-e66c-4b44-af4a-9404d63c662b\") " pod="openshift-ingress/e2e-noten-a733e2-openshift-default-66b6684854-xdvs6"
Sep 18 20:16:36.386858 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:36.386762    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56800167-e66c-4b44-af4a-9404d63c662b-workload-certs\") pod \"e2e-noten-a733e2-openshift-default-66b6684854-xdvs6\" (UID: \"56800167-e66c-4b44-af4a-9404d63c662b\") " pod="openshift-ingress/e2e-noten-a733e2-openshift-default-66b6684854-xdvs6"
Sep 18 20:16:36.387074 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:36.386868    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56800167-e66c-4b44-af4a-9404d63c662b-workload-socket\") pod \"e2e-noten-a733e2-openshift-default-66b6684854-xdvs6\" (UID: \"56800167-e66c-4b44-af4a-9404d63c662b\") " pod="openshift-ingress/e2e-noten-a733e2-openshift-default-66b6684854-xdvs6"
Sep 18 20:16:36.387074 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:36.386917    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56800167-e66c-4b44-af4a-9404d63c662b-istio-token\") pod \"e2e-noten-a733e2-openshift-default-66b6684854-xdvs6\" (UID: \"56800167-e66c-4b44-af4a-9404d63c662b\") " pod="openshift-ingress/e2e-noten-a733e2-openshift-default-66b6684854-xdvs6"
Sep 18 20:16:36.387168 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:36.387077    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56800167-e66c-4b44-af4a-9404d63c662b-istio-data\") pod \"e2e-noten-a733e2-openshift-default-66b6684854-xdvs6\" (UID: \"56800167-e66c-4b44-af4a-9404d63c662b\") " pod="openshift-ingress/e2e-noten-a733e2-openshift-default-66b6684854-xdvs6"
Sep 18 20:16:36.387211 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:36.387178    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56800167-e66c-4b44-af4a-9404d63c662b-credential-socket\") pod \"e2e-noten-a733e2-openshift-default-66b6684854-xdvs6\" (UID: \"56800167-e66c-4b44-af4a-9404d63c662b\") " pod="openshift-ingress/e2e-noten-a733e2-openshift-default-66b6684854-xdvs6"
Sep 18 20:16:36.387211 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:36.387198    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56800167-e66c-4b44-af4a-9404d63c662b-workload-certs\") pod \"e2e-noten-a733e2-openshift-default-66b6684854-xdvs6\" (UID: \"56800167-e66c-4b44-af4a-9404d63c662b\") " pod="openshift-ingress/e2e-noten-a733e2-openshift-default-66b6684854-xdvs6"
Sep 18 20:16:36.387486 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:36.387466    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56800167-e66c-4b44-af4a-9404d63c662b-workload-socket\") pod \"e2e-noten-a733e2-openshift-default-66b6684854-xdvs6\" (UID: \"56800167-e66c-4b44-af4a-9404d63c662b\") " pod="openshift-ingress/e2e-noten-a733e2-openshift-default-66b6684854-xdvs6"
Sep 18 20:16:36.387486 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:36.387476    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56800167-e66c-4b44-af4a-9404d63c662b-istiod-ca-cert\") pod \"e2e-noten-a733e2-openshift-default-66b6684854-xdvs6\" (UID: \"56800167-e66c-4b44-af4a-9404d63c662b\") " pod="openshift-ingress/e2e-noten-a733e2-openshift-default-66b6684854-xdvs6"
Sep 18 20:16:36.389444 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:36.389418    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56800167-e66c-4b44-af4a-9404d63c662b-istio-envoy\") pod \"e2e-noten-a733e2-openshift-default-66b6684854-xdvs6\" (UID: \"56800167-e66c-4b44-af4a-9404d63c662b\") " pod="openshift-ingress/e2e-noten-a733e2-openshift-default-66b6684854-xdvs6"
Sep 18 20:16:36.389611 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:36.389592    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56800167-e66c-4b44-af4a-9404d63c662b-istio-podinfo\") pod \"e2e-noten-a733e2-openshift-default-66b6684854-xdvs6\" (UID: \"56800167-e66c-4b44-af4a-9404d63c662b\") " pod="openshift-ingress/e2e-noten-a733e2-openshift-default-66b6684854-xdvs6"
Sep 18 20:16:36.401124 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:36.401104    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56800167-e66c-4b44-af4a-9404d63c662b-istio-token\") pod \"e2e-noten-a733e2-openshift-default-66b6684854-xdvs6\" (UID: \"56800167-e66c-4b44-af4a-9404d63c662b\") " pod="openshift-ingress/e2e-noten-a733e2-openshift-default-66b6684854-xdvs6"
Sep 18 20:16:36.405085 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:36.405063    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-btsr7\" (UniqueName: \"kubernetes.io/projected/56800167-e66c-4b44-af4a-9404d63c662b-kube-api-access-btsr7\") pod \"e2e-noten-a733e2-openshift-default-66b6684854-xdvs6\" (UID: \"56800167-e66c-4b44-af4a-9404d63c662b\") " pod="openshift-ingress/e2e-noten-a733e2-openshift-default-66b6684854-xdvs6"
Sep 18 20:16:36.435393 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:36.435366    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-a733e2-openshift-default-66b6684854-xdvs6"
Sep 18 20:16:36.766814 ip-10-0-140-178 kubenswrapper[2549]: W0918 20:16:36.766783    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56800167_e66c_4b44_af4a_9404d63c662b.slice/crio-50056bca80ee862a4da3319e7144cb1ecdd601d2f97eb5e1577ffb6a9ec0a390 WatchSource:0}: Error finding container 50056bca80ee862a4da3319e7144cb1ecdd601d2f97eb5e1577ffb6a9ec0a390: Status 404 returned error can't find the container with id 50056bca80ee862a4da3319e7144cb1ecdd601d2f97eb5e1577ffb6a9ec0a390
Sep 18 20:16:36.769155 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:36.769125    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:16:36.769222 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:36.769210    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:16:36.769258 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:36.769247    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:16:36.776049 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:36.776028    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-a733e2-openshift-default-66b6684854-xdvs6"]
Sep 18 20:16:36.988007 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:36.987966    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-a733e2-openshift-default-66b6684854-xdvs6" event={"ID":"56800167-e66c-4b44-af4a-9404d63c662b","Type":"ContainerStarted","Data":"fcef79d074c066cd2c499863c7ea99384449e4945dc706721b79e55926d601d1"}
Sep 18 20:16:36.988195 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:36.988026    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-a733e2-openshift-default-66b6684854-xdvs6" event={"ID":"56800167-e66c-4b44-af4a-9404d63c662b","Type":"ContainerStarted","Data":"50056bca80ee862a4da3319e7144cb1ecdd601d2f97eb5e1577ffb6a9ec0a390"}
Sep 18 20:16:37.028013 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:37.027921    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-a733e2-openshift-default-66b6684854-xdvs6" podStartSLOduration=1.027904682 podStartE2EDuration="1.027904682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:16:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:16:37.025996019 +0000 UTC m=+1181.979737187" watchObservedRunningTime="2026-09-18 20:16:37.027904682 +0000 UTC m=+1181.981645849"
Sep 18 20:16:37.436368 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:37.436330    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-a733e2-openshift-default-66b6684854-xdvs6"
Sep 18 20:16:37.438069 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:37.438035    2549 patch_prober.go:28] interesting pod/e2e-noten-a733e2-openshift-default-66b6684854-xdvs6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Sep 18 20:16:37.438201 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:37.438091    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-noten-a733e2-openshift-default-66b6684854-xdvs6" podUID="56800167-e66c-4b44-af4a-9404d63c662b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Sep 18 20:16:38.440667 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:38.440638    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-a733e2-openshift-default-66b6684854-xdvs6"
Sep 18 20:16:38.876466 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:38.876430    2549 patch_prober.go:28] interesting pod/e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Sep 18 20:16:38.876664 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:38.876487    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm" podUID="6958d670-0930-40a9-b953-db65c1a71a78" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:16:39.000248 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:39.000220    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-a733e2-openshift-default-66b6684854-xdvs6"
Sep 18 20:16:39.001231 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:39.001214    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-a733e2-openshift-default-66b6684854-xdvs6"
Sep 18 20:16:40.472825 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:40.472799    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm"
Sep 18 20:16:40.632348 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:40.632317    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6958d670-0930-40a9-b953-db65c1a71a78-istio-data\") pod \"6958d670-0930-40a9-b953-db65c1a71a78\" (UID: \"6958d670-0930-40a9-b953-db65c1a71a78\") "
Sep 18 20:16:40.632549 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:40.632358    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6958d670-0930-40a9-b953-db65c1a71a78-workload-socket\") pod \"6958d670-0930-40a9-b953-db65c1a71a78\" (UID: \"6958d670-0930-40a9-b953-db65c1a71a78\") "
Sep 18 20:16:40.632549 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:40.632432    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6958d670-0930-40a9-b953-db65c1a71a78-istio-podinfo\") pod \"6958d670-0930-40a9-b953-db65c1a71a78\" (UID: \"6958d670-0930-40a9-b953-db65c1a71a78\") "
Sep 18 20:16:40.632549 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:40.632476    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6958d670-0930-40a9-b953-db65c1a71a78-workload-certs\") pod \"6958d670-0930-40a9-b953-db65c1a71a78\" (UID: \"6958d670-0930-40a9-b953-db65c1a71a78\") "
Sep 18 20:16:40.632549 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:40.632497    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6958d670-0930-40a9-b953-db65c1a71a78-istio-envoy\") pod \"6958d670-0930-40a9-b953-db65c1a71a78\" (UID: \"6958d670-0930-40a9-b953-db65c1a71a78\") "
Sep 18 20:16:40.632549 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:40.632522    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6958d670-0930-40a9-b953-db65c1a71a78-istiod-ca-cert\") pod \"6958d670-0930-40a9-b953-db65c1a71a78\" (UID: \"6958d670-0930-40a9-b953-db65c1a71a78\") "
Sep 18 20:16:40.632549 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:40.632543    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6958d670-0930-40a9-b953-db65c1a71a78-credential-socket\") pod \"6958d670-0930-40a9-b953-db65c1a71a78\" (UID: \"6958d670-0930-40a9-b953-db65c1a71a78\") "
Sep 18 20:16:40.632872 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:40.632575    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6958d670-0930-40a9-b953-db65c1a71a78-istio-token\") pod \"6958d670-0930-40a9-b953-db65c1a71a78\" (UID: \"6958d670-0930-40a9-b953-db65c1a71a78\") "
Sep 18 20:16:40.632872 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:40.632651    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-trrvl\" (UniqueName: \"kubernetes.io/projected/6958d670-0930-40a9-b953-db65c1a71a78-kube-api-access-trrvl\") pod \"6958d670-0930-40a9-b953-db65c1a71a78\" (UID: \"6958d670-0930-40a9-b953-db65c1a71a78\") "
Sep 18 20:16:40.632872 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:40.632667    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6958d670-0930-40a9-b953-db65c1a71a78-istio-data" (OuterVolumeSpecName: "istio-data") pod "6958d670-0930-40a9-b953-db65c1a71a78" (UID: "6958d670-0930-40a9-b953-db65c1a71a78"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:16:40.632872 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:40.632793    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6958d670-0930-40a9-b953-db65c1a71a78-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6958d670-0930-40a9-b953-db65c1a71a78" (UID: "6958d670-0930-40a9-b953-db65c1a71a78"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:16:40.633098 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:40.632901    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6958d670-0930-40a9-b953-db65c1a71a78-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6958d670-0930-40a9-b953-db65c1a71a78" (UID: "6958d670-0930-40a9-b953-db65c1a71a78"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:16:40.633098 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:40.632932    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6958d670-0930-40a9-b953-db65c1a71a78-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6958d670-0930-40a9-b953-db65c1a71a78" (UID: "6958d670-0930-40a9-b953-db65c1a71a78"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:16:40.633098 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:40.633085    2549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6958d670-0930-40a9-b953-db65c1a71a78-istio-data\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:16:40.633261 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:40.633105    2549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6958d670-0930-40a9-b953-db65c1a71a78-workload-certs\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:16:40.633261 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:40.633119    2549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6958d670-0930-40a9-b953-db65c1a71a78-istiod-ca-cert\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:16:40.633261 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:40.633131    2549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6958d670-0930-40a9-b953-db65c1a71a78-credential-socket\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:16:40.633261 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:40.633232    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6958d670-0930-40a9-b953-db65c1a71a78-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6958d670-0930-40a9-b953-db65c1a71a78" (UID: "6958d670-0930-40a9-b953-db65c1a71a78"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:16:40.634685 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:40.634663    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6958d670-0930-40a9-b953-db65c1a71a78-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6958d670-0930-40a9-b953-db65c1a71a78" (UID: "6958d670-0930-40a9-b953-db65c1a71a78"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 20:16:40.634799 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:40.634788    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6958d670-0930-40a9-b953-db65c1a71a78-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6958d670-0930-40a9-b953-db65c1a71a78" (UID: "6958d670-0930-40a9-b953-db65c1a71a78"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:16:40.635032 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:40.635017    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6958d670-0930-40a9-b953-db65c1a71a78-istio-token" (OuterVolumeSpecName: "istio-token") pod "6958d670-0930-40a9-b953-db65c1a71a78" (UID: "6958d670-0930-40a9-b953-db65c1a71a78"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:16:40.635090 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:40.635069    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6958d670-0930-40a9-b953-db65c1a71a78-kube-api-access-trrvl" (OuterVolumeSpecName: "kube-api-access-trrvl") pod "6958d670-0930-40a9-b953-db65c1a71a78" (UID: "6958d670-0930-40a9-b953-db65c1a71a78"). InnerVolumeSpecName "kube-api-access-trrvl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:16:40.734344 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:40.734315    2549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6958d670-0930-40a9-b953-db65c1a71a78-istio-token\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:16:40.734344 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:40.734342    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trrvl\" (UniqueName: \"kubernetes.io/projected/6958d670-0930-40a9-b953-db65c1a71a78-kube-api-access-trrvl\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:16:40.734533 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:40.734352    2549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6958d670-0930-40a9-b953-db65c1a71a78-workload-socket\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:16:40.734533 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:40.734360    2549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6958d670-0930-40a9-b953-db65c1a71a78-istio-podinfo\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:16:40.734533 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:40.734369    2549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6958d670-0930-40a9-b953-db65c1a71a78-istio-envoy\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:16:41.015605 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:41.015510    2549 generic.go:356] "Generic (PLEG): container finished" podID="6958d670-0930-40a9-b953-db65c1a71a78" containerID="ac1a1ce6600aa1e8418c76972edbbea830f2b179d88c9a02b88d8789a80a9f92" exitCode=0
Sep 18 20:16:41.015772 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:41.015608    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm" event={"ID":"6958d670-0930-40a9-b953-db65c1a71a78","Type":"ContainerDied","Data":"ac1a1ce6600aa1e8418c76972edbbea830f2b179d88c9a02b88d8789a80a9f92"}
Sep 18 20:16:41.015772 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:41.015621    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm"
Sep 18 20:16:41.015772 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:41.015665    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm" event={"ID":"6958d670-0930-40a9-b953-db65c1a71a78","Type":"ContainerDied","Data":"481619bb9c11f576920a3a91a7c9aac0a568c41cd756faa9b6d4d6ee1b83bad9"}
Sep 18 20:16:41.015772 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:41.015694    2549 scope.go:117] "RemoveContainer" containerID="ac1a1ce6600aa1e8418c76972edbbea830f2b179d88c9a02b88d8789a80a9f92"
Sep 18 20:16:41.025695 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:41.025673    2549 scope.go:117] "RemoveContainer" containerID="876b8779c9367133d411bacd35055fcfb2b5269eebb6c9323f97a71ef94d90ce"
Sep 18 20:16:41.039193 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:41.039169    2549 scope.go:117] "RemoveContainer" containerID="ac1a1ce6600aa1e8418c76972edbbea830f2b179d88c9a02b88d8789a80a9f92"
Sep 18 20:16:41.039495 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:16:41.039474    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac1a1ce6600aa1e8418c76972edbbea830f2b179d88c9a02b88d8789a80a9f92\": container with ID starting with ac1a1ce6600aa1e8418c76972edbbea830f2b179d88c9a02b88d8789a80a9f92 not found: ID does not exist" containerID="ac1a1ce6600aa1e8418c76972edbbea830f2b179d88c9a02b88d8789a80a9f92"
Sep 18 20:16:41.039553 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:41.039506    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac1a1ce6600aa1e8418c76972edbbea830f2b179d88c9a02b88d8789a80a9f92"} err="failed to get container status \"ac1a1ce6600aa1e8418c76972edbbea830f2b179d88c9a02b88d8789a80a9f92\": rpc error: code = NotFound desc = could not find container \"ac1a1ce6600aa1e8418c76972edbbea830f2b179d88c9a02b88d8789a80a9f92\": container with ID starting with ac1a1ce6600aa1e8418c76972edbbea830f2b179d88c9a02b88d8789a80a9f92 not found: ID does not exist"
Sep 18 20:16:41.039553 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:41.039525    2549 scope.go:117] "RemoveContainer" containerID="876b8779c9367133d411bacd35055fcfb2b5269eebb6c9323f97a71ef94d90ce"
Sep 18 20:16:41.039757 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:16:41.039742    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"876b8779c9367133d411bacd35055fcfb2b5269eebb6c9323f97a71ef94d90ce\": container with ID starting with 876b8779c9367133d411bacd35055fcfb2b5269eebb6c9323f97a71ef94d90ce not found: ID does not exist" containerID="876b8779c9367133d411bacd35055fcfb2b5269eebb6c9323f97a71ef94d90ce"
Sep 18 20:16:41.039795 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:41.039761    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"876b8779c9367133d411bacd35055fcfb2b5269eebb6c9323f97a71ef94d90ce"} err="failed to get container status \"876b8779c9367133d411bacd35055fcfb2b5269eebb6c9323f97a71ef94d90ce\": rpc error: code = NotFound desc = could not find container \"876b8779c9367133d411bacd35055fcfb2b5269eebb6c9323f97a71ef94d90ce\": container with ID starting with 876b8779c9367133d411bacd35055fcfb2b5269eebb6c9323f97a71ef94d90ce not found: ID does not exist"
Sep 18 20:16:41.060574 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:41.060550    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm"]
Sep 18 20:16:41.062819 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:41.062800    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-46f024-openshift-default-b6bd79d7b-h82dm"]
Sep 18 20:16:41.524472 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:41.524357    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6958d670-0930-40a9-b953-db65c1a71a78" path="/var/lib/kubelet/pods/6958d670-0930-40a9-b953-db65c1a71a78/volumes"
Sep 18 20:16:54.623960 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:54.623920    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:16:55.523133 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:55.523106    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-scn7p_a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48/ovn-acl-logging/0.log"
Sep 18 20:16:55.527290 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:16:55.527269    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-scn7p_a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48/ovn-acl-logging/0.log"
Sep 18 20:17:04.520895 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:04.520562    2549 patch_prober.go:28] interesting pod/e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": read tcp 10.133.0.2:55138->10.133.0.68:15021: read: connection reset by peer" start-of-body=
Sep 18 20:17:04.520895 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:04.520641    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df" podUID="964d92ac-6e8a-4b90-ad38-47cfd743b76c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": read tcp 10.133.0.2:55138->10.133.0.68:15021: read: connection reset by peer"
Sep 18 20:17:05.174951 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:05.174919    2549 generic.go:356] "Generic (PLEG): container finished" podID="964d92ac-6e8a-4b90-ad38-47cfd743b76c" containerID="fba5c5d0da6af1816b9890f3f679f6c793b1d659d130304fcaa10c8497ccef4d" exitCode=0
Sep 18 20:17:05.175115 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:05.174989    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df" event={"ID":"964d92ac-6e8a-4b90-ad38-47cfd743b76c","Type":"ContainerDied","Data":"fba5c5d0da6af1816b9890f3f679f6c793b1d659d130304fcaa10c8497ccef4d"}
Sep 18 20:17:05.175427 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:05.175407    2549 scope.go:117] "RemoveContainer" containerID="fba5c5d0da6af1816b9890f3f679f6c793b1d659d130304fcaa10c8497ccef4d"
Sep 18 20:17:05.176521 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:05.176482    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:17:05.176590 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:05.176572    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:17:05.176627 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:05.176607    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:17:06.183588 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:06.183557    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df" event={"ID":"964d92ac-6e8a-4b90-ad38-47cfd743b76c","Type":"ContainerStarted","Data":"b1a229ba25900da5b8b5e63f84e9e656fa9ebd871f3fe4c185ee5ee1b4e63e07"}
Sep 18 20:17:06.517581 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:06.517505    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df"
Sep 18 20:17:06.517718 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:06.517670    2549 patch_prober.go:28] interesting pod/e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Sep 18 20:17:06.517761 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:06.517747    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df" podUID="964d92ac-6e8a-4b90-ad38-47cfd743b76c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Sep 18 20:17:07.517650 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:07.517619    2549 patch_prober.go:28] interesting pod/e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Sep 18 20:17:07.518182 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:07.517675    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df" podUID="964d92ac-6e8a-4b90-ad38-47cfd743b76c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Sep 18 20:17:08.518232 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:08.518198    2549 patch_prober.go:28] interesting pod/e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Sep 18 20:17:08.518616 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:08.518254    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df" podUID="964d92ac-6e8a-4b90-ad38-47cfd743b76c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Sep 18 20:17:09.517808 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:09.517771    2549 patch_prober.go:28] interesting pod/e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Sep 18 20:17:09.518012 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:09.517832    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df" podUID="964d92ac-6e8a-4b90-ad38-47cfd743b76c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Sep 18 20:17:10.518130 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:10.518095    2549 patch_prober.go:28] interesting pod/e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Sep 18 20:17:10.518605 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:10.518160    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df" podUID="964d92ac-6e8a-4b90-ad38-47cfd743b76c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Sep 18 20:17:11.521969 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:11.521941    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df"
Sep 18 20:17:11.522439 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:11.522291    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df"
Sep 18 20:17:11.522954 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:11.522937    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df"
Sep 18 20:17:22.250026 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:22.249987    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-aba3750c-hk78c"]
Sep 18 20:17:22.250617 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:22.250597    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6958d670-0930-40a9-b953-db65c1a71a78" containerName="istio-proxy"
Sep 18 20:17:22.250617 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:22.250618    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="6958d670-0930-40a9-b953-db65c1a71a78" containerName="istio-proxy"
Sep 18 20:17:22.250758 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:22.250746    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="6958d670-0930-40a9-b953-db65c1a71a78" containerName="istio-proxy"
Sep 18 20:17:22.250795 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:22.250762    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="6958d670-0930-40a9-b953-db65c1a71a78" containerName="istio-proxy"
Sep 18 20:17:22.255132 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:22.255111    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-aba3750c-hk78c"
Sep 18 20:17:22.258513 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:22.258495    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 20:17:22.258959 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:22.258946    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bhssz\""
Sep 18 20:17:22.278042 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:22.278015    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-aba3750c-hk78c"]
Sep 18 20:17:22.341439 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:22.341406    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-a733e2-openshift-default-66b6684854-xdvs6"]
Sep 18 20:17:22.341640 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:22.341603    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-a733e2-openshift-default-66b6684854-xdvs6" podUID="56800167-e66c-4b44-af4a-9404d63c662b" containerName="istio-proxy" containerID="cri-o://fcef79d074c066cd2c499863c7ea99384449e4945dc706721b79e55926d601d1" gracePeriod=30
Sep 18 20:17:22.341949 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:22.341919    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09454d97-93fe-4ca4-b16a-17f8997c32b4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-aba3750c-hk78c\" (UID: \"09454d97-93fe-4ca4-b16a-17f8997c32b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-aba3750c-hk78c"
Sep 18 20:17:22.443622 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:22.443581    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09454d97-93fe-4ca4-b16a-17f8997c32b4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-aba3750c-hk78c\" (UID: \"09454d97-93fe-4ca4-b16a-17f8997c32b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-aba3750c-hk78c"
Sep 18 20:17:22.444957 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:22.444932    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09454d97-93fe-4ca4-b16a-17f8997c32b4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-aba3750c-hk78c\" (UID: \"09454d97-93fe-4ca4-b16a-17f8997c32b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-aba3750c-hk78c"
Sep 18 20:17:22.566081 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:22.566002    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-aba3750c-hk78c"
Sep 18 20:17:22.849312 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:22.849274    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-aba3750c-hk78c"]
Sep 18 20:17:22.851093 ip-10-0-140-178 kubenswrapper[2549]: W0918 20:17:22.851056    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09454d97_93fe_4ca4_b16a_17f8997c32b4.slice/crio-c47764e22dda4900c047f6db0ab1eed3062df2a749031bebc6e77020c62c85d3 WatchSource:0}: Error finding container c47764e22dda4900c047f6db0ab1eed3062df2a749031bebc6e77020c62c85d3: Status 404 returned error can't find the container with id c47764e22dda4900c047f6db0ab1eed3062df2a749031bebc6e77020c62c85d3
Sep 18 20:17:23.286678 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:23.286646    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-aba3750c-hk78c" event={"ID":"09454d97-93fe-4ca4-b16a-17f8997c32b4","Type":"ContainerStarted","Data":"76c62a9017d4252d8e9e763c23ccf8fedab9c5756ee6140731ca025743af5b11"}
Sep 18 20:17:23.286994 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:23.286704    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-aba3750c-hk78c" event={"ID":"09454d97-93fe-4ca4-b16a-17f8997c32b4","Type":"ContainerStarted","Data":"c47764e22dda4900c047f6db0ab1eed3062df2a749031bebc6e77020c62c85d3"}
Sep 18 20:17:23.359833 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:23.359791    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-aba3750c-hk78c" podStartSLOduration=1.359774995 podStartE2EDuration="1.359774995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:17:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:17:23.355058818 +0000 UTC m=+1228.308799988" watchObservedRunningTime="2026-09-18 20:17:23.359774995 +0000 UTC m=+1228.313516163"
Sep 18 20:17:24.002665 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:24.002579    2549 patch_prober.go:28] interesting pod/e2e-noten-a733e2-openshift-default-66b6684854-xdvs6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 18 20:17:24.002665 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:24.002636    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-noten-a733e2-openshift-default-66b6684854-xdvs6" podUID="56800167-e66c-4b44-af4a-9404d63c662b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:17:24.188816 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:24.188776    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:17:24.294606 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:24.294515    2549 generic.go:356] "Generic (PLEG): container finished" podID="09454d97-93fe-4ca4-b16a-17f8997c32b4" containerID="76c62a9017d4252d8e9e763c23ccf8fedab9c5756ee6140731ca025743af5b11" exitCode=0
Sep 18 20:17:24.294960 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:24.294598    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-aba3750c-hk78c" event={"ID":"09454d97-93fe-4ca4-b16a-17f8997c32b4","Type":"ContainerDied","Data":"76c62a9017d4252d8e9e763c23ccf8fedab9c5756ee6140731ca025743af5b11"}
Sep 18 20:17:25.450479 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:25.450447    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-aba3750c-hk78c"
Sep 18 20:17:25.586708 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:25.586679    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09454d97-93fe-4ca4-b16a-17f8997c32b4-maas-api-service-ca\") pod \"09454d97-93fe-4ca4-b16a-17f8997c32b4\" (UID: \"09454d97-93fe-4ca4-b16a-17f8997c32b4\") "
Sep 18 20:17:25.587072 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:25.587045    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09454d97-93fe-4ca4-b16a-17f8997c32b4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "09454d97-93fe-4ca4-b16a-17f8997c32b4" (UID: "09454d97-93fe-4ca4-b16a-17f8997c32b4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:17:25.688367 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:25.688333    2549 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09454d97-93fe-4ca4-b16a-17f8997c32b4-maas-api-service-ca\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:17:26.307919 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:26.307888    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-aba3750c-hk78c"
Sep 18 20:17:26.308087 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:26.307923    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-aba3750c-hk78c" event={"ID":"09454d97-93fe-4ca4-b16a-17f8997c32b4","Type":"ContainerDied","Data":"c47764e22dda4900c047f6db0ab1eed3062df2a749031bebc6e77020c62c85d3"}
Sep 18 20:17:26.308087 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:26.307971    2549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c47764e22dda4900c047f6db0ab1eed3062df2a749031bebc6e77020c62c85d3"
Sep 18 20:17:27.602248 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:27.602195    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-a733e2-openshift-default-66b6684854-xdvs6"
Sep 18 20:17:27.605541 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:27.605523    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56800167-e66c-4b44-af4a-9404d63c662b-credential-socket\") pod \"56800167-e66c-4b44-af4a-9404d63c662b\" (UID: \"56800167-e66c-4b44-af4a-9404d63c662b\") "
Sep 18 20:17:27.605625 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:27.605576    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56800167-e66c-4b44-af4a-9404d63c662b-istiod-ca-cert\") pod \"56800167-e66c-4b44-af4a-9404d63c662b\" (UID: \"56800167-e66c-4b44-af4a-9404d63c662b\") "
Sep 18 20:17:27.605625 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:27.605598    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56800167-e66c-4b44-af4a-9404d63c662b-istio-envoy\") pod \"56800167-e66c-4b44-af4a-9404d63c662b\" (UID: \"56800167-e66c-4b44-af4a-9404d63c662b\") "
Sep 18 20:17:27.605699 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:27.605630    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56800167-e66c-4b44-af4a-9404d63c662b-workload-certs\") pod \"56800167-e66c-4b44-af4a-9404d63c662b\" (UID: \"56800167-e66c-4b44-af4a-9404d63c662b\") "
Sep 18 20:17:27.605699 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:27.605653    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56800167-e66c-4b44-af4a-9404d63c662b-istio-podinfo\") pod \"56800167-e66c-4b44-af4a-9404d63c662b\" (UID: \"56800167-e66c-4b44-af4a-9404d63c662b\") "
Sep 18 20:17:27.605699 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:27.605695    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56800167-e66c-4b44-af4a-9404d63c662b-workload-socket\") pod \"56800167-e66c-4b44-af4a-9404d63c662b\" (UID: \"56800167-e66c-4b44-af4a-9404d63c662b\") "
Sep 18 20:17:27.605836 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:27.605783    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56800167-e66c-4b44-af4a-9404d63c662b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "56800167-e66c-4b44-af4a-9404d63c662b" (UID: "56800167-e66c-4b44-af4a-9404d63c662b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:17:27.605918 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:27.605898    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56800167-e66c-4b44-af4a-9404d63c662b-istio-token\") pod \"56800167-e66c-4b44-af4a-9404d63c662b\" (UID: \"56800167-e66c-4b44-af4a-9404d63c662b\") "
Sep 18 20:17:27.605991 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:27.605948    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-btsr7\" (UniqueName: \"kubernetes.io/projected/56800167-e66c-4b44-af4a-9404d63c662b-kube-api-access-btsr7\") pod \"56800167-e66c-4b44-af4a-9404d63c662b\" (UID: \"56800167-e66c-4b44-af4a-9404d63c662b\") "
Sep 18 20:17:27.605991 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:27.605899    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56800167-e66c-4b44-af4a-9404d63c662b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "56800167-e66c-4b44-af4a-9404d63c662b" (UID: "56800167-e66c-4b44-af4a-9404d63c662b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:17:27.606096 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:27.605968    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56800167-e66c-4b44-af4a-9404d63c662b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "56800167-e66c-4b44-af4a-9404d63c662b" (UID: "56800167-e66c-4b44-af4a-9404d63c662b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:17:27.606096 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:27.606003    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56800167-e66c-4b44-af4a-9404d63c662b-istio-data\") pod \"56800167-e66c-4b44-af4a-9404d63c662b\" (UID: \"56800167-e66c-4b44-af4a-9404d63c662b\") "
Sep 18 20:17:27.606096 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:27.606064    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56800167-e66c-4b44-af4a-9404d63c662b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "56800167-e66c-4b44-af4a-9404d63c662b" (UID: "56800167-e66c-4b44-af4a-9404d63c662b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:17:27.606264 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:27.606193    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56800167-e66c-4b44-af4a-9404d63c662b-istio-data" (OuterVolumeSpecName: "istio-data") pod "56800167-e66c-4b44-af4a-9404d63c662b" (UID: "56800167-e66c-4b44-af4a-9404d63c662b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:17:27.606450 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:27.606435    2549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56800167-e66c-4b44-af4a-9404d63c662b-istiod-ca-cert\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:17:27.606537 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:27.606453    2549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56800167-e66c-4b44-af4a-9404d63c662b-workload-certs\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:17:27.606537 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:27.606462    2549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56800167-e66c-4b44-af4a-9404d63c662b-workload-socket\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:17:27.606537 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:27.606470    2549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56800167-e66c-4b44-af4a-9404d63c662b-istio-data\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:17:27.606537 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:27.606478    2549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56800167-e66c-4b44-af4a-9404d63c662b-credential-socket\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:17:27.607911 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:27.607881    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/56800167-e66c-4b44-af4a-9404d63c662b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "56800167-e66c-4b44-af4a-9404d63c662b" (UID: "56800167-e66c-4b44-af4a-9404d63c662b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 20:17:27.608035 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:27.607939    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56800167-e66c-4b44-af4a-9404d63c662b-istio-token" (OuterVolumeSpecName: "istio-token") pod "56800167-e66c-4b44-af4a-9404d63c662b" (UID: "56800167-e66c-4b44-af4a-9404d63c662b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:17:27.608108 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:27.608060    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56800167-e66c-4b44-af4a-9404d63c662b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "56800167-e66c-4b44-af4a-9404d63c662b" (UID: "56800167-e66c-4b44-af4a-9404d63c662b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:17:27.608172 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:27.608153    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56800167-e66c-4b44-af4a-9404d63c662b-kube-api-access-btsr7" (OuterVolumeSpecName: "kube-api-access-btsr7") pod "56800167-e66c-4b44-af4a-9404d63c662b" (UID: "56800167-e66c-4b44-af4a-9404d63c662b"). InnerVolumeSpecName "kube-api-access-btsr7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:17:27.707489 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:27.707457    2549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56800167-e66c-4b44-af4a-9404d63c662b-istio-envoy\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:17:27.707489 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:27.707484    2549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56800167-e66c-4b44-af4a-9404d63c662b-istio-podinfo\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:17:27.707672 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:27.707495    2549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56800167-e66c-4b44-af4a-9404d63c662b-istio-token\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:17:27.707672 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:27.707508    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btsr7\" (UniqueName: \"kubernetes.io/projected/56800167-e66c-4b44-af4a-9404d63c662b-kube-api-access-btsr7\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:17:28.322822 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:28.322787    2549 generic.go:356] "Generic (PLEG): container finished" podID="56800167-e66c-4b44-af4a-9404d63c662b" containerID="fcef79d074c066cd2c499863c7ea99384449e4945dc706721b79e55926d601d1" exitCode=0
Sep 18 20:17:28.322822 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:28.322828    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-a733e2-openshift-default-66b6684854-xdvs6" event={"ID":"56800167-e66c-4b44-af4a-9404d63c662b","Type":"ContainerDied","Data":"fcef79d074c066cd2c499863c7ea99384449e4945dc706721b79e55926d601d1"}
Sep 18 20:17:28.323074 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:28.322842    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-a733e2-openshift-default-66b6684854-xdvs6"
Sep 18 20:17:28.323074 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:28.322874    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-a733e2-openshift-default-66b6684854-xdvs6" event={"ID":"56800167-e66c-4b44-af4a-9404d63c662b","Type":"ContainerDied","Data":"50056bca80ee862a4da3319e7144cb1ecdd601d2f97eb5e1577ffb6a9ec0a390"}
Sep 18 20:17:28.323074 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:28.322897    2549 scope.go:117] "RemoveContainer" containerID="fcef79d074c066cd2c499863c7ea99384449e4945dc706721b79e55926d601d1"
Sep 18 20:17:28.332833 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:28.332817    2549 scope.go:117] "RemoveContainer" containerID="fcef79d074c066cd2c499863c7ea99384449e4945dc706721b79e55926d601d1"
Sep 18 20:17:28.333108 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:17:28.333089    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcef79d074c066cd2c499863c7ea99384449e4945dc706721b79e55926d601d1\": container with ID starting with fcef79d074c066cd2c499863c7ea99384449e4945dc706721b79e55926d601d1 not found: ID does not exist" containerID="fcef79d074c066cd2c499863c7ea99384449e4945dc706721b79e55926d601d1"
Sep 18 20:17:28.333152 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:28.333118    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcef79d074c066cd2c499863c7ea99384449e4945dc706721b79e55926d601d1"} err="failed to get container status \"fcef79d074c066cd2c499863c7ea99384449e4945dc706721b79e55926d601d1\": rpc error: code = NotFound desc = could not find container \"fcef79d074c066cd2c499863c7ea99384449e4945dc706721b79e55926d601d1\": container with ID starting with fcef79d074c066cd2c499863c7ea99384449e4945dc706721b79e55926d601d1 not found: ID does not exist"
Sep 18 20:17:28.360477 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:28.360452    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-a733e2-openshift-default-66b6684854-xdvs6"]
Sep 18 20:17:28.379558 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:28.379523    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-a733e2-openshift-default-66b6684854-xdvs6"]
Sep 18 20:17:29.521855 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:29.521815    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56800167-e66c-4b44-af4a-9404d63c662b" path="/var/lib/kubelet/pods/56800167-e66c-4b44-af4a-9404d63c662b/volumes"
Sep 18 20:17:33.591840 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:33.591808    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-aba3750c-hk78c"]
Sep 18 20:17:33.601749 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:33.601719    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-aba3750c-hk78c"]
Sep 18 20:17:35.521335 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:35.521303    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09454d97-93fe-4ca4-b16a-17f8997c32b4" path="/var/lib/kubelet/pods/09454d97-93fe-4ca4-b16a-17f8997c32b4/volumes"
Sep 18 20:17:35.589118 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:35.589086    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-c34fad/test-model-c34fad-kserve-64f6869cf5-96zdt"]
Sep 18 20:17:35.683002 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:35.682971    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3a8052-sfljb"]
Sep 18 20:17:35.683562 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:35.683541    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6958d670-0930-40a9-b953-db65c1a71a78" containerName="istio-proxy"
Sep 18 20:17:35.683562 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:35.683563    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="6958d670-0930-40a9-b953-db65c1a71a78" containerName="istio-proxy"
Sep 18 20:17:35.683740 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:35.683575    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09454d97-93fe-4ca4-b16a-17f8997c32b4" containerName="revoke-keys"
Sep 18 20:17:35.683740 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:35.683584    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="09454d97-93fe-4ca4-b16a-17f8997c32b4" containerName="revoke-keys"
Sep 18 20:17:35.683740 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:35.683612    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56800167-e66c-4b44-af4a-9404d63c662b" containerName="istio-proxy"
Sep 18 20:17:35.683740 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:35.683620    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="56800167-e66c-4b44-af4a-9404d63c662b" containerName="istio-proxy"
Sep 18 20:17:35.683740 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:35.683712    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="09454d97-93fe-4ca4-b16a-17f8997c32b4" containerName="revoke-keys"
Sep 18 20:17:35.683740 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:35.683725    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="56800167-e66c-4b44-af4a-9404d63c662b" containerName="istio-proxy"
Sep 18 20:17:35.688480 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:35.688462    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3a8052-sfljb"
Sep 18 20:17:35.705747 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:35.705724    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 20:17:35.705861 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:35.705762    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bhssz\""
Sep 18 20:17:35.756991 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:35.756957    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3a8052-sfljb"]
Sep 18 20:17:35.786536 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:35.786466    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a21196e-727f-432f-85be-a65f56ec92cf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3a8052-sfljb\" (UID: \"0a21196e-727f-432f-85be-a65f56ec92cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3a8052-sfljb"
Sep 18 20:17:35.887885 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:35.887848    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a21196e-727f-432f-85be-a65f56ec92cf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3a8052-sfljb\" (UID: \"0a21196e-727f-432f-85be-a65f56ec92cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3a8052-sfljb"
Sep 18 20:17:35.888475 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:35.888457    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a21196e-727f-432f-85be-a65f56ec92cf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3a8052-sfljb\" (UID: \"0a21196e-727f-432f-85be-a65f56ec92cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3a8052-sfljb"
Sep 18 20:17:35.998634 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:35.998606    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3a8052-sfljb"
Sep 18 20:17:36.302182 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:36.302112    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3a8052-sfljb"]
Sep 18 20:17:36.302843 ip-10-0-140-178 kubenswrapper[2549]: W0918 20:17:36.302818    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a21196e_727f_432f_85be_a65f56ec92cf.slice/crio-e7434451eb0e5da3d2f0211786709137c4ba949f5bde079b21697f5246ec5dbe WatchSource:0}: Error finding container e7434451eb0e5da3d2f0211786709137c4ba949f5bde079b21697f5246ec5dbe: Status 404 returned error can't find the container with id e7434451eb0e5da3d2f0211786709137c4ba949f5bde079b21697f5246ec5dbe
Sep 18 20:17:36.376161 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:36.376129    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3a8052-sfljb" event={"ID":"0a21196e-727f-432f-85be-a65f56ec92cf","Type":"ContainerStarted","Data":"e7434451eb0e5da3d2f0211786709137c4ba949f5bde079b21697f5246ec5dbe"}
Sep 18 20:17:37.384060 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:37.384024    2549 generic.go:356] "Generic (PLEG): container finished" podID="0a21196e-727f-432f-85be-a65f56ec92cf" containerID="5332398872c709c4074fcca2c90250ae4140755588b12553b706a5659480d0f7" exitCode=0
Sep 18 20:17:37.384458 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:37.384114    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3a8052-sfljb" event={"ID":"0a21196e-727f-432f-85be-a65f56ec92cf","Type":"ContainerDied","Data":"5332398872c709c4074fcca2c90250ae4140755588b12553b706a5659480d0f7"}
Sep 18 20:17:38.551426 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:38.551369    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3a8052-sfljb"
Sep 18 20:17:38.613324 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:38.613296    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a21196e-727f-432f-85be-a65f56ec92cf-maas-api-service-ca\") pod \"0a21196e-727f-432f-85be-a65f56ec92cf\" (UID: \"0a21196e-727f-432f-85be-a65f56ec92cf\") "
Sep 18 20:17:38.613640 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:38.613615    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a21196e-727f-432f-85be-a65f56ec92cf-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0a21196e-727f-432f-85be-a65f56ec92cf" (UID: "0a21196e-727f-432f-85be-a65f56ec92cf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:17:38.613788 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:38.613775    2549 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a21196e-727f-432f-85be-a65f56ec92cf-maas-api-service-ca\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:17:39.401617 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:39.401583    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3a8052-sfljb" event={"ID":"0a21196e-727f-432f-85be-a65f56ec92cf","Type":"ContainerDied","Data":"e7434451eb0e5da3d2f0211786709137c4ba949f5bde079b21697f5246ec5dbe"}
Sep 18 20:17:39.401617 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:39.401619    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3a8052-sfljb"
Sep 18 20:17:39.401828 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:39.401631    2549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e7434451eb0e5da3d2f0211786709137c4ba949f5bde079b21697f5246ec5dbe"
Sep 18 20:17:42.875201 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:42.875166    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df"]
Sep 18 20:17:42.875739 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:42.875373    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df" podUID="964d92ac-6e8a-4b90-ad38-47cfd743b76c" containerName="istio-proxy" containerID="cri-o://b1a229ba25900da5b8b5e63f84e9e656fa9ebd871f3fe4c185ee5ee1b4e63e07" gracePeriod=30
Sep 18 20:17:46.415487 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:46.415453    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3a8052-29829375-62q2m"]
Sep 18 20:17:46.419776 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:46.419750    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3a8052-29829375-62q2m"]
Sep 18 20:17:46.702487 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:46.702406    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3a8052-sfljb"]
Sep 18 20:17:46.707530 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:46.707504    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3a8052-sfljb"]
Sep 18 20:17:47.521182 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:47.521148    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a21196e-727f-432f-85be-a65f56ec92cf" path="/var/lib/kubelet/pods/0a21196e-727f-432f-85be-a65f56ec92cf/volumes"
Sep 18 20:17:47.521583 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:47.521469    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77597449-e2a2-4288-a23d-3610b1ac4595" path="/var/lib/kubelet/pods/77597449-e2a2-4288-a23d-3610b1ac4595/volumes"
Sep 18 20:17:48.134087 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:48.134060    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df"
Sep 18 20:17:48.203742 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:48.203665    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/964d92ac-6e8a-4b90-ad38-47cfd743b76c-istiod-ca-cert\") pod \"964d92ac-6e8a-4b90-ad38-47cfd743b76c\" (UID: \"964d92ac-6e8a-4b90-ad38-47cfd743b76c\") "
Sep 18 20:17:48.203742 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:48.203723    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/964d92ac-6e8a-4b90-ad38-47cfd743b76c-istio-envoy\") pod \"964d92ac-6e8a-4b90-ad38-47cfd743b76c\" (UID: \"964d92ac-6e8a-4b90-ad38-47cfd743b76c\") "
Sep 18 20:17:48.203963 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:48.203779    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/964d92ac-6e8a-4b90-ad38-47cfd743b76c-istio-token\") pod \"964d92ac-6e8a-4b90-ad38-47cfd743b76c\" (UID: \"964d92ac-6e8a-4b90-ad38-47cfd743b76c\") "
Sep 18 20:17:48.203963 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:48.203812    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/964d92ac-6e8a-4b90-ad38-47cfd743b76c-credential-socket\") pod \"964d92ac-6e8a-4b90-ad38-47cfd743b76c\" (UID: \"964d92ac-6e8a-4b90-ad38-47cfd743b76c\") "
Sep 18 20:17:48.203963 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:48.203844    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n8wwh\" (UniqueName: \"kubernetes.io/projected/964d92ac-6e8a-4b90-ad38-47cfd743b76c-kube-api-access-n8wwh\") pod \"964d92ac-6e8a-4b90-ad38-47cfd743b76c\" (UID: \"964d92ac-6e8a-4b90-ad38-47cfd743b76c\") "
Sep 18 20:17:48.203963 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:48.203886    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/964d92ac-6e8a-4b90-ad38-47cfd743b76c-workload-certs\") pod \"964d92ac-6e8a-4b90-ad38-47cfd743b76c\" (UID: \"964d92ac-6e8a-4b90-ad38-47cfd743b76c\") "
Sep 18 20:17:48.203963 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:48.203910    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/964d92ac-6e8a-4b90-ad38-47cfd743b76c-istio-data\") pod \"964d92ac-6e8a-4b90-ad38-47cfd743b76c\" (UID: \"964d92ac-6e8a-4b90-ad38-47cfd743b76c\") "
Sep 18 20:17:48.203963 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:48.203950    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/964d92ac-6e8a-4b90-ad38-47cfd743b76c-istio-podinfo\") pod \"964d92ac-6e8a-4b90-ad38-47cfd743b76c\" (UID: \"964d92ac-6e8a-4b90-ad38-47cfd743b76c\") "
Sep 18 20:17:48.204281 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:48.204001    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/964d92ac-6e8a-4b90-ad38-47cfd743b76c-workload-socket\") pod \"964d92ac-6e8a-4b90-ad38-47cfd743b76c\" (UID: \"964d92ac-6e8a-4b90-ad38-47cfd743b76c\") "
Sep 18 20:17:48.204281 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:48.204052    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/964d92ac-6e8a-4b90-ad38-47cfd743b76c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "964d92ac-6e8a-4b90-ad38-47cfd743b76c" (UID: "964d92ac-6e8a-4b90-ad38-47cfd743b76c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:17:48.204281 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:48.204096    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/964d92ac-6e8a-4b90-ad38-47cfd743b76c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "964d92ac-6e8a-4b90-ad38-47cfd743b76c" (UID: "964d92ac-6e8a-4b90-ad38-47cfd743b76c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:17:48.204281 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:48.204223    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/964d92ac-6e8a-4b90-ad38-47cfd743b76c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "964d92ac-6e8a-4b90-ad38-47cfd743b76c" (UID: "964d92ac-6e8a-4b90-ad38-47cfd743b76c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:17:48.204281 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:48.204250    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/964d92ac-6e8a-4b90-ad38-47cfd743b76c-istio-data" (OuterVolumeSpecName: "istio-data") pod "964d92ac-6e8a-4b90-ad38-47cfd743b76c" (UID: "964d92ac-6e8a-4b90-ad38-47cfd743b76c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:17:48.204516 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:48.204477    2549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/964d92ac-6e8a-4b90-ad38-47cfd743b76c-credential-socket\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:17:48.204516 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:48.204497    2549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/964d92ac-6e8a-4b90-ad38-47cfd743b76c-workload-certs\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:17:48.204516 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:48.204511    2549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/964d92ac-6e8a-4b90-ad38-47cfd743b76c-istio-data\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:17:48.204664 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:48.204524    2549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/964d92ac-6e8a-4b90-ad38-47cfd743b76c-istiod-ca-cert\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:17:48.204664 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:48.204617    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/964d92ac-6e8a-4b90-ad38-47cfd743b76c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "964d92ac-6e8a-4b90-ad38-47cfd743b76c" (UID: "964d92ac-6e8a-4b90-ad38-47cfd743b76c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:17:48.206020 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:48.205992    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/964d92ac-6e8a-4b90-ad38-47cfd743b76c-istio-token" (OuterVolumeSpecName: "istio-token") pod "964d92ac-6e8a-4b90-ad38-47cfd743b76c" (UID: "964d92ac-6e8a-4b90-ad38-47cfd743b76c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:17:48.206138 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:48.206015    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/964d92ac-6e8a-4b90-ad38-47cfd743b76c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "964d92ac-6e8a-4b90-ad38-47cfd743b76c" (UID: "964d92ac-6e8a-4b90-ad38-47cfd743b76c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:17:48.206414 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:48.206363    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/964d92ac-6e8a-4b90-ad38-47cfd743b76c-kube-api-access-n8wwh" (OuterVolumeSpecName: "kube-api-access-n8wwh") pod "964d92ac-6e8a-4b90-ad38-47cfd743b76c" (UID: "964d92ac-6e8a-4b90-ad38-47cfd743b76c"). InnerVolumeSpecName "kube-api-access-n8wwh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:17:48.206414 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:48.206375    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/964d92ac-6e8a-4b90-ad38-47cfd743b76c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "964d92ac-6e8a-4b90-ad38-47cfd743b76c" (UID: "964d92ac-6e8a-4b90-ad38-47cfd743b76c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 20:17:48.305684 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:48.305653    2549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/964d92ac-6e8a-4b90-ad38-47cfd743b76c-istio-token\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:17:48.305684 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:48.305679    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8wwh\" (UniqueName: \"kubernetes.io/projected/964d92ac-6e8a-4b90-ad38-47cfd743b76c-kube-api-access-n8wwh\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:17:48.305684 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:48.305691    2549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/964d92ac-6e8a-4b90-ad38-47cfd743b76c-istio-podinfo\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:17:48.305908 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:48.305699    2549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/964d92ac-6e8a-4b90-ad38-47cfd743b76c-workload-socket\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:17:48.305908 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:48.305708    2549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/964d92ac-6e8a-4b90-ad38-47cfd743b76c-istio-envoy\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:17:48.317639 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:48.317613    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:17:48.465991 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:48.465893    2549 generic.go:356] "Generic (PLEG): container finished" podID="964d92ac-6e8a-4b90-ad38-47cfd743b76c" containerID="b1a229ba25900da5b8b5e63f84e9e656fa9ebd871f3fe4c185ee5ee1b4e63e07" exitCode=0
Sep 18 20:17:48.465991 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:48.465944    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df"
Sep 18 20:17:48.466193 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:48.465984    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df" event={"ID":"964d92ac-6e8a-4b90-ad38-47cfd743b76c","Type":"ContainerDied","Data":"b1a229ba25900da5b8b5e63f84e9e656fa9ebd871f3fe4c185ee5ee1b4e63e07"}
Sep 18 20:17:48.466193 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:48.466040    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df" event={"ID":"964d92ac-6e8a-4b90-ad38-47cfd743b76c","Type":"ContainerDied","Data":"d98efadb3722733446518451e50839a313bbf600514e2f3f483314b538597ce2"}
Sep 18 20:17:48.466193 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:48.466071    2549 scope.go:117] "RemoveContainer" containerID="b1a229ba25900da5b8b5e63f84e9e656fa9ebd871f3fe4c185ee5ee1b4e63e07"
Sep 18 20:17:48.478982 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:48.478963    2549 scope.go:117] "RemoveContainer" containerID="fba5c5d0da6af1816b9890f3f679f6c793b1d659d130304fcaa10c8497ccef4d"
Sep 18 20:17:48.487528 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:48.487507    2549 scope.go:117] "RemoveContainer" containerID="b1a229ba25900da5b8b5e63f84e9e656fa9ebd871f3fe4c185ee5ee1b4e63e07"
Sep 18 20:17:48.487802 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:17:48.487784    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1a229ba25900da5b8b5e63f84e9e656fa9ebd871f3fe4c185ee5ee1b4e63e07\": container with ID starting with b1a229ba25900da5b8b5e63f84e9e656fa9ebd871f3fe4c185ee5ee1b4e63e07 not found: ID does not exist" containerID="b1a229ba25900da5b8b5e63f84e9e656fa9ebd871f3fe4c185ee5ee1b4e63e07"
Sep 18 20:17:48.487848 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:48.487810    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1a229ba25900da5b8b5e63f84e9e656fa9ebd871f3fe4c185ee5ee1b4e63e07"} err="failed to get container status \"b1a229ba25900da5b8b5e63f84e9e656fa9ebd871f3fe4c185ee5ee1b4e63e07\": rpc error: code = NotFound desc = could not find container \"b1a229ba25900da5b8b5e63f84e9e656fa9ebd871f3fe4c185ee5ee1b4e63e07\": container with ID starting with b1a229ba25900da5b8b5e63f84e9e656fa9ebd871f3fe4c185ee5ee1b4e63e07 not found: ID does not exist"
Sep 18 20:17:48.487848 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:48.487827    2549 scope.go:117] "RemoveContainer" containerID="fba5c5d0da6af1816b9890f3f679f6c793b1d659d130304fcaa10c8497ccef4d"
Sep 18 20:17:48.488097 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:17:48.488078    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fba5c5d0da6af1816b9890f3f679f6c793b1d659d130304fcaa10c8497ccef4d\": container with ID starting with fba5c5d0da6af1816b9890f3f679f6c793b1d659d130304fcaa10c8497ccef4d not found: ID does not exist" containerID="fba5c5d0da6af1816b9890f3f679f6c793b1d659d130304fcaa10c8497ccef4d"
Sep 18 20:17:48.488160 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:48.488102    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fba5c5d0da6af1816b9890f3f679f6c793b1d659d130304fcaa10c8497ccef4d"} err="failed to get container status \"fba5c5d0da6af1816b9890f3f679f6c793b1d659d130304fcaa10c8497ccef4d\": rpc error: code = NotFound desc = could not find container \"fba5c5d0da6af1816b9890f3f679f6c793b1d659d130304fcaa10c8497ccef4d\": container with ID starting with fba5c5d0da6af1816b9890f3f679f6c793b1d659d130304fcaa10c8497ccef4d not found: ID does not exist"
Sep 18 20:17:48.503104 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:48.503083    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df"]
Sep 18 20:17:48.508961 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:48.508940    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-aba3750c-openshift-default-7ffb5f9756-2w5df"]
Sep 18 20:17:49.523235 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:49.523200    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="964d92ac-6e8a-4b90-ad38-47cfd743b76c" path="/var/lib/kubelet/pods/964d92ac-6e8a-4b90-ad38-47cfd743b76c/volumes"
Sep 18 20:17:50.607984 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:50.607924    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-c34fad/test-model-c34fad-kserve-64f6869cf5-96zdt" podUID="b50ade65-c67e-43c3-8e63-7d60bcb96d0c" containerName="main" containerID="cri-o://d4ae7dfc34d4b89dd89864886c6582853b71ac97d087344c1af5c5e19695145d" gracePeriod=45
Sep 18 20:17:50.872020 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:50.871966    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c34fad/test-model-c34fad-kserve-64f6869cf5-96zdt"
Sep 18 20:17:50.927594 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:50.927563    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b50ade65-c67e-43c3-8e63-7d60bcb96d0c-tls-certs\") pod \"b50ade65-c67e-43c3-8e63-7d60bcb96d0c\" (UID: \"b50ade65-c67e-43c3-8e63-7d60bcb96d0c\") "
Sep 18 20:17:50.927751 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:50.927635    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b50ade65-c67e-43c3-8e63-7d60bcb96d0c-dshm\") pod \"b50ade65-c67e-43c3-8e63-7d60bcb96d0c\" (UID: \"b50ade65-c67e-43c3-8e63-7d60bcb96d0c\") "
Sep 18 20:17:50.927751 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:50.927686    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b50ade65-c67e-43c3-8e63-7d60bcb96d0c-home\") pod \"b50ade65-c67e-43c3-8e63-7d60bcb96d0c\" (UID: \"b50ade65-c67e-43c3-8e63-7d60bcb96d0c\") "
Sep 18 20:17:50.927751 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:50.927712    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b50ade65-c67e-43c3-8e63-7d60bcb96d0c-model-cache\") pod \"b50ade65-c67e-43c3-8e63-7d60bcb96d0c\" (UID: \"b50ade65-c67e-43c3-8e63-7d60bcb96d0c\") "
Sep 18 20:17:50.927939 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:50.927781    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b50ade65-c67e-43c3-8e63-7d60bcb96d0c-tmp-dir\") pod \"b50ade65-c67e-43c3-8e63-7d60bcb96d0c\" (UID: \"b50ade65-c67e-43c3-8e63-7d60bcb96d0c\") "
Sep 18 20:17:50.927939 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:50.927808    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x8f2w\" (UniqueName: \"kubernetes.io/projected/b50ade65-c67e-43c3-8e63-7d60bcb96d0c-kube-api-access-x8f2w\") pod \"b50ade65-c67e-43c3-8e63-7d60bcb96d0c\" (UID: \"b50ade65-c67e-43c3-8e63-7d60bcb96d0c\") "
Sep 18 20:17:50.928052 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:50.927951    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b50ade65-c67e-43c3-8e63-7d60bcb96d0c-home" (OuterVolumeSpecName: "home") pod "b50ade65-c67e-43c3-8e63-7d60bcb96d0c" (UID: "b50ade65-c67e-43c3-8e63-7d60bcb96d0c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:17:50.928163 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:50.928137    2549 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b50ade65-c67e-43c3-8e63-7d60bcb96d0c-home\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:17:50.928223 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:50.928195    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b50ade65-c67e-43c3-8e63-7d60bcb96d0c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b50ade65-c67e-43c3-8e63-7d60bcb96d0c" (UID: "b50ade65-c67e-43c3-8e63-7d60bcb96d0c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:17:50.928288 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:50.928270    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b50ade65-c67e-43c3-8e63-7d60bcb96d0c-model-cache" (OuterVolumeSpecName: "model-cache") pod "b50ade65-c67e-43c3-8e63-7d60bcb96d0c" (UID: "b50ade65-c67e-43c3-8e63-7d60bcb96d0c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:17:50.930042 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:50.930011    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b50ade65-c67e-43c3-8e63-7d60bcb96d0c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b50ade65-c67e-43c3-8e63-7d60bcb96d0c" (UID: "b50ade65-c67e-43c3-8e63-7d60bcb96d0c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:17:50.930179 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:50.930100    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b50ade65-c67e-43c3-8e63-7d60bcb96d0c-kube-api-access-x8f2w" (OuterVolumeSpecName: "kube-api-access-x8f2w") pod "b50ade65-c67e-43c3-8e63-7d60bcb96d0c" (UID: "b50ade65-c67e-43c3-8e63-7d60bcb96d0c"). InnerVolumeSpecName "kube-api-access-x8f2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:17:50.930179 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:50.930143    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b50ade65-c67e-43c3-8e63-7d60bcb96d0c-dshm" (OuterVolumeSpecName: "dshm") pod "b50ade65-c67e-43c3-8e63-7d60bcb96d0c" (UID: "b50ade65-c67e-43c3-8e63-7d60bcb96d0c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:17:51.029620 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:51.029589    2549 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b50ade65-c67e-43c3-8e63-7d60bcb96d0c-model-cache\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:17:51.029749 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:51.029630    2549 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b50ade65-c67e-43c3-8e63-7d60bcb96d0c-tmp-dir\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:17:51.029749 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:51.029639    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8f2w\" (UniqueName: \"kubernetes.io/projected/b50ade65-c67e-43c3-8e63-7d60bcb96d0c-kube-api-access-x8f2w\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:17:51.029749 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:51.029648    2549 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b50ade65-c67e-43c3-8e63-7d60bcb96d0c-tls-certs\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:17:51.029749 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:51.029657    2549 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b50ade65-c67e-43c3-8e63-7d60bcb96d0c-dshm\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:17:51.486462 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:51.486432    2549 generic.go:356] "Generic (PLEG): container finished" podID="b50ade65-c67e-43c3-8e63-7d60bcb96d0c" containerID="d4ae7dfc34d4b89dd89864886c6582853b71ac97d087344c1af5c5e19695145d" exitCode=0
Sep 18 20:17:51.486633 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:51.486473    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c34fad/test-model-c34fad-kserve-64f6869cf5-96zdt"
Sep 18 20:17:51.486633 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:51.486504    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c34fad/test-model-c34fad-kserve-64f6869cf5-96zdt" event={"ID":"b50ade65-c67e-43c3-8e63-7d60bcb96d0c","Type":"ContainerDied","Data":"d4ae7dfc34d4b89dd89864886c6582853b71ac97d087344c1af5c5e19695145d"}
Sep 18 20:17:51.486633 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:51.486536    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c34fad/test-model-c34fad-kserve-64f6869cf5-96zdt" event={"ID":"b50ade65-c67e-43c3-8e63-7d60bcb96d0c","Type":"ContainerDied","Data":"4bccc5ba8f9269c0f46db41f84e96d582e629acae03eb44d27eb202c6ecf4124"}
Sep 18 20:17:51.486633 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:51.486553    2549 scope.go:117] "RemoveContainer" containerID="d4ae7dfc34d4b89dd89864886c6582853b71ac97d087344c1af5c5e19695145d"
Sep 18 20:17:51.496401 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:51.496364    2549 scope.go:117] "RemoveContainer" containerID="d4ae7dfc34d4b89dd89864886c6582853b71ac97d087344c1af5c5e19695145d"
Sep 18 20:17:51.496670 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:17:51.496651    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4ae7dfc34d4b89dd89864886c6582853b71ac97d087344c1af5c5e19695145d\": container with ID starting with d4ae7dfc34d4b89dd89864886c6582853b71ac97d087344c1af5c5e19695145d not found: ID does not exist" containerID="d4ae7dfc34d4b89dd89864886c6582853b71ac97d087344c1af5c5e19695145d"
Sep 18 20:17:51.496718 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:51.496677    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4ae7dfc34d4b89dd89864886c6582853b71ac97d087344c1af5c5e19695145d"} err="failed to get container status \"d4ae7dfc34d4b89dd89864886c6582853b71ac97d087344c1af5c5e19695145d\": rpc error: code = NotFound desc = could not find container \"d4ae7dfc34d4b89dd89864886c6582853b71ac97d087344c1af5c5e19695145d\": container with ID starting with d4ae7dfc34d4b89dd89864886c6582853b71ac97d087344c1af5c5e19695145d not found: ID does not exist"
Sep 18 20:17:51.521641 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:51.521619    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-c34fad/test-model-c34fad-kserve-64f6869cf5-96zdt"]
Sep 18 20:17:51.522453 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:51.522433    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-c34fad/test-model-c34fad-kserve-64f6869cf5-96zdt"]
Sep 18 20:17:53.521406 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:53.521356    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b50ade65-c67e-43c3-8e63-7d60bcb96d0c" path="/var/lib/kubelet/pods/b50ade65-c67e-43c3-8e63-7d60bcb96d0c/volumes"
Sep 18 20:17:55.276596 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:55.276565    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk"]
Sep 18 20:17:55.276957 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:55.276779    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk" podUID="f3d791d0-5f42-41d0-9ca2-05d11123c178" containerName="istio-proxy" containerID="cri-o://901490bdb2fa08401b24c7ab3d701dfa5390ed1ad355f911516ca9cdd50e5ead" gracePeriod=30
Sep 18 20:17:55.635963 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:55.635934    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c34fad-pw47n"]
Sep 18 20:17:55.636325 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:55.636314    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b50ade65-c67e-43c3-8e63-7d60bcb96d0c" containerName="main"
Sep 18 20:17:55.636365 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:55.636327    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="b50ade65-c67e-43c3-8e63-7d60bcb96d0c" containerName="main"
Sep 18 20:17:55.636365 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:55.636343    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a21196e-727f-432f-85be-a65f56ec92cf" containerName="revoke-keys"
Sep 18 20:17:55.636365 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:55.636348    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a21196e-727f-432f-85be-a65f56ec92cf" containerName="revoke-keys"
Sep 18 20:17:55.636365 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:55.636359    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="964d92ac-6e8a-4b90-ad38-47cfd743b76c" containerName="istio-proxy"
Sep 18 20:17:55.636365 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:55.636365    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="964d92ac-6e8a-4b90-ad38-47cfd743b76c" containerName="istio-proxy"
Sep 18 20:17:55.636542 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:55.636373    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="964d92ac-6e8a-4b90-ad38-47cfd743b76c" containerName="istio-proxy"
Sep 18 20:17:55.636542 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:55.636378    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="964d92ac-6e8a-4b90-ad38-47cfd743b76c" containerName="istio-proxy"
Sep 18 20:17:55.636542 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:55.636461    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="964d92ac-6e8a-4b90-ad38-47cfd743b76c" containerName="istio-proxy"
Sep 18 20:17:55.636542 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:55.636469    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="b50ade65-c67e-43c3-8e63-7d60bcb96d0c" containerName="main"
Sep 18 20:17:55.636542 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:55.636477    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a21196e-727f-432f-85be-a65f56ec92cf" containerName="revoke-keys"
Sep 18 20:17:55.640806 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:55.640790    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c34fad-pw47n"
Sep 18 20:17:55.643967 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:55.643945    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bhssz\""
Sep 18 20:17:55.644257 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:55.644242    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 20:17:55.653285 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:55.653260    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c34fad-pw47n"]
Sep 18 20:17:55.770960 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:55.770925    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2488021-dda7-4c41-aa08-1b556f448a80-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c34fad-pw47n\" (UID: \"e2488021-dda7-4c41-aa08-1b556f448a80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c34fad-pw47n"
Sep 18 20:17:55.872496 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:55.872417    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2488021-dda7-4c41-aa08-1b556f448a80-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c34fad-pw47n\" (UID: \"e2488021-dda7-4c41-aa08-1b556f448a80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c34fad-pw47n"
Sep 18 20:17:55.873010 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:55.872991    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2488021-dda7-4c41-aa08-1b556f448a80-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c34fad-pw47n\" (UID: \"e2488021-dda7-4c41-aa08-1b556f448a80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c34fad-pw47n"
Sep 18 20:17:55.951829 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:55.951794    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c34fad-pw47n"
Sep 18 20:17:56.246213 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:56.246186    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c34fad-pw47n"]
Sep 18 20:17:56.248458 ip-10-0-140-178 kubenswrapper[2549]: W0918 20:17:56.248423    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2488021_dda7_4c41_aa08_1b556f448a80.slice/crio-68cfb3182a685a50eb2dd7ea20fa24b8986e7fe82e6d75c72effa37cb895fa84 WatchSource:0}: Error finding container 68cfb3182a685a50eb2dd7ea20fa24b8986e7fe82e6d75c72effa37cb895fa84: Status 404 returned error can't find the container with id 68cfb3182a685a50eb2dd7ea20fa24b8986e7fe82e6d75c72effa37cb895fa84
Sep 18 20:17:56.517314 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:56.517236    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c34fad-pw47n" event={"ID":"e2488021-dda7-4c41-aa08-1b556f448a80","Type":"ContainerStarted","Data":"68cfb3182a685a50eb2dd7ea20fa24b8986e7fe82e6d75c72effa37cb895fa84"}
Sep 18 20:17:57.524940 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:57.524907    2549 generic.go:356] "Generic (PLEG): container finished" podID="e2488021-dda7-4c41-aa08-1b556f448a80" containerID="57c31bd7525fa1dcae02846285c858deba96e873551a16273c1e0cb74d1540a8" exitCode=0
Sep 18 20:17:57.525329 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:57.524983    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c34fad-pw47n" event={"ID":"e2488021-dda7-4c41-aa08-1b556f448a80","Type":"ContainerDied","Data":"57c31bd7525fa1dcae02846285c858deba96e873551a16273c1e0cb74d1540a8"}
Sep 18 20:17:58.673412 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:58.666230    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c34fad-pw47n"
Sep 18 20:17:58.798096 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:58.798017    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2488021-dda7-4c41-aa08-1b556f448a80-maas-api-service-ca\") pod \"e2488021-dda7-4c41-aa08-1b556f448a80\" (UID: \"e2488021-dda7-4c41-aa08-1b556f448a80\") "
Sep 18 20:17:58.798358 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:58.798336    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2488021-dda7-4c41-aa08-1b556f448a80-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e2488021-dda7-4c41-aa08-1b556f448a80" (UID: "e2488021-dda7-4c41-aa08-1b556f448a80"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:17:58.899405 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:58.899354    2549 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2488021-dda7-4c41-aa08-1b556f448a80-maas-api-service-ca\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:17:59.537617 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:59.537588    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c34fad-pw47n"
Sep 18 20:17:59.537617 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:59.537611    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c34fad-pw47n" event={"ID":"e2488021-dda7-4c41-aa08-1b556f448a80","Type":"ContainerDied","Data":"68cfb3182a685a50eb2dd7ea20fa24b8986e7fe82e6d75c72effa37cb895fa84"}
Sep 18 20:17:59.537827 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:17:59.537662    2549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="68cfb3182a685a50eb2dd7ea20fa24b8986e7fe82e6d75c72effa37cb895fa84"
Sep 18 20:18:00.546748 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:00.546718    2549 generic.go:356] "Generic (PLEG): container finished" podID="f3d791d0-5f42-41d0-9ca2-05d11123c178" containerID="901490bdb2fa08401b24c7ab3d701dfa5390ed1ad355f911516ca9cdd50e5ead" exitCode=0
Sep 18 20:18:00.547076 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:00.546786    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk" event={"ID":"f3d791d0-5f42-41d0-9ca2-05d11123c178","Type":"ContainerDied","Data":"901490bdb2fa08401b24c7ab3d701dfa5390ed1ad355f911516ca9cdd50e5ead"}
Sep 18 20:18:00.547076 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:00.546872    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk" event={"ID":"f3d791d0-5f42-41d0-9ca2-05d11123c178","Type":"ContainerDied","Data":"ff5be8fd843d89b2c451174509b869c90aa2f61fdf451a122e30bff940a6898b"}
Sep 18 20:18:00.547076 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:00.546889    2549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff5be8fd843d89b2c451174509b869c90aa2f61fdf451a122e30bff940a6898b"
Sep 18 20:18:00.547076 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:00.546903    2549 scope.go:117] "RemoveContainer" containerID="f816bfed502fbeec9741f216d42b39e63756affcc99a168fdb5a0fe07e0cbe37"
Sep 18 20:18:00.562230 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:00.562204    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk"
Sep 18 20:18:00.716273 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:00.716223    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3d791d0-5f42-41d0-9ca2-05d11123c178-istio-envoy\") pod \"f3d791d0-5f42-41d0-9ca2-05d11123c178\" (UID: \"f3d791d0-5f42-41d0-9ca2-05d11123c178\") "
Sep 18 20:18:00.716273 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:00.716277    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6djjg\" (UniqueName: \"kubernetes.io/projected/f3d791d0-5f42-41d0-9ca2-05d11123c178-kube-api-access-6djjg\") pod \"f3d791d0-5f42-41d0-9ca2-05d11123c178\" (UID: \"f3d791d0-5f42-41d0-9ca2-05d11123c178\") "
Sep 18 20:18:00.716580 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:00.716309    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3d791d0-5f42-41d0-9ca2-05d11123c178-istio-token\") pod \"f3d791d0-5f42-41d0-9ca2-05d11123c178\" (UID: \"f3d791d0-5f42-41d0-9ca2-05d11123c178\") "
Sep 18 20:18:00.716580 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:00.716341    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3d791d0-5f42-41d0-9ca2-05d11123c178-istiod-ca-cert\") pod \"f3d791d0-5f42-41d0-9ca2-05d11123c178\" (UID: \"f3d791d0-5f42-41d0-9ca2-05d11123c178\") "
Sep 18 20:18:00.716580 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:00.716369    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3d791d0-5f42-41d0-9ca2-05d11123c178-credential-socket\") pod \"f3d791d0-5f42-41d0-9ca2-05d11123c178\" (UID: \"f3d791d0-5f42-41d0-9ca2-05d11123c178\") "
Sep 18 20:18:00.716580 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:00.716462    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3d791d0-5f42-41d0-9ca2-05d11123c178-workload-socket\") pod \"f3d791d0-5f42-41d0-9ca2-05d11123c178\" (UID: \"f3d791d0-5f42-41d0-9ca2-05d11123c178\") "
Sep 18 20:18:00.716580 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:00.716492    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3d791d0-5f42-41d0-9ca2-05d11123c178-istio-data\") pod \"f3d791d0-5f42-41d0-9ca2-05d11123c178\" (UID: \"f3d791d0-5f42-41d0-9ca2-05d11123c178\") "
Sep 18 20:18:00.716580 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:00.716578    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3d791d0-5f42-41d0-9ca2-05d11123c178-workload-certs\") pod \"f3d791d0-5f42-41d0-9ca2-05d11123c178\" (UID: \"f3d791d0-5f42-41d0-9ca2-05d11123c178\") "
Sep 18 20:18:00.716899 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:00.716632    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3d791d0-5f42-41d0-9ca2-05d11123c178-istio-podinfo\") pod \"f3d791d0-5f42-41d0-9ca2-05d11123c178\" (UID: \"f3d791d0-5f42-41d0-9ca2-05d11123c178\") "
Sep 18 20:18:00.716899 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:00.716774    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3d791d0-5f42-41d0-9ca2-05d11123c178-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f3d791d0-5f42-41d0-9ca2-05d11123c178" (UID: "f3d791d0-5f42-41d0-9ca2-05d11123c178"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:18:00.716899 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:00.716826    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3d791d0-5f42-41d0-9ca2-05d11123c178-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f3d791d0-5f42-41d0-9ca2-05d11123c178" (UID: "f3d791d0-5f42-41d0-9ca2-05d11123c178"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:18:00.716899 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:00.716767    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3d791d0-5f42-41d0-9ca2-05d11123c178-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f3d791d0-5f42-41d0-9ca2-05d11123c178" (UID: "f3d791d0-5f42-41d0-9ca2-05d11123c178"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:18:00.717099 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:00.716996    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3d791d0-5f42-41d0-9ca2-05d11123c178-istio-data" (OuterVolumeSpecName: "istio-data") pod "f3d791d0-5f42-41d0-9ca2-05d11123c178" (UID: "f3d791d0-5f42-41d0-9ca2-05d11123c178"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:18:00.717099 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:00.717029    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3d791d0-5f42-41d0-9ca2-05d11123c178-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f3d791d0-5f42-41d0-9ca2-05d11123c178" (UID: "f3d791d0-5f42-41d0-9ca2-05d11123c178"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:18:00.717099 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:00.717093    2549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3d791d0-5f42-41d0-9ca2-05d11123c178-istiod-ca-cert\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:18:00.717254 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:00.717112    2549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3d791d0-5f42-41d0-9ca2-05d11123c178-credential-socket\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:18:00.717254 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:00.717127    2549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3d791d0-5f42-41d0-9ca2-05d11123c178-workload-socket\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:18:00.717254 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:00.717138    2549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3d791d0-5f42-41d0-9ca2-05d11123c178-istio-data\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:18:00.718749 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:00.718719    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3d791d0-5f42-41d0-9ca2-05d11123c178-istio-token" (OuterVolumeSpecName: "istio-token") pod "f3d791d0-5f42-41d0-9ca2-05d11123c178" (UID: "f3d791d0-5f42-41d0-9ca2-05d11123c178"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:18:00.718749 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:00.718722    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3d791d0-5f42-41d0-9ca2-05d11123c178-kube-api-access-6djjg" (OuterVolumeSpecName: "kube-api-access-6djjg") pod "f3d791d0-5f42-41d0-9ca2-05d11123c178" (UID: "f3d791d0-5f42-41d0-9ca2-05d11123c178"). InnerVolumeSpecName "kube-api-access-6djjg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:18:00.718964 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:00.718947    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f3d791d0-5f42-41d0-9ca2-05d11123c178-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f3d791d0-5f42-41d0-9ca2-05d11123c178" (UID: "f3d791d0-5f42-41d0-9ca2-05d11123c178"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 20:18:00.718964 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:00.718951    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3d791d0-5f42-41d0-9ca2-05d11123c178-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f3d791d0-5f42-41d0-9ca2-05d11123c178" (UID: "f3d791d0-5f42-41d0-9ca2-05d11123c178"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:18:00.817761 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:00.817676    2549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3d791d0-5f42-41d0-9ca2-05d11123c178-workload-certs\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:18:00.817761 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:00.817703    2549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3d791d0-5f42-41d0-9ca2-05d11123c178-istio-podinfo\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:18:00.817761 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:00.817711    2549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3d791d0-5f42-41d0-9ca2-05d11123c178-istio-envoy\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:18:00.817761 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:00.817721    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6djjg\" (UniqueName: \"kubernetes.io/projected/f3d791d0-5f42-41d0-9ca2-05d11123c178-kube-api-access-6djjg\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:18:00.817761 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:00.817729    2549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3d791d0-5f42-41d0-9ca2-05d11123c178-istio-token\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:18:01.557008 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:01.556978    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk"
Sep 18 20:18:01.615847 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:01.615808    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk"]
Sep 18 20:18:01.620085 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:01.620047    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-3a8052-openshift-default-744fc9b4f6-kgknk"]
Sep 18 20:18:02.804047 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:02.804007    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:18:02.893064 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:02.893025    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x"]
Sep 18 20:18:02.893489 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:02.893474    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2488021-dda7-4c41-aa08-1b556f448a80" containerName="revoke-keys"
Sep 18 20:18:02.893544 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:02.893491    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2488021-dda7-4c41-aa08-1b556f448a80" containerName="revoke-keys"
Sep 18 20:18:02.893544 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:02.893518    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3d791d0-5f42-41d0-9ca2-05d11123c178" containerName="istio-proxy"
Sep 18 20:18:02.893544 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:02.893524    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3d791d0-5f42-41d0-9ca2-05d11123c178" containerName="istio-proxy"
Sep 18 20:18:02.893544 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:02.893535    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3d791d0-5f42-41d0-9ca2-05d11123c178" containerName="istio-proxy"
Sep 18 20:18:02.893544 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:02.893541    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3d791d0-5f42-41d0-9ca2-05d11123c178" containerName="istio-proxy"
Sep 18 20:18:02.893693 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:02.893597    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3d791d0-5f42-41d0-9ca2-05d11123c178" containerName="istio-proxy"
Sep 18 20:18:02.893693 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:02.893605    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3d791d0-5f42-41d0-9ca2-05d11123c178" containerName="istio-proxy"
Sep 18 20:18:02.893693 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:02.893612    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2488021-dda7-4c41-aa08-1b556f448a80" containerName="revoke-keys"
Sep 18 20:18:02.893693 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:02.893618    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="964d92ac-6e8a-4b90-ad38-47cfd743b76c" containerName="istio-proxy"
Sep 18 20:18:02.898133 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:02.898104    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x"
Sep 18 20:18:02.901954 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:02.901932    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-5319c6-openshift-default-dockercfg-n7vqc\""
Sep 18 20:18:02.911085 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:02.911050    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x"]
Sep 18 20:18:03.040155 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:03.040103    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a68b8bc-a551-4666-98ce-909b507edd2d-workload-certs\") pod \"e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x\" (UID: \"7a68b8bc-a551-4666-98ce-909b507edd2d\") " pod="openshift-ingress/e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x"
Sep 18 20:18:03.040155 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:03.040158    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a68b8bc-a551-4666-98ce-909b507edd2d-istio-envoy\") pod \"e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x\" (UID: \"7a68b8bc-a551-4666-98ce-909b507edd2d\") " pod="openshift-ingress/e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x"
Sep 18 20:18:03.040376 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:03.040194    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a68b8bc-a551-4666-98ce-909b507edd2d-istiod-ca-cert\") pod \"e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x\" (UID: \"7a68b8bc-a551-4666-98ce-909b507edd2d\") " pod="openshift-ingress/e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x"
Sep 18 20:18:03.040376 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:03.040266    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a68b8bc-a551-4666-98ce-909b507edd2d-credential-socket\") pod \"e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x\" (UID: \"7a68b8bc-a551-4666-98ce-909b507edd2d\") " pod="openshift-ingress/e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x"
Sep 18 20:18:03.040376 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:03.040322    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2mkw\" (UniqueName: \"kubernetes.io/projected/7a68b8bc-a551-4666-98ce-909b507edd2d-kube-api-access-d2mkw\") pod \"e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x\" (UID: \"7a68b8bc-a551-4666-98ce-909b507edd2d\") " pod="openshift-ingress/e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x"
Sep 18 20:18:03.040376 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:03.040352    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a68b8bc-a551-4666-98ce-909b507edd2d-istio-podinfo\") pod \"e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x\" (UID: \"7a68b8bc-a551-4666-98ce-909b507edd2d\") " pod="openshift-ingress/e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x"
Sep 18 20:18:03.040557 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:03.040421    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a68b8bc-a551-4666-98ce-909b507edd2d-istio-data\") pod \"e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x\" (UID: \"7a68b8bc-a551-4666-98ce-909b507edd2d\") " pod="openshift-ingress/e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x"
Sep 18 20:18:03.040557 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:03.040442    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a68b8bc-a551-4666-98ce-909b507edd2d-workload-socket\") pod \"e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x\" (UID: \"7a68b8bc-a551-4666-98ce-909b507edd2d\") " pod="openshift-ingress/e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x"
Sep 18 20:18:03.040557 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:03.040461    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a68b8bc-a551-4666-98ce-909b507edd2d-istio-token\") pod \"e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x\" (UID: \"7a68b8bc-a551-4666-98ce-909b507edd2d\") " pod="openshift-ingress/e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x"
Sep 18 20:18:03.141421 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:03.141353    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a68b8bc-a551-4666-98ce-909b507edd2d-istio-envoy\") pod \"e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x\" (UID: \"7a68b8bc-a551-4666-98ce-909b507edd2d\") " pod="openshift-ingress/e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x"
Sep 18 20:18:03.141651 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:03.141438    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a68b8bc-a551-4666-98ce-909b507edd2d-istiod-ca-cert\") pod \"e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x\" (UID: \"7a68b8bc-a551-4666-98ce-909b507edd2d\") " pod="openshift-ingress/e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x"
Sep 18 20:18:03.141651 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:03.141485    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a68b8bc-a551-4666-98ce-909b507edd2d-credential-socket\") pod \"e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x\" (UID: \"7a68b8bc-a551-4666-98ce-909b507edd2d\") " pod="openshift-ingress/e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x"
Sep 18 20:18:03.141651 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:03.141525    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2mkw\" (UniqueName: \"kubernetes.io/projected/7a68b8bc-a551-4666-98ce-909b507edd2d-kube-api-access-d2mkw\") pod \"e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x\" (UID: \"7a68b8bc-a551-4666-98ce-909b507edd2d\") " pod="openshift-ingress/e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x"
Sep 18 20:18:03.141651 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:03.141562    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a68b8bc-a551-4666-98ce-909b507edd2d-istio-podinfo\") pod \"e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x\" (UID: \"7a68b8bc-a551-4666-98ce-909b507edd2d\") " pod="openshift-ingress/e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x"
Sep 18 20:18:03.141651 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:03.141610    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a68b8bc-a551-4666-98ce-909b507edd2d-istio-data\") pod \"e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x\" (UID: \"7a68b8bc-a551-4666-98ce-909b507edd2d\") " pod="openshift-ingress/e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x"
Sep 18 20:18:03.141651 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:03.141638    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a68b8bc-a551-4666-98ce-909b507edd2d-workload-socket\") pod \"e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x\" (UID: \"7a68b8bc-a551-4666-98ce-909b507edd2d\") " pod="openshift-ingress/e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x"
Sep 18 20:18:03.141957 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:03.141666    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a68b8bc-a551-4666-98ce-909b507edd2d-istio-token\") pod \"e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x\" (UID: \"7a68b8bc-a551-4666-98ce-909b507edd2d\") " pod="openshift-ingress/e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x"
Sep 18 20:18:03.141957 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:03.141693    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a68b8bc-a551-4666-98ce-909b507edd2d-workload-certs\") pod \"e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x\" (UID: \"7a68b8bc-a551-4666-98ce-909b507edd2d\") " pod="openshift-ingress/e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x"
Sep 18 20:18:03.141957 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:03.141943    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a68b8bc-a551-4666-98ce-909b507edd2d-credential-socket\") pod \"e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x\" (UID: \"7a68b8bc-a551-4666-98ce-909b507edd2d\") " pod="openshift-ingress/e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x"
Sep 18 20:18:03.142103 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:03.142009    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a68b8bc-a551-4666-98ce-909b507edd2d-workload-certs\") pod \"e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x\" (UID: \"7a68b8bc-a551-4666-98ce-909b507edd2d\") " pod="openshift-ingress/e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x"
Sep 18 20:18:03.142103 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:03.142070    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a68b8bc-a551-4666-98ce-909b507edd2d-istio-data\") pod \"e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x\" (UID: \"7a68b8bc-a551-4666-98ce-909b507edd2d\") " pod="openshift-ingress/e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x"
Sep 18 20:18:03.142280 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:03.142262    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a68b8bc-a551-4666-98ce-909b507edd2d-workload-socket\") pod \"e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x\" (UID: \"7a68b8bc-a551-4666-98ce-909b507edd2d\") " pod="openshift-ingress/e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x"
Sep 18 20:18:03.142329 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:03.142299    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a68b8bc-a551-4666-98ce-909b507edd2d-istiod-ca-cert\") pod \"e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x\" (UID: \"7a68b8bc-a551-4666-98ce-909b507edd2d\") " pod="openshift-ingress/e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x"
Sep 18 20:18:03.143926 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:03.143903    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a68b8bc-a551-4666-98ce-909b507edd2d-istio-envoy\") pod \"e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x\" (UID: \"7a68b8bc-a551-4666-98ce-909b507edd2d\") " pod="openshift-ingress/e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x"
Sep 18 20:18:03.144074 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:03.144057    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a68b8bc-a551-4666-98ce-909b507edd2d-istio-podinfo\") pod \"e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x\" (UID: \"7a68b8bc-a551-4666-98ce-909b507edd2d\") " pod="openshift-ingress/e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x"
Sep 18 20:18:03.161553 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:03.161516    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a68b8bc-a551-4666-98ce-909b507edd2d-istio-token\") pod \"e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x\" (UID: \"7a68b8bc-a551-4666-98ce-909b507edd2d\") " pod="openshift-ingress/e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x"
Sep 18 20:18:03.161716 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:03.161629    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2mkw\" (UniqueName: \"kubernetes.io/projected/7a68b8bc-a551-4666-98ce-909b507edd2d-kube-api-access-d2mkw\") pod \"e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x\" (UID: \"7a68b8bc-a551-4666-98ce-909b507edd2d\") " pod="openshift-ingress/e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x"
Sep 18 20:18:03.209718 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:03.209670    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x"
Sep 18 20:18:03.518596 ip-10-0-140-178 kubenswrapper[2549]: W0918 20:18:03.518557    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a68b8bc_a551_4666_98ce_909b507edd2d.slice/crio-ebf8228c1383cd476dd1cb5b57588d688b0c0bb5948d7baed869a63f0114aab1 WatchSource:0}: Error finding container ebf8228c1383cd476dd1cb5b57588d688b0c0bb5948d7baed869a63f0114aab1: Status 404 returned error can't find the container with id ebf8228c1383cd476dd1cb5b57588d688b0c0bb5948d7baed869a63f0114aab1
Sep 18 20:18:03.520975 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:03.520942    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:18:03.521091 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:03.521022    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:18:03.521091 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:03.521067    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:18:03.522726 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:03.522701    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3d791d0-5f42-41d0-9ca2-05d11123c178" path="/var/lib/kubelet/pods/f3d791d0-5f42-41d0-9ca2-05d11123c178/volumes"
Sep 18 20:18:03.523014 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:03.522998    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x"]
Sep 18 20:18:03.572055 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:03.572017    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x" event={"ID":"7a68b8bc-a551-4666-98ce-909b507edd2d","Type":"ContainerStarted","Data":"ebf8228c1383cd476dd1cb5b57588d688b0c0bb5948d7baed869a63f0114aab1"}
Sep 18 20:18:04.580888 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:04.580843    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x" event={"ID":"7a68b8bc-a551-4666-98ce-909b507edd2d","Type":"ContainerStarted","Data":"e66e5f718cf77246509150fcb67777489cc6f3bbe02afa754f9a6fd19c7d8d04"}
Sep 18 20:18:04.612520 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:04.612458    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x" podStartSLOduration=2.61243877 podStartE2EDuration="2.61243877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:18:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:18:04.611163403 +0000 UTC m=+1269.564904621" watchObservedRunningTime="2026-09-18 20:18:04.61243877 +0000 UTC m=+1269.566179938"
Sep 18 20:18:05.210217 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:05.210170    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x"
Sep 18 20:18:05.216701 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:05.216653    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x"
Sep 18 20:18:05.586902 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:05.586865    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x"
Sep 18 20:18:05.588060 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:05.588036    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x"
Sep 18 20:18:13.729396 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:13.729336    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c34fad-pw47n"]
Sep 18 20:18:13.741904 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:13.741861    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c34fad-pw47n"]
Sep 18 20:18:15.522635 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:15.522599    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2488021-dda7-4c41-aa08-1b556f448a80" path="/var/lib/kubelet/pods/e2488021-dda7-4c41-aa08-1b556f448a80/volumes"
Sep 18 20:18:22.172373 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:22.172311    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:18:24.660407 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:24.660335    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm"]
Sep 18 20:18:24.661726 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:24.661666    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm" podUID="d2cb0038-fbd2-4eff-a731-227ed00eed57" containerName="istio-proxy" containerID="cri-o://43097b18c843857228ac00df6755bf900dca3812563fae55946a2d88f2f3caab" gracePeriod=30
Sep 18 20:18:25.046134 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:25.046026    2549 patch_prober.go:28] interesting pod/e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 18 20:18:25.046478 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:25.046439    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm" podUID="d2cb0038-fbd2-4eff-a731-227ed00eed57" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:18:25.992135 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:25.992095    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9"]
Sep 18 20:18:25.997704 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:25.997612    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9"
Sep 18 20:18:26.025037 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:26.024997    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-40a1a5-openshift-default-dockercfg-k4j7l\""
Sep 18 20:18:26.072413 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:26.069835    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64206b1e-8c51-4e7f-9bbe-3ea703ccb671-credential-socket\") pod \"e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9\" (UID: \"64206b1e-8c51-4e7f-9bbe-3ea703ccb671\") " pod="openshift-ingress/e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9"
Sep 18 20:18:26.072413 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:26.069896    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64206b1e-8c51-4e7f-9bbe-3ea703ccb671-istiod-ca-cert\") pod \"e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9\" (UID: \"64206b1e-8c51-4e7f-9bbe-3ea703ccb671\") " pod="openshift-ingress/e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9"
Sep 18 20:18:26.072413 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:26.069928    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64206b1e-8c51-4e7f-9bbe-3ea703ccb671-istio-token\") pod \"e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9\" (UID: \"64206b1e-8c51-4e7f-9bbe-3ea703ccb671\") " pod="openshift-ingress/e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9"
Sep 18 20:18:26.072413 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:26.069965    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64206b1e-8c51-4e7f-9bbe-3ea703ccb671-istio-data\") pod \"e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9\" (UID: \"64206b1e-8c51-4e7f-9bbe-3ea703ccb671\") " pod="openshift-ingress/e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9"
Sep 18 20:18:26.072413 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:26.070047    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64206b1e-8c51-4e7f-9bbe-3ea703ccb671-istio-envoy\") pod \"e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9\" (UID: \"64206b1e-8c51-4e7f-9bbe-3ea703ccb671\") " pod="openshift-ingress/e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9"
Sep 18 20:18:26.072413 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:26.070095    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64206b1e-8c51-4e7f-9bbe-3ea703ccb671-workload-socket\") pod \"e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9\" (UID: \"64206b1e-8c51-4e7f-9bbe-3ea703ccb671\") " pod="openshift-ingress/e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9"
Sep 18 20:18:26.072413 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:26.070180    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkxzj\" (UniqueName: \"kubernetes.io/projected/64206b1e-8c51-4e7f-9bbe-3ea703ccb671-kube-api-access-fkxzj\") pod \"e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9\" (UID: \"64206b1e-8c51-4e7f-9bbe-3ea703ccb671\") " pod="openshift-ingress/e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9"
Sep 18 20:18:26.072413 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:26.070234    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64206b1e-8c51-4e7f-9bbe-3ea703ccb671-workload-certs\") pod \"e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9\" (UID: \"64206b1e-8c51-4e7f-9bbe-3ea703ccb671\") " pod="openshift-ingress/e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9"
Sep 18 20:18:26.072413 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:26.070299    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64206b1e-8c51-4e7f-9bbe-3ea703ccb671-istio-podinfo\") pod \"e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9\" (UID: \"64206b1e-8c51-4e7f-9bbe-3ea703ccb671\") " pod="openshift-ingress/e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9"
Sep 18 20:18:26.123921 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:26.123873    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9"]
Sep 18 20:18:26.172758 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:26.172213    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64206b1e-8c51-4e7f-9bbe-3ea703ccb671-workload-certs\") pod \"e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9\" (UID: \"64206b1e-8c51-4e7f-9bbe-3ea703ccb671\") " pod="openshift-ingress/e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9"
Sep 18 20:18:26.173060 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:26.172710    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64206b1e-8c51-4e7f-9bbe-3ea703ccb671-workload-certs\") pod \"e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9\" (UID: \"64206b1e-8c51-4e7f-9bbe-3ea703ccb671\") " pod="openshift-ingress/e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9"
Sep 18 20:18:26.173236 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:26.173216    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64206b1e-8c51-4e7f-9bbe-3ea703ccb671-istio-podinfo\") pod \"e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9\" (UID: \"64206b1e-8c51-4e7f-9bbe-3ea703ccb671\") " pod="openshift-ingress/e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9"
Sep 18 20:18:26.173418 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:26.173375    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64206b1e-8c51-4e7f-9bbe-3ea703ccb671-credential-socket\") pod \"e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9\" (UID: \"64206b1e-8c51-4e7f-9bbe-3ea703ccb671\") " pod="openshift-ingress/e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9"
Sep 18 20:18:26.173557 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:26.173540    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64206b1e-8c51-4e7f-9bbe-3ea703ccb671-istiod-ca-cert\") pod \"e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9\" (UID: \"64206b1e-8c51-4e7f-9bbe-3ea703ccb671\") " pod="openshift-ingress/e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9"
Sep 18 20:18:26.173676 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:26.173661    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64206b1e-8c51-4e7f-9bbe-3ea703ccb671-istio-token\") pod \"e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9\" (UID: \"64206b1e-8c51-4e7f-9bbe-3ea703ccb671\") " pod="openshift-ingress/e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9"
Sep 18 20:18:26.173786 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:26.173772    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64206b1e-8c51-4e7f-9bbe-3ea703ccb671-istio-data\") pod \"e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9\" (UID: \"64206b1e-8c51-4e7f-9bbe-3ea703ccb671\") " pod="openshift-ingress/e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9"
Sep 18 20:18:26.173947 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:26.173929    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64206b1e-8c51-4e7f-9bbe-3ea703ccb671-istio-envoy\") pod \"e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9\" (UID: \"64206b1e-8c51-4e7f-9bbe-3ea703ccb671\") " pod="openshift-ingress/e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9"
Sep 18 20:18:26.175551 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:26.175512    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64206b1e-8c51-4e7f-9bbe-3ea703ccb671-workload-socket\") pod \"e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9\" (UID: \"64206b1e-8c51-4e7f-9bbe-3ea703ccb671\") " pod="openshift-ingress/e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9"
Sep 18 20:18:26.175679 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:26.175586    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkxzj\" (UniqueName: \"kubernetes.io/projected/64206b1e-8c51-4e7f-9bbe-3ea703ccb671-kube-api-access-fkxzj\") pod \"e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9\" (UID: \"64206b1e-8c51-4e7f-9bbe-3ea703ccb671\") " pod="openshift-ingress/e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9"
Sep 18 20:18:26.175679 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:26.175619    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64206b1e-8c51-4e7f-9bbe-3ea703ccb671-istio-data\") pod \"e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9\" (UID: \"64206b1e-8c51-4e7f-9bbe-3ea703ccb671\") " pod="openshift-ingress/e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9"
Sep 18 20:18:26.175679 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:26.174815    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64206b1e-8c51-4e7f-9bbe-3ea703ccb671-istiod-ca-cert\") pod \"e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9\" (UID: \"64206b1e-8c51-4e7f-9bbe-3ea703ccb671\") " pod="openshift-ingress/e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9"
Sep 18 20:18:26.175679 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:26.175649    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64206b1e-8c51-4e7f-9bbe-3ea703ccb671-credential-socket\") pod \"e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9\" (UID: \"64206b1e-8c51-4e7f-9bbe-3ea703ccb671\") " pod="openshift-ingress/e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9"
Sep 18 20:18:26.175985 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:26.175952    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64206b1e-8c51-4e7f-9bbe-3ea703ccb671-workload-socket\") pod \"e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9\" (UID: \"64206b1e-8c51-4e7f-9bbe-3ea703ccb671\") " pod="openshift-ingress/e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9"
Sep 18 20:18:26.178137 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:26.178111    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64206b1e-8c51-4e7f-9bbe-3ea703ccb671-istio-envoy\") pod \"e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9\" (UID: \"64206b1e-8c51-4e7f-9bbe-3ea703ccb671\") " pod="openshift-ingress/e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9"
Sep 18 20:18:26.178486 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:26.178471    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64206b1e-8c51-4e7f-9bbe-3ea703ccb671-istio-podinfo\") pod \"e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9\" (UID: \"64206b1e-8c51-4e7f-9bbe-3ea703ccb671\") " pod="openshift-ingress/e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9"
Sep 18 20:18:26.220794 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:26.220761    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64206b1e-8c51-4e7f-9bbe-3ea703ccb671-istio-token\") pod \"e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9\" (UID: \"64206b1e-8c51-4e7f-9bbe-3ea703ccb671\") " pod="openshift-ingress/e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9"
Sep 18 20:18:26.226611 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:26.226581    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkxzj\" (UniqueName: \"kubernetes.io/projected/64206b1e-8c51-4e7f-9bbe-3ea703ccb671-kube-api-access-fkxzj\") pod \"e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9\" (UID: \"64206b1e-8c51-4e7f-9bbe-3ea703ccb671\") " pod="openshift-ingress/e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9"
Sep 18 20:18:26.310197 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:26.310082    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9"
Sep 18 20:18:26.776151 ip-10-0-140-178 kubenswrapper[2549]: W0918 20:18:26.776108    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64206b1e_8c51_4e7f_9bbe_3ea703ccb671.slice/crio-5515b8606888020a9aabea91f687df10cf459c100f43c3bfc9bf9fe891bb7afd WatchSource:0}: Error finding container 5515b8606888020a9aabea91f687df10cf459c100f43c3bfc9bf9fe891bb7afd: Status 404 returned error can't find the container with id 5515b8606888020a9aabea91f687df10cf459c100f43c3bfc9bf9fe891bb7afd
Sep 18 20:18:26.776632 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:26.776605    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9"]
Sep 18 20:18:26.779615 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:26.779576    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:18:26.779721 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:26.779649    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:18:26.779721 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:26.779681    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:18:27.729234 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:27.729191    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g"]
Sep 18 20:18:27.738092 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:27.733687    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g"
Sep 18 20:18:27.748683 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:27.748638    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9" event={"ID":"64206b1e-8c51-4e7f-9bbe-3ea703ccb671","Type":"ContainerStarted","Data":"8b23930ee8074f5d24393a3249ad26e96ba0db34880259b5f3ae9630e9e82bd3"}
Sep 18 20:18:27.748975 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:27.748949    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9" event={"ID":"64206b1e-8c51-4e7f-9bbe-3ea703ccb671","Type":"ContainerStarted","Data":"5515b8606888020a9aabea91f687df10cf459c100f43c3bfc9bf9fe891bb7afd"}
Sep 18 20:18:27.771402 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:27.771346    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-c5206e-openshift-default-dockercfg-xncjz\""
Sep 18 20:18:27.794047 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:27.794005    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a390c820-232c-4b8c-b4e4-711699f90d2b-istio-podinfo\") pod \"e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g\" (UID: \"a390c820-232c-4b8c-b4e4-711699f90d2b\") " pod="openshift-ingress/e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g"
Sep 18 20:18:27.794269 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:27.794187    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a390c820-232c-4b8c-b4e4-711699f90d2b-workload-socket\") pod \"e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g\" (UID: \"a390c820-232c-4b8c-b4e4-711699f90d2b\") " pod="openshift-ingress/e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g"
Sep 18 20:18:27.794269 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:27.794232    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a390c820-232c-4b8c-b4e4-711699f90d2b-credential-socket\") pod \"e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g\" (UID: \"a390c820-232c-4b8c-b4e4-711699f90d2b\") " pod="openshift-ingress/e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g"
Sep 18 20:18:27.794435 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:27.794300    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plwgb\" (UniqueName: \"kubernetes.io/projected/a390c820-232c-4b8c-b4e4-711699f90d2b-kube-api-access-plwgb\") pod \"e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g\" (UID: \"a390c820-232c-4b8c-b4e4-711699f90d2b\") " pod="openshift-ingress/e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g"
Sep 18 20:18:27.794435 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:27.794323    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a390c820-232c-4b8c-b4e4-711699f90d2b-istiod-ca-cert\") pod \"e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g\" (UID: \"a390c820-232c-4b8c-b4e4-711699f90d2b\") " pod="openshift-ingress/e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g"
Sep 18 20:18:27.794435 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:27.794348    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a390c820-232c-4b8c-b4e4-711699f90d2b-istio-token\") pod \"e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g\" (UID: \"a390c820-232c-4b8c-b4e4-711699f90d2b\") " pod="openshift-ingress/e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g"
Sep 18 20:18:27.794435 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:27.794379    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a390c820-232c-4b8c-b4e4-711699f90d2b-workload-certs\") pod \"e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g\" (UID: \"a390c820-232c-4b8c-b4e4-711699f90d2b\") " pod="openshift-ingress/e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g"
Sep 18 20:18:27.794662 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:27.794560    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a390c820-232c-4b8c-b4e4-711699f90d2b-istio-envoy\") pod \"e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g\" (UID: \"a390c820-232c-4b8c-b4e4-711699f90d2b\") " pod="openshift-ingress/e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g"
Sep 18 20:18:27.794662 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:27.794602    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a390c820-232c-4b8c-b4e4-711699f90d2b-istio-data\") pod \"e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g\" (UID: \"a390c820-232c-4b8c-b4e4-711699f90d2b\") " pod="openshift-ingress/e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g"
Sep 18 20:18:27.882968 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:27.882902    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9" podStartSLOduration=2.882880739 podStartE2EDuration="2.882880739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:18:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:18:27.880072135 +0000 UTC m=+1292.833813303" watchObservedRunningTime="2026-09-18 20:18:27.882880739 +0000 UTC m=+1292.836621906"
Sep 18 20:18:27.895807 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:27.895771    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a390c820-232c-4b8c-b4e4-711699f90d2b-workload-socket\") pod \"e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g\" (UID: \"a390c820-232c-4b8c-b4e4-711699f90d2b\") " pod="openshift-ingress/e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g"
Sep 18 20:18:27.895933 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:27.895821    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a390c820-232c-4b8c-b4e4-711699f90d2b-credential-socket\") pod \"e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g\" (UID: \"a390c820-232c-4b8c-b4e4-711699f90d2b\") " pod="openshift-ingress/e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g"
Sep 18 20:18:27.895983 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:27.895939    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plwgb\" (UniqueName: \"kubernetes.io/projected/a390c820-232c-4b8c-b4e4-711699f90d2b-kube-api-access-plwgb\") pod \"e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g\" (UID: \"a390c820-232c-4b8c-b4e4-711699f90d2b\") " pod="openshift-ingress/e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g"
Sep 18 20:18:27.896032 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:27.895979    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a390c820-232c-4b8c-b4e4-711699f90d2b-istiod-ca-cert\") pod \"e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g\" (UID: \"a390c820-232c-4b8c-b4e4-711699f90d2b\") " pod="openshift-ingress/e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g"
Sep 18 20:18:27.896032 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:27.896003    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a390c820-232c-4b8c-b4e4-711699f90d2b-istio-token\") pod \"e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g\" (UID: \"a390c820-232c-4b8c-b4e4-711699f90d2b\") " pod="openshift-ingress/e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g"
Sep 18 20:18:27.896032 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:27.896030    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a390c820-232c-4b8c-b4e4-711699f90d2b-workload-certs\") pod \"e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g\" (UID: \"a390c820-232c-4b8c-b4e4-711699f90d2b\") " pod="openshift-ingress/e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g"
Sep 18 20:18:27.896194 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:27.896111    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a390c820-232c-4b8c-b4e4-711699f90d2b-istio-envoy\") pod \"e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g\" (UID: \"a390c820-232c-4b8c-b4e4-711699f90d2b\") " pod="openshift-ingress/e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g"
Sep 18 20:18:27.896194 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:27.896133    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a390c820-232c-4b8c-b4e4-711699f90d2b-istio-data\") pod \"e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g\" (UID: \"a390c820-232c-4b8c-b4e4-711699f90d2b\") " pod="openshift-ingress/e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g"
Sep 18 20:18:27.896194 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:27.896161    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a390c820-232c-4b8c-b4e4-711699f90d2b-istio-podinfo\") pod \"e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g\" (UID: \"a390c820-232c-4b8c-b4e4-711699f90d2b\") " pod="openshift-ingress/e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g"
Sep 18 20:18:27.896354 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:27.896213    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a390c820-232c-4b8c-b4e4-711699f90d2b-credential-socket\") pod \"e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g\" (UID: \"a390c820-232c-4b8c-b4e4-711699f90d2b\") " pod="openshift-ingress/e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g"
Sep 18 20:18:27.896354 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:27.896267    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a390c820-232c-4b8c-b4e4-711699f90d2b-workload-socket\") pod \"e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g\" (UID: \"a390c820-232c-4b8c-b4e4-711699f90d2b\") " pod="openshift-ingress/e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g"
Sep 18 20:18:27.896523 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:27.896495    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a390c820-232c-4b8c-b4e4-711699f90d2b-istio-data\") pod \"e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g\" (UID: \"a390c820-232c-4b8c-b4e4-711699f90d2b\") " pod="openshift-ingress/e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g"
Sep 18 20:18:27.896623 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:27.896603    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a390c820-232c-4b8c-b4e4-711699f90d2b-workload-certs\") pod \"e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g\" (UID: \"a390c820-232c-4b8c-b4e4-711699f90d2b\") " pod="openshift-ingress/e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g"
Sep 18 20:18:27.896785 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:27.896764    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a390c820-232c-4b8c-b4e4-711699f90d2b-istiod-ca-cert\") pod \"e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g\" (UID: \"a390c820-232c-4b8c-b4e4-711699f90d2b\") " pod="openshift-ingress/e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g"
Sep 18 20:18:27.898570 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:27.898540    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a390c820-232c-4b8c-b4e4-711699f90d2b-istio-envoy\") pod \"e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g\" (UID: \"a390c820-232c-4b8c-b4e4-711699f90d2b\") " pod="openshift-ingress/e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g"
Sep 18 20:18:27.899016 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:27.898995    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a390c820-232c-4b8c-b4e4-711699f90d2b-istio-podinfo\") pod \"e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g\" (UID: \"a390c820-232c-4b8c-b4e4-711699f90d2b\") " pod="openshift-ingress/e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g"
Sep 18 20:18:27.920859 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:27.920812    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g"]
Sep 18 20:18:27.933196 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:27.933146    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-plwgb\" (UniqueName: \"kubernetes.io/projected/a390c820-232c-4b8c-b4e4-711699f90d2b-kube-api-access-plwgb\") pod \"e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g\" (UID: \"a390c820-232c-4b8c-b4e4-711699f90d2b\") " pod="openshift-ingress/e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g"
Sep 18 20:18:27.966408 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:27.965031    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a390c820-232c-4b8c-b4e4-711699f90d2b-istio-token\") pod \"e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g\" (UID: \"a390c820-232c-4b8c-b4e4-711699f90d2b\") " pod="openshift-ingress/e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g"
Sep 18 20:18:28.075947 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:28.075901    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g"
Sep 18 20:18:28.310843 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:28.310789    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9"
Sep 18 20:18:28.316906 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:28.316874    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9"
Sep 18 20:18:28.396302 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:28.396275    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g"]
Sep 18 20:18:28.398636 ip-10-0-140-178 kubenswrapper[2549]: W0918 20:18:28.398596    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda390c820_232c_4b8c_b4e4_711699f90d2b.slice/crio-e8065335562dd04210158558e6ca20a11bd0a1995669948e3d72ca85db731a8a WatchSource:0}: Error finding container e8065335562dd04210158558e6ca20a11bd0a1995669948e3d72ca85db731a8a: Status 404 returned error can't find the container with id e8065335562dd04210158558e6ca20a11bd0a1995669948e3d72ca85db731a8a
Sep 18 20:18:28.401016 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:28.400985    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:18:28.401124 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:28.401057    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:18:28.401124 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:28.401087    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:18:28.764933 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:28.764816    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g" event={"ID":"a390c820-232c-4b8c-b4e4-711699f90d2b","Type":"ContainerStarted","Data":"7010c2fd412a05d5026549528e8b7e2cc026fcefacc2d6dec96f5c626d750063"}
Sep 18 20:18:28.764933 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:28.764894    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g" event={"ID":"a390c820-232c-4b8c-b4e4-711699f90d2b","Type":"ContainerStarted","Data":"e8065335562dd04210158558e6ca20a11bd0a1995669948e3d72ca85db731a8a"}
Sep 18 20:18:28.764933 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:28.764926    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9"
Sep 18 20:18:28.769299 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:28.769262    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9"
Sep 18 20:18:28.799910 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:28.799850    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g" podStartSLOduration=1.7998294129999999 podStartE2EDuration="1.799829413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:18:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:18:28.797040999 +0000 UTC m=+1293.750782168" watchObservedRunningTime="2026-09-18 20:18:28.799829413 +0000 UTC m=+1293.753570583"
Sep 18 20:18:29.076132 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:29.076039    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g"
Sep 18 20:18:29.082278 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:29.082238    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g"
Sep 18 20:18:29.771679 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:29.771600    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g"
Sep 18 20:18:29.772516 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:29.772490    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g"
Sep 18 20:18:29.947661 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:29.947626    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm"
Sep 18 20:18:30.017135 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:30.017085    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2cb0038-fbd2-4eff-a731-227ed00eed57-istio-token\") pod \"d2cb0038-fbd2-4eff-a731-227ed00eed57\" (UID: \"d2cb0038-fbd2-4eff-a731-227ed00eed57\") "
Sep 18 20:18:30.017354 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:30.017151    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2cb0038-fbd2-4eff-a731-227ed00eed57-istiod-ca-cert\") pod \"d2cb0038-fbd2-4eff-a731-227ed00eed57\" (UID: \"d2cb0038-fbd2-4eff-a731-227ed00eed57\") "
Sep 18 20:18:30.017354 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:30.017226    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2cb0038-fbd2-4eff-a731-227ed00eed57-credential-socket\") pod \"d2cb0038-fbd2-4eff-a731-227ed00eed57\" (UID: \"d2cb0038-fbd2-4eff-a731-227ed00eed57\") "
Sep 18 20:18:30.017354 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:30.017268    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2cb0038-fbd2-4eff-a731-227ed00eed57-workload-socket\") pod \"d2cb0038-fbd2-4eff-a731-227ed00eed57\" (UID: \"d2cb0038-fbd2-4eff-a731-227ed00eed57\") "
Sep 18 20:18:30.017354 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:30.017315    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2cb0038-fbd2-4eff-a731-227ed00eed57-istio-podinfo\") pod \"d2cb0038-fbd2-4eff-a731-227ed00eed57\" (UID: \"d2cb0038-fbd2-4eff-a731-227ed00eed57\") "
Sep 18 20:18:30.017668 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:30.017377    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fhnj4\" (UniqueName: \"kubernetes.io/projected/d2cb0038-fbd2-4eff-a731-227ed00eed57-kube-api-access-fhnj4\") pod \"d2cb0038-fbd2-4eff-a731-227ed00eed57\" (UID: \"d2cb0038-fbd2-4eff-a731-227ed00eed57\") "
Sep 18 20:18:30.017668 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:30.017446    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2cb0038-fbd2-4eff-a731-227ed00eed57-istio-envoy\") pod \"d2cb0038-fbd2-4eff-a731-227ed00eed57\" (UID: \"d2cb0038-fbd2-4eff-a731-227ed00eed57\") "
Sep 18 20:18:30.017668 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:30.017494    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2cb0038-fbd2-4eff-a731-227ed00eed57-workload-certs\") pod \"d2cb0038-fbd2-4eff-a731-227ed00eed57\" (UID: \"d2cb0038-fbd2-4eff-a731-227ed00eed57\") "
Sep 18 20:18:30.017668 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:30.017526    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2cb0038-fbd2-4eff-a731-227ed00eed57-istio-data\") pod \"d2cb0038-fbd2-4eff-a731-227ed00eed57\" (UID: \"d2cb0038-fbd2-4eff-a731-227ed00eed57\") "
Sep 18 20:18:30.017668 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:30.017533    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2cb0038-fbd2-4eff-a731-227ed00eed57-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d2cb0038-fbd2-4eff-a731-227ed00eed57" (UID: "d2cb0038-fbd2-4eff-a731-227ed00eed57"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:18:30.017668 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:30.017605    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2cb0038-fbd2-4eff-a731-227ed00eed57-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d2cb0038-fbd2-4eff-a731-227ed00eed57" (UID: "d2cb0038-fbd2-4eff-a731-227ed00eed57"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:18:30.020074 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:30.017851    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2cb0038-fbd2-4eff-a731-227ed00eed57-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d2cb0038-fbd2-4eff-a731-227ed00eed57" (UID: "d2cb0038-fbd2-4eff-a731-227ed00eed57"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:18:30.020074 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:30.018102    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2cb0038-fbd2-4eff-a731-227ed00eed57-istio-data" (OuterVolumeSpecName: "istio-data") pod "d2cb0038-fbd2-4eff-a731-227ed00eed57" (UID: "d2cb0038-fbd2-4eff-a731-227ed00eed57"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:18:30.020074 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:30.018441    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2cb0038-fbd2-4eff-a731-227ed00eed57-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d2cb0038-fbd2-4eff-a731-227ed00eed57" (UID: "d2cb0038-fbd2-4eff-a731-227ed00eed57"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:18:30.020413 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:30.020359    2549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2cb0038-fbd2-4eff-a731-227ed00eed57-credential-socket\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:18:30.020413 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:30.020410    2549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2cb0038-fbd2-4eff-a731-227ed00eed57-workload-socket\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:18:30.020694 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:30.020426    2549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2cb0038-fbd2-4eff-a731-227ed00eed57-workload-certs\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:18:30.020694 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:30.020439    2549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2cb0038-fbd2-4eff-a731-227ed00eed57-istio-data\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:18:30.020694 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:30.020451    2549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2cb0038-fbd2-4eff-a731-227ed00eed57-istiod-ca-cert\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:18:30.022255 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:30.022218    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2cb0038-fbd2-4eff-a731-227ed00eed57-istio-token" (OuterVolumeSpecName: "istio-token") pod "d2cb0038-fbd2-4eff-a731-227ed00eed57" (UID: "d2cb0038-fbd2-4eff-a731-227ed00eed57"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:18:30.023495 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:30.023434    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2cb0038-fbd2-4eff-a731-227ed00eed57-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d2cb0038-fbd2-4eff-a731-227ed00eed57" (UID: "d2cb0038-fbd2-4eff-a731-227ed00eed57"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:18:30.023578 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:30.023501    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2cb0038-fbd2-4eff-a731-227ed00eed57-kube-api-access-fhnj4" (OuterVolumeSpecName: "kube-api-access-fhnj4") pod "d2cb0038-fbd2-4eff-a731-227ed00eed57" (UID: "d2cb0038-fbd2-4eff-a731-227ed00eed57"). InnerVolumeSpecName "kube-api-access-fhnj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:18:30.023578 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:30.023528    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d2cb0038-fbd2-4eff-a731-227ed00eed57-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d2cb0038-fbd2-4eff-a731-227ed00eed57" (UID: "d2cb0038-fbd2-4eff-a731-227ed00eed57"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 20:18:30.121642 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:30.121532    2549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2cb0038-fbd2-4eff-a731-227ed00eed57-istio-token\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:18:30.121642 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:30.121571    2549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2cb0038-fbd2-4eff-a731-227ed00eed57-istio-podinfo\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:18:30.121642 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:30.121584    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhnj4\" (UniqueName: \"kubernetes.io/projected/d2cb0038-fbd2-4eff-a731-227ed00eed57-kube-api-access-fhnj4\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:18:30.121642 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:30.121597    2549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2cb0038-fbd2-4eff-a731-227ed00eed57-istio-envoy\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:18:30.779845 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:30.779812    2549 generic.go:356] "Generic (PLEG): container finished" podID="d2cb0038-fbd2-4eff-a731-227ed00eed57" containerID="43097b18c843857228ac00df6755bf900dca3812563fae55946a2d88f2f3caab" exitCode=0
Sep 18 20:18:30.780328 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:30.779864    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm"
Sep 18 20:18:30.780328 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:30.779897    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm" event={"ID":"d2cb0038-fbd2-4eff-a731-227ed00eed57","Type":"ContainerDied","Data":"43097b18c843857228ac00df6755bf900dca3812563fae55946a2d88f2f3caab"}
Sep 18 20:18:30.780328 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:30.779957    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm" event={"ID":"d2cb0038-fbd2-4eff-a731-227ed00eed57","Type":"ContainerDied","Data":"67cbb3a04ba98372260ea8ec3e6b79741633246debfa21c35dea85d4407cffe8"}
Sep 18 20:18:30.780328 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:30.779990    2549 scope.go:117] "RemoveContainer" containerID="43097b18c843857228ac00df6755bf900dca3812563fae55946a2d88f2f3caab"
Sep 18 20:18:30.791236 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:30.791078    2549 scope.go:117] "RemoveContainer" containerID="474e7a33c8ee247f675562129318c3a865174d5d26ecbd96ea8cf2f6b027af8b"
Sep 18 20:18:30.801308 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:30.801286    2549 scope.go:117] "RemoveContainer" containerID="43097b18c843857228ac00df6755bf900dca3812563fae55946a2d88f2f3caab"
Sep 18 20:18:30.801711 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:18:30.801687    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43097b18c843857228ac00df6755bf900dca3812563fae55946a2d88f2f3caab\": container with ID starting with 43097b18c843857228ac00df6755bf900dca3812563fae55946a2d88f2f3caab not found: ID does not exist" containerID="43097b18c843857228ac00df6755bf900dca3812563fae55946a2d88f2f3caab"
Sep 18 20:18:30.801804 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:30.801726    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43097b18c843857228ac00df6755bf900dca3812563fae55946a2d88f2f3caab"} err="failed to get container status \"43097b18c843857228ac00df6755bf900dca3812563fae55946a2d88f2f3caab\": rpc error: code = NotFound desc = could not find container \"43097b18c843857228ac00df6755bf900dca3812563fae55946a2d88f2f3caab\": container with ID starting with 43097b18c843857228ac00df6755bf900dca3812563fae55946a2d88f2f3caab not found: ID does not exist"
Sep 18 20:18:30.801804 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:30.801753    2549 scope.go:117] "RemoveContainer" containerID="474e7a33c8ee247f675562129318c3a865174d5d26ecbd96ea8cf2f6b027af8b"
Sep 18 20:18:30.802064 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:18:30.802037    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"474e7a33c8ee247f675562129318c3a865174d5d26ecbd96ea8cf2f6b027af8b\": container with ID starting with 474e7a33c8ee247f675562129318c3a865174d5d26ecbd96ea8cf2f6b027af8b not found: ID does not exist" containerID="474e7a33c8ee247f675562129318c3a865174d5d26ecbd96ea8cf2f6b027af8b"
Sep 18 20:18:30.802110 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:30.802074    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"474e7a33c8ee247f675562129318c3a865174d5d26ecbd96ea8cf2f6b027af8b"} err="failed to get container status \"474e7a33c8ee247f675562129318c3a865174d5d26ecbd96ea8cf2f6b027af8b\": rpc error: code = NotFound desc = could not find container \"474e7a33c8ee247f675562129318c3a865174d5d26ecbd96ea8cf2f6b027af8b\": container with ID starting with 474e7a33c8ee247f675562129318c3a865174d5d26ecbd96ea8cf2f6b027af8b not found: ID does not exist"
Sep 18 20:18:30.830645 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:30.830610    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm"]
Sep 18 20:18:30.849861 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:30.849814    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-c34fad-openshift-default-6b5889c74b-vpxnm"]
Sep 18 20:18:31.522665 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:31.522630    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2cb0038-fbd2-4eff-a731-227ed00eed57" path="/var/lib/kubelet/pods/d2cb0038-fbd2-4eff-a731-227ed00eed57/volumes"
Sep 18 20:18:32.800296 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:32.800261    2549 generic.go:356] "Generic (PLEG): container finished" podID="7a68b8bc-a551-4666-98ce-909b507edd2d" containerID="e66e5f718cf77246509150fcb67777489cc6f3bbe02afa754f9a6fd19c7d8d04" exitCode=0
Sep 18 20:18:32.800835 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:32.800333    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x" event={"ID":"7a68b8bc-a551-4666-98ce-909b507edd2d","Type":"ContainerDied","Data":"e66e5f718cf77246509150fcb67777489cc6f3bbe02afa754f9a6fd19c7d8d04"}
Sep 18 20:18:32.800910 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:32.800836    2549 scope.go:117] "RemoveContainer" containerID="e66e5f718cf77246509150fcb67777489cc6f3bbe02afa754f9a6fd19c7d8d04"
Sep 18 20:18:32.802105 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:32.802074    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:18:32.802215 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:32.802159    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:18:32.802215 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:32.802208    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:18:33.812872 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:33.812834    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x" event={"ID":"7a68b8bc-a551-4666-98ce-909b507edd2d","Type":"ContainerStarted","Data":"0e4dc9e0562572920eff16d4bb3f3371a3f9f7f0fa4188cba4c03c8838dd52ff"}
Sep 18 20:18:34.209910 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:34.209873    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x"
Sep 18 20:18:34.211207 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:34.211180    2549 patch_prober.go:28] interesting pod/e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Sep 18 20:18:34.211342 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:34.211234    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x" podUID="7a68b8bc-a551-4666-98ce-909b507edd2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 18 20:18:35.210792 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:35.210746    2549 patch_prober.go:28] interesting pod/e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Sep 18 20:18:35.211219 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:35.210829    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x" podUID="7a68b8bc-a551-4666-98ce-909b507edd2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 18 20:18:35.587208 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:35.587167    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x"
Sep 18 20:18:36.210866 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:36.210780    2549 patch_prober.go:28] interesting pod/e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Sep 18 20:18:36.210866 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:36.210843    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x" podUID="7a68b8bc-a551-4666-98ce-909b507edd2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 18 20:18:37.210920 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:37.210878    2549 patch_prober.go:28] interesting pod/e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Sep 18 20:18:37.211343 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:37.210945    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x" podUID="7a68b8bc-a551-4666-98ce-909b507edd2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 18 20:18:38.210164 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:38.210121    2549 patch_prober.go:28] interesting pod/e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Sep 18 20:18:38.210372 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:38.210186    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x" podUID="7a68b8bc-a551-4666-98ce-909b507edd2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 18 20:18:39.210770 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:39.210726    2549 patch_prober.go:28] interesting pod/e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Sep 18 20:18:39.211198 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:39.210799    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x" podUID="7a68b8bc-a551-4666-98ce-909b507edd2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 18 20:18:40.214907 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:40.214874    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x"
Sep 18 20:18:40.215955 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:40.215935    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x"
Sep 18 20:18:45.341418 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:45.341357    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:18:55.638994 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:55.638946    2549 scope.go:117] "RemoveContainer" containerID="c7dd8980f037cc888ad7d53c342c5cfc6f3ad2fd88c8dd63fdc4db0a36832317"
Sep 18 20:18:55.650490 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:55.650458    2549 scope.go:117] "RemoveContainer" containerID="901490bdb2fa08401b24c7ab3d701dfa5390ed1ad355f911516ca9cdd50e5ead"
Sep 18 20:18:56.981578 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:56.981470    2549 generic.go:356] "Generic (PLEG): container finished" podID="64206b1e-8c51-4e7f-9bbe-3ea703ccb671" containerID="8b23930ee8074f5d24393a3249ad26e96ba0db34880259b5f3ae9630e9e82bd3" exitCode=0
Sep 18 20:18:56.981578 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:56.981545    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9" event={"ID":"64206b1e-8c51-4e7f-9bbe-3ea703ccb671","Type":"ContainerDied","Data":"8b23930ee8074f5d24393a3249ad26e96ba0db34880259b5f3ae9630e9e82bd3"}
Sep 18 20:18:56.982259 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:56.982197    2549 scope.go:117] "RemoveContainer" containerID="8b23930ee8074f5d24393a3249ad26e96ba0db34880259b5f3ae9630e9e82bd3"
Sep 18 20:18:56.983676 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:56.983638    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:18:56.983816 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:56.983734    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:18:56.983816 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:56.983785    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:18:56.983939 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:56.983888    2549 generic.go:356] "Generic (PLEG): container finished" podID="a390c820-232c-4b8c-b4e4-711699f90d2b" containerID="7010c2fd412a05d5026549528e8b7e2cc026fcefacc2d6dec96f5c626d750063" exitCode=0
Sep 18 20:18:56.984060 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:56.984036    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g" event={"ID":"a390c820-232c-4b8c-b4e4-711699f90d2b","Type":"ContainerDied","Data":"7010c2fd412a05d5026549528e8b7e2cc026fcefacc2d6dec96f5c626d750063"}
Sep 18 20:18:56.984490 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:56.984475    2549 scope.go:117] "RemoveContainer" containerID="7010c2fd412a05d5026549528e8b7e2cc026fcefacc2d6dec96f5c626d750063"
Sep 18 20:18:56.985627 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:56.985600    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:18:56.985727 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:56.985694    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:18:56.985771 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:56.985744    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:18:57.992766 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:57.992724    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9" event={"ID":"64206b1e-8c51-4e7f-9bbe-3ea703ccb671","Type":"ContainerStarted","Data":"13c1df7f2880c2f58f13c9e5f8a31c14bfe147363c4324622c4a0801065f5e2f"}
Sep 18 20:18:57.994918 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:57.994884    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g" event={"ID":"a390c820-232c-4b8c-b4e4-711699f90d2b","Type":"ContainerStarted","Data":"774e37b5243071bde848d20fdd9c92ef4534f975e551ff4a5e8343118b09d395"}
Sep 18 20:18:58.076129 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:58.076086    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g"
Sep 18 20:18:58.077426 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:58.077398    2549 patch_prober.go:28] interesting pod/e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Sep 18 20:18:58.077576 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:58.077454    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g" podUID="a390c820-232c-4b8c-b4e4-711699f90d2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Sep 18 20:18:58.311054 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:58.310950    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9"
Sep 18 20:18:58.312287 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:58.312254    2549 patch_prober.go:28] interesting pod/e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Sep 18 20:18:58.312438 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:58.312312    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9" podUID="64206b1e-8c51-4e7f-9bbe-3ea703ccb671" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Sep 18 20:18:58.764336 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:58.764289    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9"
Sep 18 20:18:59.077015 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:59.076969    2549 patch_prober.go:28] interesting pod/e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Sep 18 20:18:59.077691 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:59.077650    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g" podUID="a390c820-232c-4b8c-b4e4-711699f90d2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Sep 18 20:18:59.310975 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:59.310937    2549 patch_prober.go:28] interesting pod/e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Sep 18 20:18:59.311206 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:59.311183    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9" podUID="64206b1e-8c51-4e7f-9bbe-3ea703ccb671" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Sep 18 20:18:59.771495 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:18:59.771440    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g"
Sep 18 20:19:00.077321 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:00.077273    2549 patch_prober.go:28] interesting pod/e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Sep 18 20:19:00.077878 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:00.077355    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g" podUID="a390c820-232c-4b8c-b4e4-711699f90d2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Sep 18 20:19:00.311031 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:00.310988    2549 patch_prober.go:28] interesting pod/e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Sep 18 20:19:00.311229 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:00.311067    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9" podUID="64206b1e-8c51-4e7f-9bbe-3ea703ccb671" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Sep 18 20:19:01.077299 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:01.077259    2549 patch_prober.go:28] interesting pod/e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Sep 18 20:19:01.077684 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:01.077633    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g" podUID="a390c820-232c-4b8c-b4e4-711699f90d2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Sep 18 20:19:01.311495 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:01.311457    2549 patch_prober.go:28] interesting pod/e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Sep 18 20:19:01.311712 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:01.311524    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9" podUID="64206b1e-8c51-4e7f-9bbe-3ea703ccb671" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Sep 18 20:19:01.738680 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:01.738635    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7735-6f95d7"]
Sep 18 20:19:01.739365 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:01.739334    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2cb0038-fbd2-4eff-a731-227ed00eed57" containerName="istio-proxy"
Sep 18 20:19:01.739365 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:01.739364    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2cb0038-fbd2-4eff-a731-227ed00eed57" containerName="istio-proxy"
Sep 18 20:19:01.739545 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:01.739396    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2cb0038-fbd2-4eff-a731-227ed00eed57" containerName="istio-proxy"
Sep 18 20:19:01.739545 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:01.739404    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2cb0038-fbd2-4eff-a731-227ed00eed57" containerName="istio-proxy"
Sep 18 20:19:01.739545 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:01.739496    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2cb0038-fbd2-4eff-a731-227ed00eed57" containerName="istio-proxy"
Sep 18 20:19:01.739545 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:01.739505    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2cb0038-fbd2-4eff-a731-227ed00eed57" containerName="istio-proxy"
Sep 18 20:19:01.742332 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:01.742308    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7735-6f95d7"
Sep 18 20:19:01.748415 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:01.748352    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-jcdsw\""
Sep 18 20:19:01.757825 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:01.755945    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7735-6f95d7"]
Sep 18 20:19:01.885085 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:01.885011    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsdvp\" (UniqueName: \"kubernetes.io/projected/89616b2c-5461-4531-81e7-b217c977ced1-kube-api-access-nsdvp\") pod \"test-curl-7735-6f95d7\" (UID: \"89616b2c-5461-4531-81e7-b217c977ced1\") " pod="opendatahub/test-curl-7735-6f95d7"
Sep 18 20:19:01.986654 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:01.986588    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsdvp\" (UniqueName: \"kubernetes.io/projected/89616b2c-5461-4531-81e7-b217c977ced1-kube-api-access-nsdvp\") pod \"test-curl-7735-6f95d7\" (UID: \"89616b2c-5461-4531-81e7-b217c977ced1\") " pod="opendatahub/test-curl-7735-6f95d7"
Sep 18 20:19:02.005327 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:02.005224    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsdvp\" (UniqueName: \"kubernetes.io/projected/89616b2c-5461-4531-81e7-b217c977ced1-kube-api-access-nsdvp\") pod \"test-curl-7735-6f95d7\" (UID: \"89616b2c-5461-4531-81e7-b217c977ced1\") " pod="opendatahub/test-curl-7735-6f95d7"
Sep 18 20:19:02.063821 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:02.063780    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7735-6f95d7"
Sep 18 20:19:02.083978 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:02.083935    2549 patch_prober.go:28] interesting pod/e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Sep 18 20:19:02.084598 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:02.084558    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g" podUID="a390c820-232c-4b8c-b4e4-711699f90d2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Sep 18 20:19:02.334576 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:02.327327    2549 patch_prober.go:28] interesting pod/e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Sep 18 20:19:02.334576 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:02.333653    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9" podUID="64206b1e-8c51-4e7f-9bbe-3ea703ccb671" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Sep 18 20:19:02.508429 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:02.508291    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7735-6f95d7"]
Sep 18 20:19:03.032589 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:03.032485    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7735-6f95d7" event={"ID":"89616b2c-5461-4531-81e7-b217c977ced1","Type":"ContainerStarted","Data":"3e7b9655d85982899f609c09be1301f0055c050163634ddf47c9ae9fba6afcaa"}
Sep 18 20:19:03.032589 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:03.032533    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7735-6f95d7" event={"ID":"89616b2c-5461-4531-81e7-b217c977ced1","Type":"ContainerStarted","Data":"b34aff405720b054215d5f6f475271086bc37e997f6884992109007b16ca410a"}
Sep 18 20:19:03.059065 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:03.058978    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7735-6f95d7" podStartSLOduration=2.058948122 podStartE2EDuration="2.058948122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:19:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:19:03.054855786 +0000 UTC m=+1328.008596955" watchObservedRunningTime="2026-09-18 20:19:03.058948122 +0000 UTC m=+1328.012689292"
Sep 18 20:19:03.076887 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:03.076332    2549 patch_prober.go:28] interesting pod/e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Sep 18 20:19:03.076887 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:03.076438    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g" podUID="a390c820-232c-4b8c-b4e4-711699f90d2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Sep 18 20:19:03.310651 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:03.310544    2549 patch_prober.go:28] interesting pod/e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Sep 18 20:19:03.310651 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:03.310609    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9" podUID="64206b1e-8c51-4e7f-9bbe-3ea703ccb671" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Sep 18 20:19:03.886229 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:03.886190    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7735-6f95d7"]
Sep 18 20:19:03.894573 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:03.894537    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7735-6f95d7"]
Sep 18 20:19:04.022003 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:04.021953    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7735-8b546d"]
Sep 18 20:19:04.022483 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:04.022461    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89616b2c-5461-4531-81e7-b217c977ced1" containerName="test-curl-7735-6f95d7"
Sep 18 20:19:04.022483 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:04.022483    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="89616b2c-5461-4531-81e7-b217c977ced1" containerName="test-curl-7735-6f95d7"
Sep 18 20:19:04.022720 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:04.022574    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="89616b2c-5461-4531-81e7-b217c977ced1" containerName="test-curl-7735-6f95d7"
Sep 18 20:19:04.024902 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:04.024880    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7735-8b546d"
Sep 18 20:19:04.037797 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:04.037762    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7735-8b546d"]
Sep 18 20:19:04.043818 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:19:04.043784    2549 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7735-6f95d7" container="test-curl-7735-6f95d7"
Sep 18 20:19:04.044074 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:04.044041    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6qnq\" (UniqueName: \"kubernetes.io/projected/c0a0cc19-b3b6-400b-9c20-525ef2fade74-kube-api-access-z6qnq\") pod \"test-curl-7735-8b546d\" (UID: \"c0a0cc19-b3b6-400b-9c20-525ef2fade74\") " pod="opendatahub/test-curl-7735-8b546d"
Sep 18 20:19:04.079312 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:04.078840    2549 patch_prober.go:28] interesting pod/e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Sep 18 20:19:04.079312 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:04.078910    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g" podUID="a390c820-232c-4b8c-b4e4-711699f90d2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Sep 18 20:19:04.144931 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:04.144822    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6qnq\" (UniqueName: \"kubernetes.io/projected/c0a0cc19-b3b6-400b-9c20-525ef2fade74-kube-api-access-z6qnq\") pod \"test-curl-7735-8b546d\" (UID: \"c0a0cc19-b3b6-400b-9c20-525ef2fade74\") " pod="opendatahub/test-curl-7735-8b546d"
Sep 18 20:19:04.167413 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:04.159792    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6qnq\" (UniqueName: \"kubernetes.io/projected/c0a0cc19-b3b6-400b-9c20-525ef2fade74-kube-api-access-z6qnq\") pod \"test-curl-7735-8b546d\" (UID: \"c0a0cc19-b3b6-400b-9c20-525ef2fade74\") " pod="opendatahub/test-curl-7735-8b546d"
Sep 18 20:19:04.315470 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:04.315438    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9"
Sep 18 20:19:04.316627 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:04.316603    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9"
Sep 18 20:19:04.336660 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:04.336624    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7735-8b546d"
Sep 18 20:19:04.643016 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:04.642976    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7735-8b546d"]
Sep 18 20:19:04.647204 ip-10-0-140-178 kubenswrapper[2549]: W0918 20:19:04.647158    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc0a0cc19_b3b6_400b_9c20_525ef2fade74.slice/crio-65ab4f3c7013d7672b5b886c45076f5d1139eddc7a8f584d5f1abfc001493f05 WatchSource:0}: Error finding container 65ab4f3c7013d7672b5b886c45076f5d1139eddc7a8f584d5f1abfc001493f05: Status 404 returned error can't find the container with id 65ab4f3c7013d7672b5b886c45076f5d1139eddc7a8f584d5f1abfc001493f05
Sep 18 20:19:05.051999 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:05.051886    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7735-8b546d" event={"ID":"c0a0cc19-b3b6-400b-9c20-525ef2fade74","Type":"ContainerStarted","Data":"a70cb22f3735d024b722f28394b6e0189600b0bbdae6843f35474606eb53b5dd"}
Sep 18 20:19:05.051999 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:05.051947    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7735-8b546d" event={"ID":"c0a0cc19-b3b6-400b-9c20-525ef2fade74","Type":"ContainerStarted","Data":"65ab4f3c7013d7672b5b886c45076f5d1139eddc7a8f584d5f1abfc001493f05"}
Sep 18 20:19:05.052250 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:05.052211    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7735-6f95d7" podUID="89616b2c-5461-4531-81e7-b217c977ced1" containerName="test-curl-7735-6f95d7" containerID="cri-o://3e7b9655d85982899f609c09be1301f0055c050163634ddf47c9ae9fba6afcaa" gracePeriod=2
Sep 18 20:19:05.072109 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:05.072048    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7735-8b546d" podStartSLOduration=1.0720263 podStartE2EDuration="1.0720263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:19:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:19:05.071103478 +0000 UTC m=+1330.024844637" watchObservedRunningTime="2026-09-18 20:19:05.0720263 +0000 UTC m=+1330.025767471"
Sep 18 20:19:05.074657 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:19:05.074622    2549 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7735-6f95d7\" is forbidden: User \"system:node:ip-10-0-140-178.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-178.ec2.internal' and this object" podUID="89616b2c-5461-4531-81e7-b217c977ced1" pod="opendatahub/test-curl-7735-6f95d7"
Sep 18 20:19:05.081373 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:05.081339    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g"
Sep 18 20:19:05.082665 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:05.082644    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g"
Sep 18 20:19:05.120037 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:19:05.119988    2549 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7735-6f95d7\" is forbidden: User \"system:node:ip-10-0-140-178.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-178.ec2.internal' and this object" podUID="89616b2c-5461-4531-81e7-b217c977ced1" pod="opendatahub/test-curl-7735-6f95d7"
Sep 18 20:19:05.125937 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:19:05.125903    2549 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7735-6f95d7\" is forbidden: User \"system:node:ip-10-0-140-178.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-178.ec2.internal' and this object" podUID="89616b2c-5461-4531-81e7-b217c977ced1" pod="opendatahub/test-curl-7735-6f95d7"
Sep 18 20:19:05.521767 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:19:05.521724    2549 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7735-6f95d7\" is forbidden: User \"system:node:ip-10-0-140-178.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-178.ec2.internal' and this object" podUID="89616b2c-5461-4531-81e7-b217c977ced1" pod="opendatahub/test-curl-7735-6f95d7"
Sep 18 20:19:05.817272 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:05.817177    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7735-8b546d"]
Sep 18 20:19:05.822809 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:05.822773    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7735-8b546d"]
Sep 18 20:19:05.897397 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:05.897341    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq"]
Sep 18 20:19:05.897819 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:05.897805    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0a0cc19-b3b6-400b-9c20-525ef2fade74" containerName="test-curl-7735-8b546d"
Sep 18 20:19:05.897866 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:05.897821    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0a0cc19-b3b6-400b-9c20-525ef2fade74" containerName="test-curl-7735-8b546d"
Sep 18 20:19:05.897912 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:05.897903    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0a0cc19-b3b6-400b-9c20-525ef2fade74" containerName="test-curl-7735-8b546d"
Sep 18 20:19:05.900193 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:05.900172    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq"
Sep 18 20:19:05.903174 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:05.903146    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-81f19d-openshift-default-dockercfg-8xf5r\""
Sep 18 20:19:05.919257 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:05.919218    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq"]
Sep 18 20:19:05.960880 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:05.960841    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/137f5df4-4bde-48f8-bb7e-9f629e941e0d-istio-token\") pod \"e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq\" (UID: \"137f5df4-4bde-48f8-bb7e-9f629e941e0d\") " pod="openshift-ingress/e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq"
Sep 18 20:19:05.960880 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:05.960885    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvjhj\" (UniqueName: \"kubernetes.io/projected/137f5df4-4bde-48f8-bb7e-9f629e941e0d-kube-api-access-tvjhj\") pod \"e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq\" (UID: \"137f5df4-4bde-48f8-bb7e-9f629e941e0d\") " pod="openshift-ingress/e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq"
Sep 18 20:19:05.961117 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:05.960925    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/137f5df4-4bde-48f8-bb7e-9f629e941e0d-istiod-ca-cert\") pod \"e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq\" (UID: \"137f5df4-4bde-48f8-bb7e-9f629e941e0d\") " pod="openshift-ingress/e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq"
Sep 18 20:19:05.961117 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:05.960986    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/137f5df4-4bde-48f8-bb7e-9f629e941e0d-istio-envoy\") pod \"e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq\" (UID: \"137f5df4-4bde-48f8-bb7e-9f629e941e0d\") " pod="openshift-ingress/e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq"
Sep 18 20:19:05.961117 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:05.961052    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/137f5df4-4bde-48f8-bb7e-9f629e941e0d-credential-socket\") pod \"e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq\" (UID: \"137f5df4-4bde-48f8-bb7e-9f629e941e0d\") " pod="openshift-ingress/e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq"
Sep 18 20:19:05.961117 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:05.961078    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/137f5df4-4bde-48f8-bb7e-9f629e941e0d-workload-certs\") pod \"e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq\" (UID: \"137f5df4-4bde-48f8-bb7e-9f629e941e0d\") " pod="openshift-ingress/e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq"
Sep 18 20:19:05.961251 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:05.961134    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/137f5df4-4bde-48f8-bb7e-9f629e941e0d-istio-data\") pod \"e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq\" (UID: \"137f5df4-4bde-48f8-bb7e-9f629e941e0d\") " pod="openshift-ingress/e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq"
Sep 18 20:19:05.961251 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:05.961153    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/137f5df4-4bde-48f8-bb7e-9f629e941e0d-workload-socket\") pod \"e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq\" (UID: \"137f5df4-4bde-48f8-bb7e-9f629e941e0d\") " pod="openshift-ingress/e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq"
Sep 18 20:19:05.961251 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:05.961179    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/137f5df4-4bde-48f8-bb7e-9f629e941e0d-istio-podinfo\") pod \"e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq\" (UID: \"137f5df4-4bde-48f8-bb7e-9f629e941e0d\") " pod="openshift-ingress/e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq"
Sep 18 20:19:05.972509 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:05.972464    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7735-2f44c3"]
Sep 18 20:19:05.975558 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:05.975269    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7735-2f44c3"
Sep 18 20:19:05.986102 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:05.985928    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7735-2f44c3"]
Sep 18 20:19:06.059631 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:19:06.059602    2549 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7735-8b546d" container="test-curl-7735-8b546d"
Sep 18 20:19:06.062230 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:06.062199    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/137f5df4-4bde-48f8-bb7e-9f629e941e0d-credential-socket\") pod \"e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq\" (UID: \"137f5df4-4bde-48f8-bb7e-9f629e941e0d\") " pod="openshift-ingress/e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq"
Sep 18 20:19:06.062333 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:06.062238    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/137f5df4-4bde-48f8-bb7e-9f629e941e0d-workload-certs\") pod \"e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq\" (UID: \"137f5df4-4bde-48f8-bb7e-9f629e941e0d\") " pod="openshift-ingress/e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq"
Sep 18 20:19:06.062333 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:06.062264    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/137f5df4-4bde-48f8-bb7e-9f629e941e0d-istio-data\") pod \"e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq\" (UID: \"137f5df4-4bde-48f8-bb7e-9f629e941e0d\") " pod="openshift-ingress/e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq"
Sep 18 20:19:06.062333 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:06.062306    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/137f5df4-4bde-48f8-bb7e-9f629e941e0d-workload-socket\") pod \"e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq\" (UID: \"137f5df4-4bde-48f8-bb7e-9f629e941e0d\") " pod="openshift-ingress/e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq"
Sep 18 20:19:06.062485 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:06.062351    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/137f5df4-4bde-48f8-bb7e-9f629e941e0d-istio-podinfo\") pod \"e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq\" (UID: \"137f5df4-4bde-48f8-bb7e-9f629e941e0d\") " pod="openshift-ingress/e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq"
Sep 18 20:19:06.062485 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:06.062427    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/137f5df4-4bde-48f8-bb7e-9f629e941e0d-istio-token\") pod \"e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq\" (UID: \"137f5df4-4bde-48f8-bb7e-9f629e941e0d\") " pod="openshift-ingress/e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq"
Sep 18 20:19:06.062485 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:06.062452    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvjhj\" (UniqueName: \"kubernetes.io/projected/137f5df4-4bde-48f8-bb7e-9f629e941e0d-kube-api-access-tvjhj\") pod \"e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq\" (UID: \"137f5df4-4bde-48f8-bb7e-9f629e941e0d\") " pod="openshift-ingress/e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq"
Sep 18 20:19:06.062638 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:06.062518    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/137f5df4-4bde-48f8-bb7e-9f629e941e0d-istiod-ca-cert\") pod \"e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq\" (UID: \"137f5df4-4bde-48f8-bb7e-9f629e941e0d\") " pod="openshift-ingress/e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq"
Sep 18 20:19:06.062638 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:06.062564    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/137f5df4-4bde-48f8-bb7e-9f629e941e0d-istio-envoy\") pod \"e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq\" (UID: \"137f5df4-4bde-48f8-bb7e-9f629e941e0d\") " pod="openshift-ingress/e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq"
Sep 18 20:19:06.062638 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:06.062604    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5k76d\" (UniqueName: \"kubernetes.io/projected/93964805-31d8-44e8-b602-a3cc9552de50-kube-api-access-5k76d\") pod \"test-curl-7735-2f44c3\" (UID: \"93964805-31d8-44e8-b602-a3cc9552de50\") " pod="opendatahub/test-curl-7735-2f44c3"
Sep 18 20:19:06.062842 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:06.062820    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/137f5df4-4bde-48f8-bb7e-9f629e941e0d-workload-certs\") pod \"e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq\" (UID: \"137f5df4-4bde-48f8-bb7e-9f629e941e0d\") " pod="openshift-ingress/e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq"
Sep 18 20:19:06.062903 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:06.062842    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/137f5df4-4bde-48f8-bb7e-9f629e941e0d-workload-socket\") pod \"e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq\" (UID: \"137f5df4-4bde-48f8-bb7e-9f629e941e0d\") " pod="openshift-ingress/e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq"
Sep 18 20:19:06.062903 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:06.062866    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/137f5df4-4bde-48f8-bb7e-9f629e941e0d-istio-data\") pod \"e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq\" (UID: \"137f5df4-4bde-48f8-bb7e-9f629e941e0d\") " pod="openshift-ingress/e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq"
Sep 18 20:19:06.062974 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:06.062965    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/137f5df4-4bde-48f8-bb7e-9f629e941e0d-credential-socket\") pod \"e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq\" (UID: \"137f5df4-4bde-48f8-bb7e-9f629e941e0d\") " pod="openshift-ingress/e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq"
Sep 18 20:19:06.063538 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:06.063511    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/137f5df4-4bde-48f8-bb7e-9f629e941e0d-istiod-ca-cert\") pod \"e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq\" (UID: \"137f5df4-4bde-48f8-bb7e-9f629e941e0d\") " pod="openshift-ingress/e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq"
Sep 18 20:19:06.065127 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:06.065102    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/137f5df4-4bde-48f8-bb7e-9f629e941e0d-istio-envoy\") pod \"e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq\" (UID: \"137f5df4-4bde-48f8-bb7e-9f629e941e0d\") " pod="openshift-ingress/e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq"
Sep 18 20:19:06.065257 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:06.065203    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/137f5df4-4bde-48f8-bb7e-9f629e941e0d-istio-podinfo\") pod \"e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq\" (UID: \"137f5df4-4bde-48f8-bb7e-9f629e941e0d\") " pod="openshift-ingress/e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq"
Sep 18 20:19:06.088304 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:06.088260    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvjhj\" (UniqueName: \"kubernetes.io/projected/137f5df4-4bde-48f8-bb7e-9f629e941e0d-kube-api-access-tvjhj\") pod \"e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq\" (UID: \"137f5df4-4bde-48f8-bb7e-9f629e941e0d\") " pod="openshift-ingress/e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq"
Sep 18 20:19:06.091556 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:06.091524    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/137f5df4-4bde-48f8-bb7e-9f629e941e0d-istio-token\") pod \"e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq\" (UID: \"137f5df4-4bde-48f8-bb7e-9f629e941e0d\") " pod="openshift-ingress/e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq"
Sep 18 20:19:06.163833 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:06.163789    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k76d\" (UniqueName: \"kubernetes.io/projected/93964805-31d8-44e8-b602-a3cc9552de50-kube-api-access-5k76d\") pod \"test-curl-7735-2f44c3\" (UID: \"93964805-31d8-44e8-b602-a3cc9552de50\") " pod="opendatahub/test-curl-7735-2f44c3"
Sep 18 20:19:06.179504 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:06.179463    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k76d\" (UniqueName: \"kubernetes.io/projected/93964805-31d8-44e8-b602-a3cc9552de50-kube-api-access-5k76d\") pod \"test-curl-7735-2f44c3\" (UID: \"93964805-31d8-44e8-b602-a3cc9552de50\") " pod="opendatahub/test-curl-7735-2f44c3"
Sep 18 20:19:06.211422 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:06.211363    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq"
Sep 18 20:19:06.295977 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:06.295925    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7735-2f44c3"
Sep 18 20:19:06.518645 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:06.518617    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq"]
Sep 18 20:19:06.520509 ip-10-0-140-178 kubenswrapper[2549]: W0918 20:19:06.520474    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod137f5df4_4bde_48f8_bb7e_9f629e941e0d.slice/crio-63446d91966e884fb8b1c240047a46431afbb1153dfe2db129915665ea57266d WatchSource:0}: Error finding container 63446d91966e884fb8b1c240047a46431afbb1153dfe2db129915665ea57266d: Status 404 returned error can't find the container with id 63446d91966e884fb8b1c240047a46431afbb1153dfe2db129915665ea57266d
Sep 18 20:19:06.523059 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:06.523022    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:19:06.523352 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:06.523096    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:19:06.523352 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:06.523127    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:19:06.597186 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:06.597155    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7735-2f44c3"]
Sep 18 20:19:06.597734 ip-10-0-140-178 kubenswrapper[2549]: W0918 20:19:06.597709    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod93964805_31d8_44e8_b602_a3cc9552de50.slice/crio-e16fbfb99b51a7ed802654199ceed68bfbf0c01b1cc6428e8651c663def7d591 WatchSource:0}: Error finding container e16fbfb99b51a7ed802654199ceed68bfbf0c01b1cc6428e8651c663def7d591: Status 404 returned error can't find the container with id e16fbfb99b51a7ed802654199ceed68bfbf0c01b1cc6428e8651c663def7d591
Sep 18 20:19:07.068097 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:07.068052    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7735-2f44c3" event={"ID":"93964805-31d8-44e8-b602-a3cc9552de50","Type":"ContainerStarted","Data":"742c061af596fcd2b93188d19c7dcf10accbe7a9d4506cbec6c97ef0e79b1bb7"}
Sep 18 20:19:07.068310 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:07.068107    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7735-2f44c3" event={"ID":"93964805-31d8-44e8-b602-a3cc9552de50","Type":"ContainerStarted","Data":"e16fbfb99b51a7ed802654199ceed68bfbf0c01b1cc6428e8651c663def7d591"}
Sep 18 20:19:07.070083 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:07.070046    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq" event={"ID":"137f5df4-4bde-48f8-bb7e-9f629e941e0d","Type":"ContainerStarted","Data":"553d73c599b9e77dd37f4d4cc516d6478ba9d410c7a12795a29e144c8c4a273b"}
Sep 18 20:19:07.070270 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:07.070106    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq" event={"ID":"137f5df4-4bde-48f8-bb7e-9f629e941e0d","Type":"ContainerStarted","Data":"63446d91966e884fb8b1c240047a46431afbb1153dfe2db129915665ea57266d"}
Sep 18 20:19:07.070270 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:07.070141    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7735-8b546d" podUID="c0a0cc19-b3b6-400b-9c20-525ef2fade74" containerName="test-curl-7735-8b546d" containerID="cri-o://a70cb22f3735d024b722f28394b6e0189600b0bbdae6843f35474606eb53b5dd" gracePeriod=2
Sep 18 20:19:07.115070 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:07.115014    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7735-2f44c3" podStartSLOduration=2.114986491 podStartE2EDuration="2.114986491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:19:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:19:07.113064225 +0000 UTC m=+1332.066805392" watchObservedRunningTime="2026-09-18 20:19:07.114986491 +0000 UTC m=+1332.068727662"
Sep 18 20:19:07.196083 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:07.196022    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq" podStartSLOduration=2.196000738 podStartE2EDuration="2.196000738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:19:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:19:07.194813687 +0000 UTC m=+1332.148554858" watchObservedRunningTime="2026-09-18 20:19:07.196000738 +0000 UTC m=+1332.149741906"
Sep 18 20:19:07.202482 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:19:07.202436    2549 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7735-8b546d\" is forbidden: User \"system:node:ip-10-0-140-178.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-178.ec2.internal' and this object" podUID="c0a0cc19-b3b6-400b-9c20-525ef2fade74" pod="opendatahub/test-curl-7735-8b546d"
Sep 18 20:19:07.212293 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:07.212254    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq"
Sep 18 20:19:07.218042 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:07.218008    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq"
Sep 18 20:19:07.220659 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:19:07.220617    2549 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7735-8b546d\" is forbidden: User \"system:node:ip-10-0-140-178.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-178.ec2.internal' and this object" podUID="c0a0cc19-b3b6-400b-9c20-525ef2fade74" pod="opendatahub/test-curl-7735-8b546d"
Sep 18 20:19:07.250072 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:07.250037    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7735-6f95d7"
Sep 18 20:19:07.274290 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:07.274253    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nsdvp\" (UniqueName: \"kubernetes.io/projected/89616b2c-5461-4531-81e7-b217c977ced1-kube-api-access-nsdvp\") pod \"89616b2c-5461-4531-81e7-b217c977ced1\" (UID: \"89616b2c-5461-4531-81e7-b217c977ced1\") "
Sep 18 20:19:07.276651 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:07.276606    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89616b2c-5461-4531-81e7-b217c977ced1-kube-api-access-nsdvp" (OuterVolumeSpecName: "kube-api-access-nsdvp") pod "89616b2c-5461-4531-81e7-b217c977ced1" (UID: "89616b2c-5461-4531-81e7-b217c977ced1"). InnerVolumeSpecName "kube-api-access-nsdvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:19:07.286770 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:19:07.286731    2549 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7735-8b546d\" is forbidden: User \"system:node:ip-10-0-140-178.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-178.ec2.internal' and this object" podUID="c0a0cc19-b3b6-400b-9c20-525ef2fade74" pod="opendatahub/test-curl-7735-8b546d"
Sep 18 20:19:07.375688 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:07.375637    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsdvp\" (UniqueName: \"kubernetes.io/projected/89616b2c-5461-4531-81e7-b217c977ced1-kube-api-access-nsdvp\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:19:07.523812 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:07.523774    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89616b2c-5461-4531-81e7-b217c977ced1" path="/var/lib/kubelet/pods/89616b2c-5461-4531-81e7-b217c977ced1/volumes"
Sep 18 20:19:07.921156 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:07.921115    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7735-2f44c3"]
Sep 18 20:19:07.927185 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:07.927089    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7735-2f44c3"]
Sep 18 20:19:08.040947 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:08.040899    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7735-e76aa4"]
Sep 18 20:19:08.041395 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:08.041367    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93964805-31d8-44e8-b602-a3cc9552de50" containerName="test-curl-7735-2f44c3"
Sep 18 20:19:08.041453 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:08.041397    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="93964805-31d8-44e8-b602-a3cc9552de50" containerName="test-curl-7735-2f44c3"
Sep 18 20:19:08.041518 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:08.041508    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="93964805-31d8-44e8-b602-a3cc9552de50" containerName="test-curl-7735-2f44c3"
Sep 18 20:19:08.043698 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:08.043678    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7735-e76aa4"
Sep 18 20:19:08.059429 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:08.059393    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7735-e76aa4"]
Sep 18 20:19:08.080854 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:08.080812    2549 generic.go:356] "Generic (PLEG): container finished" podID="89616b2c-5461-4531-81e7-b217c977ced1" containerID="3e7b9655d85982899f609c09be1301f0055c050163634ddf47c9ae9fba6afcaa" exitCode=137
Sep 18 20:19:08.081095 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:08.080871    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7735-6f95d7"
Sep 18 20:19:08.081095 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:08.080906    2549 scope.go:117] "RemoveContainer" containerID="3e7b9655d85982899f609c09be1301f0055c050163634ddf47c9ae9fba6afcaa"
Sep 18 20:19:08.081357 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:08.081295    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq"
Sep 18 20:19:08.081491 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:08.081397    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft5st\" (UniqueName: \"kubernetes.io/projected/0d0b9203-289a-4690-ad37-79e398b7da22-kube-api-access-ft5st\") pod \"test-curl-7735-e76aa4\" (UID: \"0d0b9203-289a-4690-ad37-79e398b7da22\") " pod="opendatahub/test-curl-7735-e76aa4"
Sep 18 20:19:08.081557 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:19:08.081488    2549 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7735-2f44c3" container="test-curl-7735-2f44c3"
Sep 18 20:19:08.082645 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:08.082621    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq"
Sep 18 20:19:08.091487 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:08.091460    2549 scope.go:117] "RemoveContainer" containerID="3e7b9655d85982899f609c09be1301f0055c050163634ddf47c9ae9fba6afcaa"
Sep 18 20:19:08.091894 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:19:08.091867    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e7b9655d85982899f609c09be1301f0055c050163634ddf47c9ae9fba6afcaa\": container with ID starting with 3e7b9655d85982899f609c09be1301f0055c050163634ddf47c9ae9fba6afcaa not found: ID does not exist" containerID="3e7b9655d85982899f609c09be1301f0055c050163634ddf47c9ae9fba6afcaa"
Sep 18 20:19:08.091971 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:08.091909    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e7b9655d85982899f609c09be1301f0055c050163634ddf47c9ae9fba6afcaa"} err="failed to get container status \"3e7b9655d85982899f609c09be1301f0055c050163634ddf47c9ae9fba6afcaa\": rpc error: code = NotFound desc = could not find container \"3e7b9655d85982899f609c09be1301f0055c050163634ddf47c9ae9fba6afcaa\": container with ID starting with 3e7b9655d85982899f609c09be1301f0055c050163634ddf47c9ae9fba6afcaa not found: ID does not exist"
Sep 18 20:19:08.182535 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:08.182437    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft5st\" (UniqueName: \"kubernetes.io/projected/0d0b9203-289a-4690-ad37-79e398b7da22-kube-api-access-ft5st\") pod \"test-curl-7735-e76aa4\" (UID: \"0d0b9203-289a-4690-ad37-79e398b7da22\") " pod="opendatahub/test-curl-7735-e76aa4"
Sep 18 20:19:08.194176 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:08.194143    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft5st\" (UniqueName: \"kubernetes.io/projected/0d0b9203-289a-4690-ad37-79e398b7da22-kube-api-access-ft5st\") pod \"test-curl-7735-e76aa4\" (UID: \"0d0b9203-289a-4690-ad37-79e398b7da22\") " pod="opendatahub/test-curl-7735-e76aa4"
Sep 18 20:19:08.353562 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:08.353524    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7735-e76aa4"
Sep 18 20:19:08.634355 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:08.634330    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7735-e76aa4"]
Sep 18 20:19:08.636568 ip-10-0-140-178 kubenswrapper[2549]: W0918 20:19:08.636532    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0d0b9203_289a_4690_ad37_79e398b7da22.slice/crio-2df4b39423dcf54b3b4f42b144fd12142042e3ec1403c52fd1327169f6b48108 WatchSource:0}: Error finding container 2df4b39423dcf54b3b4f42b144fd12142042e3ec1403c52fd1327169f6b48108: Status 404 returned error can't find the container with id 2df4b39423dcf54b3b4f42b144fd12142042e3ec1403c52fd1327169f6b48108
Sep 18 20:19:09.089278 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:09.089235    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7735-e76aa4" event={"ID":"0d0b9203-289a-4690-ad37-79e398b7da22","Type":"ContainerStarted","Data":"3eac121c9e030b9c02c48e50833b548323722fc2916dffe2f1b694bdc9a5d54e"}
Sep 18 20:19:09.089486 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:09.089290    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7735-e76aa4" event={"ID":"0d0b9203-289a-4690-ad37-79e398b7da22","Type":"ContainerStarted","Data":"2df4b39423dcf54b3b4f42b144fd12142042e3ec1403c52fd1327169f6b48108"}
Sep 18 20:19:09.090657 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:09.090626    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7735-2f44c3" podUID="93964805-31d8-44e8-b602-a3cc9552de50" containerName="test-curl-7735-2f44c3" containerID="cri-o://742c061af596fcd2b93188d19c7dcf10accbe7a9d4506cbec6c97ef0e79b1bb7" gracePeriod=2
Sep 18 20:19:09.116931 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:09.116874    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7735-e76aa4" podStartSLOduration=1.116855335 podStartE2EDuration="1.116855335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:19:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:19:09.115439182 +0000 UTC m=+1334.069180352" watchObservedRunningTime="2026-09-18 20:19:09.116855335 +0000 UTC m=+1334.070596503"
Sep 18 20:19:09.121013 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:19:09.120975    2549 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7735-2f44c3\" is forbidden: User \"system:node:ip-10-0-140-178.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-178.ec2.internal' and this object" podUID="93964805-31d8-44e8-b602-a3cc9552de50" pod="opendatahub/test-curl-7735-2f44c3"
Sep 18 20:19:09.268206 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:09.268178    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7735-8b546d"
Sep 18 20:19:09.277260 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:19:09.277168    2549 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7735-2f44c3\" is forbidden: User \"system:node:ip-10-0-140-178.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-178.ec2.internal' and this object" podUID="93964805-31d8-44e8-b602-a3cc9552de50" pod="opendatahub/test-curl-7735-2f44c3"
Sep 18 20:19:09.294705 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:09.294671    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z6qnq\" (UniqueName: \"kubernetes.io/projected/c0a0cc19-b3b6-400b-9c20-525ef2fade74-kube-api-access-z6qnq\") pod \"c0a0cc19-b3b6-400b-9c20-525ef2fade74\" (UID: \"c0a0cc19-b3b6-400b-9c20-525ef2fade74\") "
Sep 18 20:19:09.296976 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:09.296939    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0a0cc19-b3b6-400b-9c20-525ef2fade74-kube-api-access-z6qnq" (OuterVolumeSpecName: "kube-api-access-z6qnq") pod "c0a0cc19-b3b6-400b-9c20-525ef2fade74" (UID: "c0a0cc19-b3b6-400b-9c20-525ef2fade74"). InnerVolumeSpecName "kube-api-access-z6qnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:19:09.396027 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:09.395921    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6qnq\" (UniqueName: \"kubernetes.io/projected/c0a0cc19-b3b6-400b-9c20-525ef2fade74-kube-api-access-z6qnq\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:19:09.521956 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:09.521920    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0a0cc19-b3b6-400b-9c20-525ef2fade74" path="/var/lib/kubelet/pods/c0a0cc19-b3b6-400b-9c20-525ef2fade74/volumes"
Sep 18 20:19:10.098685 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:10.098643    2549 generic.go:356] "Generic (PLEG): container finished" podID="c0a0cc19-b3b6-400b-9c20-525ef2fade74" containerID="a70cb22f3735d024b722f28394b6e0189600b0bbdae6843f35474606eb53b5dd" exitCode=137
Sep 18 20:19:10.099158 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:10.098708    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7735-8b546d"
Sep 18 20:19:10.099158 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:10.098759    2549 scope.go:117] "RemoveContainer" containerID="a70cb22f3735d024b722f28394b6e0189600b0bbdae6843f35474606eb53b5dd"
Sep 18 20:19:10.111262 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:10.111233    2549 scope.go:117] "RemoveContainer" containerID="a70cb22f3735d024b722f28394b6e0189600b0bbdae6843f35474606eb53b5dd"
Sep 18 20:19:10.111790 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:19:10.111769    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a70cb22f3735d024b722f28394b6e0189600b0bbdae6843f35474606eb53b5dd\": container with ID starting with a70cb22f3735d024b722f28394b6e0189600b0bbdae6843f35474606eb53b5dd not found: ID does not exist" containerID="a70cb22f3735d024b722f28394b6e0189600b0bbdae6843f35474606eb53b5dd"
Sep 18 20:19:10.111850 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:10.111801    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a70cb22f3735d024b722f28394b6e0189600b0bbdae6843f35474606eb53b5dd"} err="failed to get container status \"a70cb22f3735d024b722f28394b6e0189600b0bbdae6843f35474606eb53b5dd\": rpc error: code = NotFound desc = could not find container \"a70cb22f3735d024b722f28394b6e0189600b0bbdae6843f35474606eb53b5dd\": container with ID starting with a70cb22f3735d024b722f28394b6e0189600b0bbdae6843f35474606eb53b5dd not found: ID does not exist"
Sep 18 20:19:10.120490 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:10.120450    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7735-e76aa4"]
Sep 18 20:19:10.132327 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:10.132295    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7735-e76aa4"]
Sep 18 20:19:11.107321 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:11.107282    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7735-e76aa4" podUID="0d0b9203-289a-4690-ad37-79e398b7da22" containerName="test-curl-7735-e76aa4" containerID="cri-o://3eac121c9e030b9c02c48e50833b548323722fc2916dffe2f1b694bdc9a5d54e" gracePeriod=2
Sep 18 20:19:11.112375 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:19:11.112342    2549 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7735-e76aa4\" is forbidden: User \"system:node:ip-10-0-140-178.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-178.ec2.internal' and this object" podUID="0d0b9203-289a-4690-ad37-79e398b7da22" pod="opendatahub/test-curl-7735-e76aa4"
Sep 18 20:19:11.212637 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:11.212216    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4"]
Sep 18 20:19:11.212952 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:11.212923    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d0b9203-289a-4690-ad37-79e398b7da22" containerName="test-curl-7735-e76aa4"
Sep 18 20:19:11.212952 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:11.212951    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d0b9203-289a-4690-ad37-79e398b7da22" containerName="test-curl-7735-e76aa4"
Sep 18 20:19:11.213159 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:11.213078    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d0b9203-289a-4690-ad37-79e398b7da22" containerName="test-curl-7735-e76aa4"
Sep 18 20:19:11.215664 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:11.215640    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4"
Sep 18 20:19:11.219893 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:11.219713    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-7d49a30d-openshift-default-dockercfg-mnsd8\""
Sep 18 20:19:11.239101 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:11.238928    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4"]
Sep 18 20:19:11.270208 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:19:11.270164    2549 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7735-e76aa4\" is forbidden: User \"system:node:ip-10-0-140-178.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-178.ec2.internal' and this object" podUID="0d0b9203-289a-4690-ad37-79e398b7da22" pod="opendatahub/test-curl-7735-e76aa4"
Sep 18 20:19:11.319408 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:11.319181    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0658bf28-e97b-4d0d-945e-2f43253220b6-credential-socket\") pod \"e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4\" (UID: \"0658bf28-e97b-4d0d-945e-2f43253220b6\") " pod="openshift-ingress/e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4"
Sep 18 20:19:11.319408 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:11.319233    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0658bf28-e97b-4d0d-945e-2f43253220b6-istio-data\") pod \"e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4\" (UID: \"0658bf28-e97b-4d0d-945e-2f43253220b6\") " pod="openshift-ingress/e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4"
Sep 18 20:19:11.319408 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:11.319265    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0658bf28-e97b-4d0d-945e-2f43253220b6-workload-socket\") pod \"e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4\" (UID: \"0658bf28-e97b-4d0d-945e-2f43253220b6\") " pod="openshift-ingress/e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4"
Sep 18 20:19:11.319408 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:11.319312    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0658bf28-e97b-4d0d-945e-2f43253220b6-istio-podinfo\") pod \"e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4\" (UID: \"0658bf28-e97b-4d0d-945e-2f43253220b6\") " pod="openshift-ingress/e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4"
Sep 18 20:19:11.319408 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:11.319338    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0658bf28-e97b-4d0d-945e-2f43253220b6-istiod-ca-cert\") pod \"e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4\" (UID: \"0658bf28-e97b-4d0d-945e-2f43253220b6\") " pod="openshift-ingress/e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4"
Sep 18 20:19:11.319853 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:11.319540    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0658bf28-e97b-4d0d-945e-2f43253220b6-istio-token\") pod \"e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4\" (UID: \"0658bf28-e97b-4d0d-945e-2f43253220b6\") " pod="openshift-ingress/e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4"
Sep 18 20:19:11.319853 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:11.319613    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0658bf28-e97b-4d0d-945e-2f43253220b6-workload-certs\") pod \"e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4\" (UID: \"0658bf28-e97b-4d0d-945e-2f43253220b6\") " pod="openshift-ingress/e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4"
Sep 18 20:19:11.319853 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:11.319645    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0658bf28-e97b-4d0d-945e-2f43253220b6-istio-envoy\") pod \"e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4\" (UID: \"0658bf28-e97b-4d0d-945e-2f43253220b6\") " pod="openshift-ingress/e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4"
Sep 18 20:19:11.319853 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:11.319689    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6pgx\" (UniqueName: \"kubernetes.io/projected/0658bf28-e97b-4d0d-945e-2f43253220b6-kube-api-access-z6pgx\") pod \"e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4\" (UID: \"0658bf28-e97b-4d0d-945e-2f43253220b6\") " pod="openshift-ingress/e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4"
Sep 18 20:19:11.336705 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:11.336281    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7735-2f44c3"
Sep 18 20:19:11.339114 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:19:11.339078    2549 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7735-e76aa4\" is forbidden: User \"system:node:ip-10-0-140-178.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-178.ec2.internal' and this object" podUID="0d0b9203-289a-4690-ad37-79e398b7da22" pod="opendatahub/test-curl-7735-e76aa4"
Sep 18 20:19:11.420989 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:11.420876    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5k76d\" (UniqueName: \"kubernetes.io/projected/93964805-31d8-44e8-b602-a3cc9552de50-kube-api-access-5k76d\") pod \"93964805-31d8-44e8-b602-a3cc9552de50\" (UID: \"93964805-31d8-44e8-b602-a3cc9552de50\") "
Sep 18 20:19:11.421167 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:11.421085    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0658bf28-e97b-4d0d-945e-2f43253220b6-istio-token\") pod \"e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4\" (UID: \"0658bf28-e97b-4d0d-945e-2f43253220b6\") " pod="openshift-ingress/e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4"
Sep 18 20:19:11.421167 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:11.421150    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0658bf28-e97b-4d0d-945e-2f43253220b6-workload-certs\") pod \"e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4\" (UID: \"0658bf28-e97b-4d0d-945e-2f43253220b6\") " pod="openshift-ingress/e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4"
Sep 18 20:19:11.421303 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:11.421179    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0658bf28-e97b-4d0d-945e-2f43253220b6-istio-envoy\") pod \"e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4\" (UID: \"0658bf28-e97b-4d0d-945e-2f43253220b6\") " pod="openshift-ingress/e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4"
Sep 18 20:19:11.421303 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:11.421219    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6pgx\" (UniqueName: \"kubernetes.io/projected/0658bf28-e97b-4d0d-945e-2f43253220b6-kube-api-access-z6pgx\") pod \"e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4\" (UID: \"0658bf28-e97b-4d0d-945e-2f43253220b6\") " pod="openshift-ingress/e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4"
Sep 18 20:19:11.421303 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:11.421278    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0658bf28-e97b-4d0d-945e-2f43253220b6-credential-socket\") pod \"e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4\" (UID: \"0658bf28-e97b-4d0d-945e-2f43253220b6\") " pod="openshift-ingress/e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4"
Sep 18 20:19:11.421608 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:11.421303    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0658bf28-e97b-4d0d-945e-2f43253220b6-istio-data\") pod \"e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4\" (UID: \"0658bf28-e97b-4d0d-945e-2f43253220b6\") " pod="openshift-ingress/e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4"
Sep 18 20:19:11.421608 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:11.421333    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0658bf28-e97b-4d0d-945e-2f43253220b6-workload-socket\") pod \"e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4\" (UID: \"0658bf28-e97b-4d0d-945e-2f43253220b6\") " pod="openshift-ingress/e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4"
Sep 18 20:19:11.421608 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:11.421374    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0658bf28-e97b-4d0d-945e-2f43253220b6-istio-podinfo\") pod \"e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4\" (UID: \"0658bf28-e97b-4d0d-945e-2f43253220b6\") " pod="openshift-ingress/e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4"
Sep 18 20:19:11.421608 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:11.421443    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0658bf28-e97b-4d0d-945e-2f43253220b6-istiod-ca-cert\") pod \"e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4\" (UID: \"0658bf28-e97b-4d0d-945e-2f43253220b6\") " pod="openshift-ingress/e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4"
Sep 18 20:19:11.421608 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:11.421577    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0658bf28-e97b-4d0d-945e-2f43253220b6-workload-certs\") pod \"e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4\" (UID: \"0658bf28-e97b-4d0d-945e-2f43253220b6\") " pod="openshift-ingress/e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4"
Sep 18 20:19:11.421878 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:11.421822    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0658bf28-e97b-4d0d-945e-2f43253220b6-istio-data\") pod \"e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4\" (UID: \"0658bf28-e97b-4d0d-945e-2f43253220b6\") " pod="openshift-ingress/e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4"
Sep 18 20:19:11.421941 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:11.421892    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0658bf28-e97b-4d0d-945e-2f43253220b6-workload-socket\") pod \"e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4\" (UID: \"0658bf28-e97b-4d0d-945e-2f43253220b6\") " pod="openshift-ingress/e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4"
Sep 18 20:19:11.422052 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:11.422031    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0658bf28-e97b-4d0d-945e-2f43253220b6-credential-socket\") pod \"e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4\" (UID: \"0658bf28-e97b-4d0d-945e-2f43253220b6\") " pod="openshift-ingress/e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4"
Sep 18 20:19:11.422269 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:11.422234    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0658bf28-e97b-4d0d-945e-2f43253220b6-istiod-ca-cert\") pod \"e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4\" (UID: \"0658bf28-e97b-4d0d-945e-2f43253220b6\") " pod="openshift-ingress/e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4"
Sep 18 20:19:11.423752 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:11.423724    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93964805-31d8-44e8-b602-a3cc9552de50-kube-api-access-5k76d" (OuterVolumeSpecName: "kube-api-access-5k76d") pod "93964805-31d8-44e8-b602-a3cc9552de50" (UID: "93964805-31d8-44e8-b602-a3cc9552de50"). InnerVolumeSpecName "kube-api-access-5k76d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:19:11.423895 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:11.423878    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0658bf28-e97b-4d0d-945e-2f43253220b6-istio-envoy\") pod \"e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4\" (UID: \"0658bf28-e97b-4d0d-945e-2f43253220b6\") " pod="openshift-ingress/e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4"
Sep 18 20:19:11.424031 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:11.424013    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0658bf28-e97b-4d0d-945e-2f43253220b6-istio-podinfo\") pod \"e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4\" (UID: \"0658bf28-e97b-4d0d-945e-2f43253220b6\") " pod="openshift-ingress/e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4"
Sep 18 20:19:11.432820 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:11.432786    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6pgx\" (UniqueName: \"kubernetes.io/projected/0658bf28-e97b-4d0d-945e-2f43253220b6-kube-api-access-z6pgx\") pod \"e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4\" (UID: \"0658bf28-e97b-4d0d-945e-2f43253220b6\") " pod="openshift-ingress/e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4"
Sep 18 20:19:11.434025 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:11.433995    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0658bf28-e97b-4d0d-945e-2f43253220b6-istio-token\") pod \"e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4\" (UID: \"0658bf28-e97b-4d0d-945e-2f43253220b6\") " pod="openshift-ingress/e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4"
Sep 18 20:19:11.522369 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:11.522328    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5k76d\" (UniqueName: \"kubernetes.io/projected/93964805-31d8-44e8-b602-a3cc9552de50-kube-api-access-5k76d\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:19:11.522565 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:11.522449    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93964805-31d8-44e8-b602-a3cc9552de50" path="/var/lib/kubelet/pods/93964805-31d8-44e8-b602-a3cc9552de50/volumes"
Sep 18 20:19:11.530113 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:11.530079    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4"
Sep 18 20:19:11.818352 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:11.818325    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4"]
Sep 18 20:19:11.819333 ip-10-0-140-178 kubenswrapper[2549]: W0918 20:19:11.819295    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0658bf28_e97b_4d0d_945e_2f43253220b6.slice/crio-0af0d94d14e5ae58b9a20c5e912201055d2cf5ba23072e9e5d291e5aeb169bb5 WatchSource:0}: Error finding container 0af0d94d14e5ae58b9a20c5e912201055d2cf5ba23072e9e5d291e5aeb169bb5: Status 404 returned error can't find the container with id 0af0d94d14e5ae58b9a20c5e912201055d2cf5ba23072e9e5d291e5aeb169bb5
Sep 18 20:19:11.821769 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:11.821734    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:19:11.821850 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:11.821807    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:19:11.821850 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:11.821841    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:19:12.118638 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:12.118596    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4" event={"ID":"0658bf28-e97b-4d0d-945e-2f43253220b6","Type":"ContainerStarted","Data":"c3e7cd13d085eddc5f8e3ccacbb7de0601518bcc35bae99b48523cc0f261f96b"}
Sep 18 20:19:12.119090 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:12.118666    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4" event={"ID":"0658bf28-e97b-4d0d-945e-2f43253220b6","Type":"ContainerStarted","Data":"0af0d94d14e5ae58b9a20c5e912201055d2cf5ba23072e9e5d291e5aeb169bb5"}
Sep 18 20:19:12.120433 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:12.120378    2549 generic.go:356] "Generic (PLEG): container finished" podID="93964805-31d8-44e8-b602-a3cc9552de50" containerID="742c061af596fcd2b93188d19c7dcf10accbe7a9d4506cbec6c97ef0e79b1bb7" exitCode=137
Sep 18 20:19:12.120575 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:12.120443    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7735-2f44c3"
Sep 18 20:19:12.120575 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:12.120498    2549 scope.go:117] "RemoveContainer" containerID="742c061af596fcd2b93188d19c7dcf10accbe7a9d4506cbec6c97ef0e79b1bb7"
Sep 18 20:19:12.131428 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:12.131405    2549 scope.go:117] "RemoveContainer" containerID="742c061af596fcd2b93188d19c7dcf10accbe7a9d4506cbec6c97ef0e79b1bb7"
Sep 18 20:19:12.131768 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:19:12.131749    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"742c061af596fcd2b93188d19c7dcf10accbe7a9d4506cbec6c97ef0e79b1bb7\": container with ID starting with 742c061af596fcd2b93188d19c7dcf10accbe7a9d4506cbec6c97ef0e79b1bb7 not found: ID does not exist" containerID="742c061af596fcd2b93188d19c7dcf10accbe7a9d4506cbec6c97ef0e79b1bb7"
Sep 18 20:19:12.131818 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:12.131778    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"742c061af596fcd2b93188d19c7dcf10accbe7a9d4506cbec6c97ef0e79b1bb7"} err="failed to get container status \"742c061af596fcd2b93188d19c7dcf10accbe7a9d4506cbec6c97ef0e79b1bb7\": rpc error: code = NotFound desc = could not find container \"742c061af596fcd2b93188d19c7dcf10accbe7a9d4506cbec6c97ef0e79b1bb7\": container with ID starting with 742c061af596fcd2b93188d19c7dcf10accbe7a9d4506cbec6c97ef0e79b1bb7 not found: ID does not exist"
Sep 18 20:19:12.160283 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:12.160216    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4" podStartSLOduration=1.1601948069999999 podStartE2EDuration="1.160194807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:19:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:19:12.157515219 +0000 UTC m=+1337.111256388" watchObservedRunningTime="2026-09-18 20:19:12.160194807 +0000 UTC m=+1337.113935974"
Sep 18 20:19:12.436851 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:12.436748    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:19:12.530532 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:12.530489    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4"
Sep 18 20:19:12.536377 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:12.536342    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4"
Sep 18 20:19:13.130461 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:13.130429    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4"
Sep 18 20:19:13.131700 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:13.131680    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4"
Sep 18 20:19:13.327194 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:13.327166    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7735-e76aa4"
Sep 18 20:19:13.342340 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:13.342310    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ft5st\" (UniqueName: \"kubernetes.io/projected/0d0b9203-289a-4690-ad37-79e398b7da22-kube-api-access-ft5st\") pod \"0d0b9203-289a-4690-ad37-79e398b7da22\" (UID: \"0d0b9203-289a-4690-ad37-79e398b7da22\") "
Sep 18 20:19:13.344682 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:13.344632    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d0b9203-289a-4690-ad37-79e398b7da22-kube-api-access-ft5st" (OuterVolumeSpecName: "kube-api-access-ft5st") pod "0d0b9203-289a-4690-ad37-79e398b7da22" (UID: "0d0b9203-289a-4690-ad37-79e398b7da22"). InnerVolumeSpecName "kube-api-access-ft5st". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:19:13.443791 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:13.443688    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ft5st\" (UniqueName: \"kubernetes.io/projected/0d0b9203-289a-4690-ad37-79e398b7da22-kube-api-access-ft5st\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:19:13.522516 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:13.522475    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d0b9203-289a-4690-ad37-79e398b7da22" path="/var/lib/kubelet/pods/0d0b9203-289a-4690-ad37-79e398b7da22/volumes"
Sep 18 20:19:14.138682 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:14.138644    2549 generic.go:356] "Generic (PLEG): container finished" podID="0d0b9203-289a-4690-ad37-79e398b7da22" containerID="3eac121c9e030b9c02c48e50833b548323722fc2916dffe2f1b694bdc9a5d54e" exitCode=137
Sep 18 20:19:14.139200 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:14.138691    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7735-e76aa4"
Sep 18 20:19:14.139200 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:14.138747    2549 scope.go:117] "RemoveContainer" containerID="3eac121c9e030b9c02c48e50833b548323722fc2916dffe2f1b694bdc9a5d54e"
Sep 18 20:19:14.150625 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:14.150592    2549 scope.go:117] "RemoveContainer" containerID="3eac121c9e030b9c02c48e50833b548323722fc2916dffe2f1b694bdc9a5d54e"
Sep 18 20:19:14.151215 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:19:14.151180    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3eac121c9e030b9c02c48e50833b548323722fc2916dffe2f1b694bdc9a5d54e\": container with ID starting with 3eac121c9e030b9c02c48e50833b548323722fc2916dffe2f1b694bdc9a5d54e not found: ID does not exist" containerID="3eac121c9e030b9c02c48e50833b548323722fc2916dffe2f1b694bdc9a5d54e"
Sep 18 20:19:14.151348 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:14.151231    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3eac121c9e030b9c02c48e50833b548323722fc2916dffe2f1b694bdc9a5d54e"} err="failed to get container status \"3eac121c9e030b9c02c48e50833b548323722fc2916dffe2f1b694bdc9a5d54e\": rpc error: code = NotFound desc = could not find container \"3eac121c9e030b9c02c48e50833b548323722fc2916dffe2f1b694bdc9a5d54e\": container with ID starting with 3eac121c9e030b9c02c48e50833b548323722fc2916dffe2f1b694bdc9a5d54e not found: ID does not exist"
Sep 18 20:19:38.234658 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:38.234608    2549 patch_prober.go:28] interesting pod/e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": read tcp 10.133.0.2:54726->10.133.0.78:15021: read: connection reset by peer" start-of-body=
Sep 18 20:19:38.235087 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:38.234691    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq" podUID="137f5df4-4bde-48f8-bb7e-9f629e941e0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": read tcp 10.133.0.2:54726->10.133.0.78:15021: read: connection reset by peer"
Sep 18 20:19:38.344443 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:38.344404    2549 generic.go:356] "Generic (PLEG): container finished" podID="0658bf28-e97b-4d0d-945e-2f43253220b6" containerID="c3e7cd13d085eddc5f8e3ccacbb7de0601518bcc35bae99b48523cc0f261f96b" exitCode=0
Sep 18 20:19:38.344643 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:38.344491    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4" event={"ID":"0658bf28-e97b-4d0d-945e-2f43253220b6","Type":"ContainerDied","Data":"c3e7cd13d085eddc5f8e3ccacbb7de0601518bcc35bae99b48523cc0f261f96b"}
Sep 18 20:19:38.345066 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:38.345046    2549 scope.go:117] "RemoveContainer" containerID="c3e7cd13d085eddc5f8e3ccacbb7de0601518bcc35bae99b48523cc0f261f96b"
Sep 18 20:19:38.346513 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:38.346478    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:19:38.346635 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:38.346561    2549 generic.go:356] "Generic (PLEG): container finished" podID="137f5df4-4bde-48f8-bb7e-9f629e941e0d" containerID="553d73c599b9e77dd37f4d4cc516d6478ba9d410c7a12795a29e144c8c4a273b" exitCode=0
Sep 18 20:19:38.346635 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:38.346575    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:19:38.346635 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:38.346621    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:19:38.346794 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:38.346634    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq" event={"ID":"137f5df4-4bde-48f8-bb7e-9f629e941e0d","Type":"ContainerDied","Data":"553d73c599b9e77dd37f4d4cc516d6478ba9d410c7a12795a29e144c8c4a273b"}
Sep 18 20:19:38.347128 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:38.347111    2549 scope.go:117] "RemoveContainer" containerID="553d73c599b9e77dd37f4d4cc516d6478ba9d410c7a12795a29e144c8c4a273b"
Sep 18 20:19:38.348525 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:38.348492    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:19:38.348632 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:38.348577    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:19:38.348632 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:38.348621    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:19:39.364082 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:39.363230    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4" event={"ID":"0658bf28-e97b-4d0d-945e-2f43253220b6","Type":"ContainerStarted","Data":"fbb6f9623fd80f82cc90e6bd50ac2aa9c5065a74fb4516a0cadcdbb31b21d02e"}
Sep 18 20:19:39.368141 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:39.367240    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq" event={"ID":"137f5df4-4bde-48f8-bb7e-9f629e941e0d","Type":"ContainerStarted","Data":"51d181a0c791d363668f8a7991f1926e9465ef7be04953181e4277364619c6b0"}
Sep 18 20:19:39.530927 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:39.530887    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4"
Sep 18 20:19:39.532119 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:39.532091    2549 patch_prober.go:28] interesting pod/e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Sep 18 20:19:39.532373 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:39.532352    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4" podUID="0658bf28-e97b-4d0d-945e-2f43253220b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Sep 18 20:19:40.211952 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:40.211915    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq"
Sep 18 20:19:40.212152 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:40.212053    2549 patch_prober.go:28] interesting pod/e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Sep 18 20:19:40.212152 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:40.212094    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq" podUID="137f5df4-4bde-48f8-bb7e-9f629e941e0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Sep 18 20:19:40.531169 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:40.531070    2549 patch_prober.go:28] interesting pod/e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Sep 18 20:19:40.531169 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:40.531143    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4" podUID="0658bf28-e97b-4d0d-945e-2f43253220b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Sep 18 20:19:41.212314 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:41.212277    2549 patch_prober.go:28] interesting pod/e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Sep 18 20:19:41.212546 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:41.212345    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq" podUID="137f5df4-4bde-48f8-bb7e-9f629e941e0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Sep 18 20:19:41.531235 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:41.531126    2549 patch_prober.go:28] interesting pod/e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Sep 18 20:19:41.531235 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:41.531188    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4" podUID="0658bf28-e97b-4d0d-945e-2f43253220b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Sep 18 20:19:42.211828 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:42.211785    2549 patch_prober.go:28] interesting pod/e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Sep 18 20:19:42.212034 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:42.211847    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq" podUID="137f5df4-4bde-48f8-bb7e-9f629e941e0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Sep 18 20:19:42.531239 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:42.531128    2549 patch_prober.go:28] interesting pod/e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Sep 18 20:19:42.531239 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:42.531201    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4" podUID="0658bf28-e97b-4d0d-945e-2f43253220b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Sep 18 20:19:43.130746 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:43.130704    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4"
Sep 18 20:19:43.212720 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:43.212679    2549 patch_prober.go:28] interesting pod/e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Sep 18 20:19:43.212911 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:43.212742    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq" podUID="137f5df4-4bde-48f8-bb7e-9f629e941e0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Sep 18 20:19:43.531116 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:43.531008    2549 patch_prober.go:28] interesting pod/e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Sep 18 20:19:43.531116 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:43.531092    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4" podUID="0658bf28-e97b-4d0d-945e-2f43253220b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Sep 18 20:19:44.211975 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:44.211936    2549 patch_prober.go:28] interesting pod/e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Sep 18 20:19:44.212448 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:44.212007    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq" podUID="137f5df4-4bde-48f8-bb7e-9f629e941e0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Sep 18 20:19:44.531031 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:44.530936    2549 patch_prober.go:28] interesting pod/e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Sep 18 20:19:44.531031 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:44.531007    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4" podUID="0658bf28-e97b-4d0d-945e-2f43253220b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Sep 18 20:19:45.216365 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:45.216333    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq"
Sep 18 20:19:45.216837 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:45.216780    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq"
Sep 18 20:19:45.217528 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:45.217506    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq"
Sep 18 20:19:45.535143 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:45.535053    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4"
Sep 18 20:19:45.536402 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:45.536354    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4"
Sep 18 20:19:45.956854 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:45.956807    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:19:52.528198 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:52.528158    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-40a1a5/auth-test-model-40a1a5-kserve-78fdfcfdc5-6f279"]
Sep 18 20:19:52.531545 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:52.531524    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-40a1a5/auth-test-model-40a1a5-kserve-78fdfcfdc5-6f279"
Sep 18 20:19:52.535550 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:52.535509    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-40a1a5\"/\"openshift-service-ca.crt\""
Sep 18 20:19:52.536880 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:52.536859    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-40a1a5\"/\"default-dockercfg-5vgdp\""
Sep 18 20:19:52.536974 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:52.536914    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-40a1a5\"/\"auth-test-model-40a1a5-kserve-self-signed-certs\""
Sep 18 20:19:52.536974 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:52.536913    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-40a1a5\"/\"kube-root-ca.crt\""
Sep 18 20:19:52.547162 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:52.547127    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-40a1a5/auth-test-model-40a1a5-kserve-78fdfcfdc5-6f279"]
Sep 18 20:19:52.593633 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:52.593582    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4b5a5a7-c19d-45eb-b615-5fec088e0610-home\") pod \"auth-test-model-40a1a5-kserve-78fdfcfdc5-6f279\" (UID: \"d4b5a5a7-c19d-45eb-b615-5fec088e0610\") " pod="ai-tenant-e2e-shared-a-w4-40a1a5/auth-test-model-40a1a5-kserve-78fdfcfdc5-6f279"
Sep 18 20:19:52.593633 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:52.593638    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4b5a5a7-c19d-45eb-b615-5fec088e0610-dshm\") pod \"auth-test-model-40a1a5-kserve-78fdfcfdc5-6f279\" (UID: \"d4b5a5a7-c19d-45eb-b615-5fec088e0610\") " pod="ai-tenant-e2e-shared-a-w4-40a1a5/auth-test-model-40a1a5-kserve-78fdfcfdc5-6f279"
Sep 18 20:19:52.593883 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:52.593853    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jslc\" (UniqueName: \"kubernetes.io/projected/d4b5a5a7-c19d-45eb-b615-5fec088e0610-kube-api-access-9jslc\") pod \"auth-test-model-40a1a5-kserve-78fdfcfdc5-6f279\" (UID: \"d4b5a5a7-c19d-45eb-b615-5fec088e0610\") " pod="ai-tenant-e2e-shared-a-w4-40a1a5/auth-test-model-40a1a5-kserve-78fdfcfdc5-6f279"
Sep 18 20:19:52.594093 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:52.594068    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4b5a5a7-c19d-45eb-b615-5fec088e0610-model-cache\") pod \"auth-test-model-40a1a5-kserve-78fdfcfdc5-6f279\" (UID: \"d4b5a5a7-c19d-45eb-b615-5fec088e0610\") " pod="ai-tenant-e2e-shared-a-w4-40a1a5/auth-test-model-40a1a5-kserve-78fdfcfdc5-6f279"
Sep 18 20:19:52.594156 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:52.594111    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4b5a5a7-c19d-45eb-b615-5fec088e0610-tmp-dir\") pod \"auth-test-model-40a1a5-kserve-78fdfcfdc5-6f279\" (UID: \"d4b5a5a7-c19d-45eb-b615-5fec088e0610\") " pod="ai-tenant-e2e-shared-a-w4-40a1a5/auth-test-model-40a1a5-kserve-78fdfcfdc5-6f279"
Sep 18 20:19:52.594156 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:52.594128    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4b5a5a7-c19d-45eb-b615-5fec088e0610-tls-certs\") pod \"auth-test-model-40a1a5-kserve-78fdfcfdc5-6f279\" (UID: \"d4b5a5a7-c19d-45eb-b615-5fec088e0610\") " pod="ai-tenant-e2e-shared-a-w4-40a1a5/auth-test-model-40a1a5-kserve-78fdfcfdc5-6f279"
Sep 18 20:19:52.695464 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:52.695420    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4b5a5a7-c19d-45eb-b615-5fec088e0610-home\") pod \"auth-test-model-40a1a5-kserve-78fdfcfdc5-6f279\" (UID: \"d4b5a5a7-c19d-45eb-b615-5fec088e0610\") " pod="ai-tenant-e2e-shared-a-w4-40a1a5/auth-test-model-40a1a5-kserve-78fdfcfdc5-6f279"
Sep 18 20:19:52.695464 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:52.695471    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4b5a5a7-c19d-45eb-b615-5fec088e0610-dshm\") pod \"auth-test-model-40a1a5-kserve-78fdfcfdc5-6f279\" (UID: \"d4b5a5a7-c19d-45eb-b615-5fec088e0610\") " pod="ai-tenant-e2e-shared-a-w4-40a1a5/auth-test-model-40a1a5-kserve-78fdfcfdc5-6f279"
Sep 18 20:19:52.695781 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:52.695513    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jslc\" (UniqueName: \"kubernetes.io/projected/d4b5a5a7-c19d-45eb-b615-5fec088e0610-kube-api-access-9jslc\") pod \"auth-test-model-40a1a5-kserve-78fdfcfdc5-6f279\" (UID: \"d4b5a5a7-c19d-45eb-b615-5fec088e0610\") " pod="ai-tenant-e2e-shared-a-w4-40a1a5/auth-test-model-40a1a5-kserve-78fdfcfdc5-6f279"
Sep 18 20:19:52.695781 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:52.695594    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4b5a5a7-c19d-45eb-b615-5fec088e0610-model-cache\") pod \"auth-test-model-40a1a5-kserve-78fdfcfdc5-6f279\" (UID: \"d4b5a5a7-c19d-45eb-b615-5fec088e0610\") " pod="ai-tenant-e2e-shared-a-w4-40a1a5/auth-test-model-40a1a5-kserve-78fdfcfdc5-6f279"
Sep 18 20:19:52.695781 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:52.695616    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4b5a5a7-c19d-45eb-b615-5fec088e0610-tmp-dir\") pod \"auth-test-model-40a1a5-kserve-78fdfcfdc5-6f279\" (UID: \"d4b5a5a7-c19d-45eb-b615-5fec088e0610\") " pod="ai-tenant-e2e-shared-a-w4-40a1a5/auth-test-model-40a1a5-kserve-78fdfcfdc5-6f279"
Sep 18 20:19:52.695781 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:52.695643    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4b5a5a7-c19d-45eb-b615-5fec088e0610-tls-certs\") pod \"auth-test-model-40a1a5-kserve-78fdfcfdc5-6f279\" (UID: \"d4b5a5a7-c19d-45eb-b615-5fec088e0610\") " pod="ai-tenant-e2e-shared-a-w4-40a1a5/auth-test-model-40a1a5-kserve-78fdfcfdc5-6f279"
Sep 18 20:19:52.695997 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:52.695911    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4b5a5a7-c19d-45eb-b615-5fec088e0610-home\") pod \"auth-test-model-40a1a5-kserve-78fdfcfdc5-6f279\" (UID: \"d4b5a5a7-c19d-45eb-b615-5fec088e0610\") " pod="ai-tenant-e2e-shared-a-w4-40a1a5/auth-test-model-40a1a5-kserve-78fdfcfdc5-6f279"
Sep 18 20:19:52.695997 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:52.695978    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4b5a5a7-c19d-45eb-b615-5fec088e0610-model-cache\") pod \"auth-test-model-40a1a5-kserve-78fdfcfdc5-6f279\" (UID: \"d4b5a5a7-c19d-45eb-b615-5fec088e0610\") " pod="ai-tenant-e2e-shared-a-w4-40a1a5/auth-test-model-40a1a5-kserve-78fdfcfdc5-6f279"
Sep 18 20:19:52.696074 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:52.696011    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4b5a5a7-c19d-45eb-b615-5fec088e0610-tmp-dir\") pod \"auth-test-model-40a1a5-kserve-78fdfcfdc5-6f279\" (UID: \"d4b5a5a7-c19d-45eb-b615-5fec088e0610\") " pod="ai-tenant-e2e-shared-a-w4-40a1a5/auth-test-model-40a1a5-kserve-78fdfcfdc5-6f279"
Sep 18 20:19:52.697955 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:52.697931    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4b5a5a7-c19d-45eb-b615-5fec088e0610-dshm\") pod \"auth-test-model-40a1a5-kserve-78fdfcfdc5-6f279\" (UID: \"d4b5a5a7-c19d-45eb-b615-5fec088e0610\") " pod="ai-tenant-e2e-shared-a-w4-40a1a5/auth-test-model-40a1a5-kserve-78fdfcfdc5-6f279"
Sep 18 20:19:52.698247 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:52.698231    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4b5a5a7-c19d-45eb-b615-5fec088e0610-tls-certs\") pod \"auth-test-model-40a1a5-kserve-78fdfcfdc5-6f279\" (UID: \"d4b5a5a7-c19d-45eb-b615-5fec088e0610\") " pod="ai-tenant-e2e-shared-a-w4-40a1a5/auth-test-model-40a1a5-kserve-78fdfcfdc5-6f279"
Sep 18 20:19:52.706590 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:52.706562    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jslc\" (UniqueName: \"kubernetes.io/projected/d4b5a5a7-c19d-45eb-b615-5fec088e0610-kube-api-access-9jslc\") pod \"auth-test-model-40a1a5-kserve-78fdfcfdc5-6f279\" (UID: \"d4b5a5a7-c19d-45eb-b615-5fec088e0610\") " pod="ai-tenant-e2e-shared-a-w4-40a1a5/auth-test-model-40a1a5-kserve-78fdfcfdc5-6f279"
Sep 18 20:19:52.841493 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:52.841449    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-40a1a5/auth-test-model-40a1a5-kserve-78fdfcfdc5-6f279"
Sep 18 20:19:53.123045 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:53.123014    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-40a1a5/auth-test-model-40a1a5-kserve-78fdfcfdc5-6f279"]
Sep 18 20:19:53.124536 ip-10-0-140-178 kubenswrapper[2549]: W0918 20:19:53.124509    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd4b5a5a7_c19d_45eb_b615_5fec088e0610.slice/crio-d0f02fa4e5526ccb49231d5f064a83f510280710632f37738c96487bbce4ca28 WatchSource:0}: Error finding container d0f02fa4e5526ccb49231d5f064a83f510280710632f37738c96487bbce4ca28: Status 404 returned error can't find the container with id d0f02fa4e5526ccb49231d5f064a83f510280710632f37738c96487bbce4ca28
Sep 18 20:19:53.506325 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:53.506217    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-40a1a5/auth-test-model-40a1a5-kserve-78fdfcfdc5-6f279" event={"ID":"d4b5a5a7-c19d-45eb-b615-5fec088e0610","Type":"ContainerStarted","Data":"bdfee2be0eb4780f32cfbecd76d9efd0e6041edb7cfb7f70c949adacdb27e60f"}
Sep 18 20:19:53.506325 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:53.506266    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-40a1a5/auth-test-model-40a1a5-kserve-78fdfcfdc5-6f279" event={"ID":"d4b5a5a7-c19d-45eb-b615-5fec088e0610","Type":"ContainerStarted","Data":"d0f02fa4e5526ccb49231d5f064a83f510280710632f37738c96487bbce4ca28"}
Sep 18 20:19:53.549122 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:53.549042    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-40a1a5/auth-test-model-40a1a5-kserve-78fdfcfdc5-6f279" podStartSLOduration=1.549014223 podStartE2EDuration="1.549014223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:19:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:19:53.540319088 +0000 UTC m=+1378.494060256" watchObservedRunningTime="2026-09-18 20:19:53.549014223 +0000 UTC m=+1378.502755391"
Sep 18 20:19:55.688951 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:55.688914    2549 scope.go:117] "RemoveContainer" containerID="0a59a8974185c9390dd16ea7098df0e9bcfd4851c7988e899ed2a5ea5c2527a2"
Sep 18 20:19:55.699543 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:19:55.699517    2549 scope.go:117] "RemoveContainer" containerID="af2af129b91d52a1a95d03a9a049c7abdee7ba9a245526b8ef24e5211de85eb3"
Sep 18 20:20:02.842687 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:20:02.842639    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-40a1a5/auth-test-model-40a1a5-kserve-78fdfcfdc5-6f279"
Sep 18 20:20:02.843125 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:20:02.842716    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-40a1a5/auth-test-model-40a1a5-kserve-78fdfcfdc5-6f279"
Sep 18 20:20:02.858781 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:20:02.858739    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-40a1a5/auth-test-model-40a1a5-kserve-78fdfcfdc5-6f279"
Sep 18 20:20:03.585085 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:20:03.585048    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-40a1a5/auth-test-model-40a1a5-kserve-78fdfcfdc5-6f279"
Sep 18 20:20:05.625590 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:20:05.625555    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-7d49a30d/e2e-lifecycle-model-7d49a30d-kserve-b887f9547-cjpx2"]
Sep 18 20:20:05.629051 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:20:05.629028    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7d49a30d/e2e-lifecycle-model-7d49a30d-kserve-b887f9547-cjpx2"
Sep 18 20:20:05.633061 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:20:05.633030    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-7d49a30d\"/\"e2e-lifecycle-model-7d49a30d-kserve-self-signed-certs\""
Sep 18 20:20:05.634444 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:20:05.634363    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-7d49a30d\"/\"openshift-service-ca.crt\""
Sep 18 20:20:05.634444 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:20:05.634364    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-7d49a30d\"/\"kube-root-ca.crt\""
Sep 18 20:20:05.634644 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:20:05.634547    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-7d49a30d\"/\"default-dockercfg-f4kph\""
Sep 18 20:20:05.642024 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:20:05.641991    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-7d49a30d/e2e-lifecycle-model-7d49a30d-kserve-b887f9547-cjpx2"]
Sep 18 20:20:05.738967 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:20:05.738929    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64ea02f9-ae44-4a3d-aeaa-8b041a5288ab-tmp-dir\") pod \"e2e-lifecycle-model-7d49a30d-kserve-b887f9547-cjpx2\" (UID: \"64ea02f9-ae44-4a3d-aeaa-8b041a5288ab\") " pod="ai-tenant-e2e-mt-7d49a30d/e2e-lifecycle-model-7d49a30d-kserve-b887f9547-cjpx2"
Sep 18 20:20:05.739186 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:20:05.738995    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64ea02f9-ae44-4a3d-aeaa-8b041a5288ab-dshm\") pod \"e2e-lifecycle-model-7d49a30d-kserve-b887f9547-cjpx2\" (UID: \"64ea02f9-ae44-4a3d-aeaa-8b041a5288ab\") " pod="ai-tenant-e2e-mt-7d49a30d/e2e-lifecycle-model-7d49a30d-kserve-b887f9547-cjpx2"
Sep 18 20:20:05.739186 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:20:05.739034    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64ea02f9-ae44-4a3d-aeaa-8b041a5288ab-tls-certs\") pod \"e2e-lifecycle-model-7d49a30d-kserve-b887f9547-cjpx2\" (UID: \"64ea02f9-ae44-4a3d-aeaa-8b041a5288ab\") " pod="ai-tenant-e2e-mt-7d49a30d/e2e-lifecycle-model-7d49a30d-kserve-b887f9547-cjpx2"
Sep 18 20:20:05.739186 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:20:05.739050    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64ea02f9-ae44-4a3d-aeaa-8b041a5288ab-home\") pod \"e2e-lifecycle-model-7d49a30d-kserve-b887f9547-cjpx2\" (UID: \"64ea02f9-ae44-4a3d-aeaa-8b041a5288ab\") " pod="ai-tenant-e2e-mt-7d49a30d/e2e-lifecycle-model-7d49a30d-kserve-b887f9547-cjpx2"
Sep 18 20:20:05.739186 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:20:05.739075    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdxqk\" (UniqueName: \"kubernetes.io/projected/64ea02f9-ae44-4a3d-aeaa-8b041a5288ab-kube-api-access-zdxqk\") pod \"e2e-lifecycle-model-7d49a30d-kserve-b887f9547-cjpx2\" (UID: \"64ea02f9-ae44-4a3d-aeaa-8b041a5288ab\") " pod="ai-tenant-e2e-mt-7d49a30d/e2e-lifecycle-model-7d49a30d-kserve-b887f9547-cjpx2"
Sep 18 20:20:05.739331 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:20:05.739202    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64ea02f9-ae44-4a3d-aeaa-8b041a5288ab-model-cache\") pod \"e2e-lifecycle-model-7d49a30d-kserve-b887f9547-cjpx2\" (UID: \"64ea02f9-ae44-4a3d-aeaa-8b041a5288ab\") " pod="ai-tenant-e2e-mt-7d49a30d/e2e-lifecycle-model-7d49a30d-kserve-b887f9547-cjpx2"
Sep 18 20:20:05.839904 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:20:05.839866    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64ea02f9-ae44-4a3d-aeaa-8b041a5288ab-tmp-dir\") pod \"e2e-lifecycle-model-7d49a30d-kserve-b887f9547-cjpx2\" (UID: \"64ea02f9-ae44-4a3d-aeaa-8b041a5288ab\") " pod="ai-tenant-e2e-mt-7d49a30d/e2e-lifecycle-model-7d49a30d-kserve-b887f9547-cjpx2"
Sep 18 20:20:05.840119 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:20:05.839945    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64ea02f9-ae44-4a3d-aeaa-8b041a5288ab-dshm\") pod \"e2e-lifecycle-model-7d49a30d-kserve-b887f9547-cjpx2\" (UID: \"64ea02f9-ae44-4a3d-aeaa-8b041a5288ab\") " pod="ai-tenant-e2e-mt-7d49a30d/e2e-lifecycle-model-7d49a30d-kserve-b887f9547-cjpx2"
Sep 18 20:20:05.840119 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:20:05.839986    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64ea02f9-ae44-4a3d-aeaa-8b041a5288ab-tls-certs\") pod \"e2e-lifecycle-model-7d49a30d-kserve-b887f9547-cjpx2\" (UID: \"64ea02f9-ae44-4a3d-aeaa-8b041a5288ab\") " pod="ai-tenant-e2e-mt-7d49a30d/e2e-lifecycle-model-7d49a30d-kserve-b887f9547-cjpx2"
Sep 18 20:20:05.840119 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:20:05.840011    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64ea02f9-ae44-4a3d-aeaa-8b041a5288ab-home\") pod \"e2e-lifecycle-model-7d49a30d-kserve-b887f9547-cjpx2\" (UID: \"64ea02f9-ae44-4a3d-aeaa-8b041a5288ab\") " pod="ai-tenant-e2e-mt-7d49a30d/e2e-lifecycle-model-7d49a30d-kserve-b887f9547-cjpx2"
Sep 18 20:20:05.840119 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:20:05.840042    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdxqk\" (UniqueName: \"kubernetes.io/projected/64ea02f9-ae44-4a3d-aeaa-8b041a5288ab-kube-api-access-zdxqk\") pod \"e2e-lifecycle-model-7d49a30d-kserve-b887f9547-cjpx2\" (UID: \"64ea02f9-ae44-4a3d-aeaa-8b041a5288ab\") " pod="ai-tenant-e2e-mt-7d49a30d/e2e-lifecycle-model-7d49a30d-kserve-b887f9547-cjpx2"
Sep 18 20:20:05.840366 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:20:05.840143    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64ea02f9-ae44-4a3d-aeaa-8b041a5288ab-model-cache\") pod \"e2e-lifecycle-model-7d49a30d-kserve-b887f9547-cjpx2\" (UID: \"64ea02f9-ae44-4a3d-aeaa-8b041a5288ab\") " pod="ai-tenant-e2e-mt-7d49a30d/e2e-lifecycle-model-7d49a30d-kserve-b887f9547-cjpx2"
Sep 18 20:20:05.840366 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:20:05.840326    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64ea02f9-ae44-4a3d-aeaa-8b041a5288ab-tmp-dir\") pod \"e2e-lifecycle-model-7d49a30d-kserve-b887f9547-cjpx2\" (UID: \"64ea02f9-ae44-4a3d-aeaa-8b041a5288ab\") " pod="ai-tenant-e2e-mt-7d49a30d/e2e-lifecycle-model-7d49a30d-kserve-b887f9547-cjpx2"
Sep 18 20:20:05.840704 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:20:05.840666    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64ea02f9-ae44-4a3d-aeaa-8b041a5288ab-model-cache\") pod \"e2e-lifecycle-model-7d49a30d-kserve-b887f9547-cjpx2\" (UID: \"64ea02f9-ae44-4a3d-aeaa-8b041a5288ab\") " pod="ai-tenant-e2e-mt-7d49a30d/e2e-lifecycle-model-7d49a30d-kserve-b887f9547-cjpx2"
Sep 18 20:20:05.840824 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:20:05.840708    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64ea02f9-ae44-4a3d-aeaa-8b041a5288ab-home\") pod \"e2e-lifecycle-model-7d49a30d-kserve-b887f9547-cjpx2\" (UID: \"64ea02f9-ae44-4a3d-aeaa-8b041a5288ab\") " pod="ai-tenant-e2e-mt-7d49a30d/e2e-lifecycle-model-7d49a30d-kserve-b887f9547-cjpx2"
Sep 18 20:20:05.843010 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:20:05.842978    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64ea02f9-ae44-4a3d-aeaa-8b041a5288ab-dshm\") pod \"e2e-lifecycle-model-7d49a30d-kserve-b887f9547-cjpx2\" (UID: \"64ea02f9-ae44-4a3d-aeaa-8b041a5288ab\") " pod="ai-tenant-e2e-mt-7d49a30d/e2e-lifecycle-model-7d49a30d-kserve-b887f9547-cjpx2"
Sep 18 20:20:05.843294 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:20:05.843264    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64ea02f9-ae44-4a3d-aeaa-8b041a5288ab-tls-certs\") pod \"e2e-lifecycle-model-7d49a30d-kserve-b887f9547-cjpx2\" (UID: \"64ea02f9-ae44-4a3d-aeaa-8b041a5288ab\") " pod="ai-tenant-e2e-mt-7d49a30d/e2e-lifecycle-model-7d49a30d-kserve-b887f9547-cjpx2"
Sep 18 20:20:05.853024 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:20:05.852984    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdxqk\" (UniqueName: \"kubernetes.io/projected/64ea02f9-ae44-4a3d-aeaa-8b041a5288ab-kube-api-access-zdxqk\") pod \"e2e-lifecycle-model-7d49a30d-kserve-b887f9547-cjpx2\" (UID: \"64ea02f9-ae44-4a3d-aeaa-8b041a5288ab\") " pod="ai-tenant-e2e-mt-7d49a30d/e2e-lifecycle-model-7d49a30d-kserve-b887f9547-cjpx2"
Sep 18 20:20:05.939372 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:20:05.939264    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7d49a30d/e2e-lifecycle-model-7d49a30d-kserve-b887f9547-cjpx2"
Sep 18 20:20:06.221649 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:20:06.221602    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-7d49a30d/e2e-lifecycle-model-7d49a30d-kserve-b887f9547-cjpx2"]
Sep 18 20:20:06.221973 ip-10-0-140-178 kubenswrapper[2549]: W0918 20:20:06.221947    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod64ea02f9_ae44_4a3d_aeaa_8b041a5288ab.slice/crio-28d12cdfe4e93b5f092802a4920514d19b0e04537481268a917f908823c26292 WatchSource:0}: Error finding container 28d12cdfe4e93b5f092802a4920514d19b0e04537481268a917f908823c26292: Status 404 returned error can't find the container with id 28d12cdfe4e93b5f092802a4920514d19b0e04537481268a917f908823c26292
Sep 18 20:20:06.593875 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:20:06.593837    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7d49a30d/e2e-lifecycle-model-7d49a30d-kserve-b887f9547-cjpx2" event={"ID":"64ea02f9-ae44-4a3d-aeaa-8b041a5288ab","Type":"ContainerStarted","Data":"56ffd66100de87bbb0931b6b2c04c7d4a7bde3637ef847a667f6f194e152c99b"}
Sep 18 20:20:06.593875 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:20:06.593884    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7d49a30d/e2e-lifecycle-model-7d49a30d-kserve-b887f9547-cjpx2" event={"ID":"64ea02f9-ae44-4a3d-aeaa-8b041a5288ab","Type":"ContainerStarted","Data":"28d12cdfe4e93b5f092802a4920514d19b0e04537481268a917f908823c26292"}
Sep 18 20:20:06.624379 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:20:06.624292    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-7d49a30d/e2e-lifecycle-model-7d49a30d-kserve-b887f9547-cjpx2" podStartSLOduration=1.6242706519999999 podStartE2EDuration="1.624270652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:20:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:20:06.619613246 +0000 UTC m=+1391.573354414" watchObservedRunningTime="2026-09-18 20:20:06.624270652 +0000 UTC m=+1391.578011819"
Sep 18 20:20:13.645146 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:20:13.645104    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:20:15.940094 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:20:15.940033    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-7d49a30d/e2e-lifecycle-model-7d49a30d-kserve-b887f9547-cjpx2"
Sep 18 20:20:15.940574 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:20:15.940120    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-7d49a30d/e2e-lifecycle-model-7d49a30d-kserve-b887f9547-cjpx2"
Sep 18 20:20:15.953965 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:20:15.953930    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-7d49a30d/e2e-lifecycle-model-7d49a30d-kserve-b887f9547-cjpx2"
Sep 18 20:20:16.674286 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:20:16.674256    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-7d49a30d/e2e-lifecycle-model-7d49a30d-kserve-b887f9547-cjpx2"
Sep 18 20:20:42.331182 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:20:42.331135    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:20:55.750509 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:20:55.750467    2549 scope.go:117] "RemoveContainer" containerID="bfc8cdec651459455f5c62880403f0ec6e1570072514ae2d7cf4a468e398fb5a"
Sep 18 20:20:55.761300 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:20:55.761273    2549 scope.go:117] "RemoveContainer" containerID="7f0d32084081d5cf007495b2dee0cf5872c59d36bc3b62343022b04af604a75a"
Sep 18 20:20:55.772871 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:20:55.772843    2549 scope.go:117] "RemoveContainer" containerID="55935c5afe1d093bfff9339a6a352a8ccb9bdf4210724272d2f359e696ddf0f8"
Sep 18 20:21:09.523102 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:21:09.523060    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:21:15.498686 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:21:15.498638    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d49a30d-fp25j"]
Sep 18 20:21:15.504156 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:21:15.504126    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d49a30d-fp25j"
Sep 18 20:21:15.508709 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:21:15.508674    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bhssz\""
Sep 18 20:21:15.509213 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:21:15.509183    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 20:21:15.513113 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:21:15.513078    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d49a30d-fp25j"]
Sep 18 20:21:15.514743 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:21:15.514614    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3242aca7-e91c-4471-91ca-128800e6cdc0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-7d49a30d-fp25j\" (UID: \"3242aca7-e91c-4471-91ca-128800e6cdc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d49a30d-fp25j"
Sep 18 20:21:15.615896 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:21:15.615859    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3242aca7-e91c-4471-91ca-128800e6cdc0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-7d49a30d-fp25j\" (UID: \"3242aca7-e91c-4471-91ca-128800e6cdc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d49a30d-fp25j"
Sep 18 20:21:15.616634 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:21:15.616615    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3242aca7-e91c-4471-91ca-128800e6cdc0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-7d49a30d-fp25j\" (UID: \"3242aca7-e91c-4471-91ca-128800e6cdc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d49a30d-fp25j"
Sep 18 20:21:15.818840 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:21:15.818744    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d49a30d-fp25j"
Sep 18 20:21:16.121010 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:21:16.120975    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d49a30d-fp25j"]
Sep 18 20:21:16.123667 ip-10-0-140-178 kubenswrapper[2549]: W0918 20:21:16.123631    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3242aca7_e91c_4471_91ca_128800e6cdc0.slice/crio-6e8a0de5ca32415e3ee553f58997c23030f04e0fbbcb8bf39269d3ac24ae31fe WatchSource:0}: Error finding container 6e8a0de5ca32415e3ee553f58997c23030f04e0fbbcb8bf39269d3ac24ae31fe: Status 404 returned error can't find the container with id 6e8a0de5ca32415e3ee553f58997c23030f04e0fbbcb8bf39269d3ac24ae31fe
Sep 18 20:21:17.066769 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:21:17.066733    2549 generic.go:356] "Generic (PLEG): container finished" podID="3242aca7-e91c-4471-91ca-128800e6cdc0" containerID="0dfda88c801eb9b8e33096d7384c5047f2a7017381c3fe8ab9089b08885a3ac7" exitCode=0
Sep 18 20:21:17.067186 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:21:17.066813    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d49a30d-fp25j" event={"ID":"3242aca7-e91c-4471-91ca-128800e6cdc0","Type":"ContainerDied","Data":"0dfda88c801eb9b8e33096d7384c5047f2a7017381c3fe8ab9089b08885a3ac7"}
Sep 18 20:21:17.067186 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:21:17.066925    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d49a30d-fp25j" event={"ID":"3242aca7-e91c-4471-91ca-128800e6cdc0","Type":"ContainerStarted","Data":"6e8a0de5ca32415e3ee553f58997c23030f04e0fbbcb8bf39269d3ac24ae31fe"}
Sep 18 20:21:18.222455 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:21:18.222425    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d49a30d-fp25j"
Sep 18 20:21:18.246439 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:21:18.246369    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3242aca7-e91c-4471-91ca-128800e6cdc0-maas-api-service-ca\") pod \"3242aca7-e91c-4471-91ca-128800e6cdc0\" (UID: \"3242aca7-e91c-4471-91ca-128800e6cdc0\") "
Sep 18 20:21:18.246851 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:21:18.246821    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3242aca7-e91c-4471-91ca-128800e6cdc0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3242aca7-e91c-4471-91ca-128800e6cdc0" (UID: "3242aca7-e91c-4471-91ca-128800e6cdc0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:21:18.348374 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:21:18.348276    2549 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3242aca7-e91c-4471-91ca-128800e6cdc0-maas-api-service-ca\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:21:19.082628 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:21:19.082598    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d49a30d-fp25j"
Sep 18 20:21:19.082823 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:21:19.082643    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d49a30d-fp25j" event={"ID":"3242aca7-e91c-4471-91ca-128800e6cdc0","Type":"ContainerDied","Data":"6e8a0de5ca32415e3ee553f58997c23030f04e0fbbcb8bf39269d3ac24ae31fe"}
Sep 18 20:21:19.082823 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:21:19.082691    2549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e8a0de5ca32415e3ee553f58997c23030f04e0fbbcb8bf39269d3ac24ae31fe"
Sep 18 20:21:31.567157 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:21:31.567108    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d49a30d-fp25j"]
Sep 18 20:21:31.570276 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:21:31.570240    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d49a30d-fp25j"]
Sep 18 20:21:33.522345 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:21:33.522303    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3242aca7-e91c-4471-91ca-128800e6cdc0" path="/var/lib/kubelet/pods/3242aca7-e91c-4471-91ca-128800e6cdc0/volumes"
Sep 18 20:21:34.126448 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:21:34.126403    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:21:39.731812 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:21:39.731776    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-7d49a30d/e2e-lifecycle-model-7d49a30d-kserve-b887f9547-cjpx2"]
Sep 18 20:21:54.754165 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:21:54.754095    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-7d49a30d/e2e-lifecycle-model-7d49a30d-kserve-b887f9547-cjpx2" podUID="64ea02f9-ae44-4a3d-aeaa-8b041a5288ab" containerName="main" containerID="cri-o://56ffd66100de87bbb0931b6b2c04c7d4a7bde3637ef847a667f6f194e152c99b" gracePeriod=45
Sep 18 20:21:55.027986 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:21:55.027958    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7d49a30d/e2e-lifecycle-model-7d49a30d-kserve-b887f9547-cjpx2"
Sep 18 20:21:55.163894 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:21:55.163859    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64ea02f9-ae44-4a3d-aeaa-8b041a5288ab-home\") pod \"64ea02f9-ae44-4a3d-aeaa-8b041a5288ab\" (UID: \"64ea02f9-ae44-4a3d-aeaa-8b041a5288ab\") "
Sep 18 20:21:55.164123 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:21:55.163913    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64ea02f9-ae44-4a3d-aeaa-8b041a5288ab-model-cache\") pod \"64ea02f9-ae44-4a3d-aeaa-8b041a5288ab\" (UID: \"64ea02f9-ae44-4a3d-aeaa-8b041a5288ab\") "
Sep 18 20:21:55.164123 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:21:55.163987    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64ea02f9-ae44-4a3d-aeaa-8b041a5288ab-dshm\") pod \"64ea02f9-ae44-4a3d-aeaa-8b041a5288ab\" (UID: \"64ea02f9-ae44-4a3d-aeaa-8b041a5288ab\") "
Sep 18 20:21:55.164336 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:21:55.164134    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64ea02f9-ae44-4a3d-aeaa-8b041a5288ab-tmp-dir\") pod \"64ea02f9-ae44-4a3d-aeaa-8b041a5288ab\" (UID: \"64ea02f9-ae44-4a3d-aeaa-8b041a5288ab\") "
Sep 18 20:21:55.164336 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:21:55.164209    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zdxqk\" (UniqueName: \"kubernetes.io/projected/64ea02f9-ae44-4a3d-aeaa-8b041a5288ab-kube-api-access-zdxqk\") pod \"64ea02f9-ae44-4a3d-aeaa-8b041a5288ab\" (UID: \"64ea02f9-ae44-4a3d-aeaa-8b041a5288ab\") "
Sep 18 20:21:55.164336 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:21:55.164252    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64ea02f9-ae44-4a3d-aeaa-8b041a5288ab-tls-certs\") pod \"64ea02f9-ae44-4a3d-aeaa-8b041a5288ab\" (UID: \"64ea02f9-ae44-4a3d-aeaa-8b041a5288ab\") "
Sep 18 20:21:55.164336 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:21:55.164199    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64ea02f9-ae44-4a3d-aeaa-8b041a5288ab-model-cache" (OuterVolumeSpecName: "model-cache") pod "64ea02f9-ae44-4a3d-aeaa-8b041a5288ab" (UID: "64ea02f9-ae44-4a3d-aeaa-8b041a5288ab"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:21:55.164336 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:21:55.164214    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64ea02f9-ae44-4a3d-aeaa-8b041a5288ab-home" (OuterVolumeSpecName: "home") pod "64ea02f9-ae44-4a3d-aeaa-8b041a5288ab" (UID: "64ea02f9-ae44-4a3d-aeaa-8b041a5288ab"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:21:55.164651 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:21:55.164491    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64ea02f9-ae44-4a3d-aeaa-8b041a5288ab-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "64ea02f9-ae44-4a3d-aeaa-8b041a5288ab" (UID: "64ea02f9-ae44-4a3d-aeaa-8b041a5288ab"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:21:55.164739 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:21:55.164714    2549 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64ea02f9-ae44-4a3d-aeaa-8b041a5288ab-home\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:21:55.164809 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:21:55.164746    2549 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64ea02f9-ae44-4a3d-aeaa-8b041a5288ab-model-cache\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:21:55.164809 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:21:55.164762    2549 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64ea02f9-ae44-4a3d-aeaa-8b041a5288ab-tmp-dir\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:21:55.166773 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:21:55.166734    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64ea02f9-ae44-4a3d-aeaa-8b041a5288ab-dshm" (OuterVolumeSpecName: "dshm") pod "64ea02f9-ae44-4a3d-aeaa-8b041a5288ab" (UID: "64ea02f9-ae44-4a3d-aeaa-8b041a5288ab"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:21:55.166897 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:21:55.166790    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64ea02f9-ae44-4a3d-aeaa-8b041a5288ab-kube-api-access-zdxqk" (OuterVolumeSpecName: "kube-api-access-zdxqk") pod "64ea02f9-ae44-4a3d-aeaa-8b041a5288ab" (UID: "64ea02f9-ae44-4a3d-aeaa-8b041a5288ab"). InnerVolumeSpecName "kube-api-access-zdxqk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:21:55.166956 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:21:55.166933    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64ea02f9-ae44-4a3d-aeaa-8b041a5288ab-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "64ea02f9-ae44-4a3d-aeaa-8b041a5288ab" (UID: "64ea02f9-ae44-4a3d-aeaa-8b041a5288ab"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:21:55.265674 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:21:55.265568    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdxqk\" (UniqueName: \"kubernetes.io/projected/64ea02f9-ae44-4a3d-aeaa-8b041a5288ab-kube-api-access-zdxqk\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:21:55.265674 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:21:55.265608    2549 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64ea02f9-ae44-4a3d-aeaa-8b041a5288ab-tls-certs\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:21:55.265674 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:21:55.265619    2549 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64ea02f9-ae44-4a3d-aeaa-8b041a5288ab-dshm\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:21:55.343576 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:21:55.343538    2549 generic.go:356] "Generic (PLEG): container finished" podID="64ea02f9-ae44-4a3d-aeaa-8b041a5288ab" containerID="56ffd66100de87bbb0931b6b2c04c7d4a7bde3637ef847a667f6f194e152c99b" exitCode=0
Sep 18 20:21:55.343765 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:21:55.343589    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7d49a30d/e2e-lifecycle-model-7d49a30d-kserve-b887f9547-cjpx2"
Sep 18 20:21:55.343765 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:21:55.343624    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7d49a30d/e2e-lifecycle-model-7d49a30d-kserve-b887f9547-cjpx2" event={"ID":"64ea02f9-ae44-4a3d-aeaa-8b041a5288ab","Type":"ContainerDied","Data":"56ffd66100de87bbb0931b6b2c04c7d4a7bde3637ef847a667f6f194e152c99b"}
Sep 18 20:21:55.343765 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:21:55.343677    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7d49a30d/e2e-lifecycle-model-7d49a30d-kserve-b887f9547-cjpx2" event={"ID":"64ea02f9-ae44-4a3d-aeaa-8b041a5288ab","Type":"ContainerDied","Data":"28d12cdfe4e93b5f092802a4920514d19b0e04537481268a917f908823c26292"}
Sep 18 20:21:55.343765 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:21:55.343699    2549 scope.go:117] "RemoveContainer" containerID="56ffd66100de87bbb0931b6b2c04c7d4a7bde3637ef847a667f6f194e152c99b"
Sep 18 20:21:55.356117 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:21:55.356096    2549 scope.go:117] "RemoveContainer" containerID="56ffd66100de87bbb0931b6b2c04c7d4a7bde3637ef847a667f6f194e152c99b"
Sep 18 20:21:55.356489 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:21:55.356467    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56ffd66100de87bbb0931b6b2c04c7d4a7bde3637ef847a667f6f194e152c99b\": container with ID starting with 56ffd66100de87bbb0931b6b2c04c7d4a7bde3637ef847a667f6f194e152c99b not found: ID does not exist" containerID="56ffd66100de87bbb0931b6b2c04c7d4a7bde3637ef847a667f6f194e152c99b"
Sep 18 20:21:55.356612 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:21:55.356504    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56ffd66100de87bbb0931b6b2c04c7d4a7bde3637ef847a667f6f194e152c99b"} err="failed to get container status \"56ffd66100de87bbb0931b6b2c04c7d4a7bde3637ef847a667f6f194e152c99b\": rpc error: code = NotFound desc = could not find container \"56ffd66100de87bbb0931b6b2c04c7d4a7bde3637ef847a667f6f194e152c99b\": container with ID starting with 56ffd66100de87bbb0931b6b2c04c7d4a7bde3637ef847a667f6f194e152c99b not found: ID does not exist"
Sep 18 20:21:55.369499 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:21:55.369458    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-7d49a30d/e2e-lifecycle-model-7d49a30d-kserve-b887f9547-cjpx2"]
Sep 18 20:21:55.376545 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:21:55.376511    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-7d49a30d/e2e-lifecycle-model-7d49a30d-kserve-b887f9547-cjpx2"]
Sep 18 20:21:55.523318 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:21:55.523227    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64ea02f9-ae44-4a3d-aeaa-8b041a5288ab" path="/var/lib/kubelet/pods/64ea02f9-ae44-4a3d-aeaa-8b041a5288ab/volumes"
Sep 18 20:21:55.568718 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:21:55.568687    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-scn7p_a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48/ovn-acl-logging/0.log"
Sep 18 20:21:55.568902 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:21:55.568811    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-scn7p_a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48/ovn-acl-logging/0.log"
Sep 18 20:21:55.805082 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:21:55.804999    2549 scope.go:117] "RemoveContainer" containerID="bc4b5659567b5458a457c0200342c3abf07ca2cf691d5533901aaf3726d96f7d"
Sep 18 20:21:59.215284 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:21:59.215236    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-5284dc-r2v5g"]
Sep 18 20:21:59.215721 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:21:59.215706    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3242aca7-e91c-4471-91ca-128800e6cdc0" containerName="revoke-keys"
Sep 18 20:21:59.215721 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:21:59.215722    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="3242aca7-e91c-4471-91ca-128800e6cdc0" containerName="revoke-keys"
Sep 18 20:21:59.215798 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:21:59.215737    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64ea02f9-ae44-4a3d-aeaa-8b041a5288ab" containerName="main"
Sep 18 20:21:59.215798 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:21:59.215743    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="64ea02f9-ae44-4a3d-aeaa-8b041a5288ab" containerName="main"
Sep 18 20:21:59.215859 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:21:59.215817    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="64ea02f9-ae44-4a3d-aeaa-8b041a5288ab" containerName="main"
Sep 18 20:21:59.215859 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:21:59.215825    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="3242aca7-e91c-4471-91ca-128800e6cdc0" containerName="revoke-keys"
Sep 18 20:21:59.219022 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:21:59.219001    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-5284dc-r2v5g"
Sep 18 20:21:59.222201 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:21:59.222177    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bhssz\""
Sep 18 20:21:59.222584 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:21:59.222564    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 20:21:59.233462 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:21:59.233432    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-5284dc-r2v5g"]
Sep 18 20:21:59.315932 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:21:59.315870    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9ea7fe1-104f-40e1-b269-889ce292658a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-5284dc-r2v5g\" (UID: \"e9ea7fe1-104f-40e1-b269-889ce292658a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-5284dc-r2v5g"
Sep 18 20:21:59.417504 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:21:59.417460    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9ea7fe1-104f-40e1-b269-889ce292658a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-5284dc-r2v5g\" (UID: \"e9ea7fe1-104f-40e1-b269-889ce292658a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-5284dc-r2v5g"
Sep 18 20:21:59.418144 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:21:59.418123    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9ea7fe1-104f-40e1-b269-889ce292658a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-5284dc-r2v5g\" (UID: \"e9ea7fe1-104f-40e1-b269-889ce292658a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-5284dc-r2v5g"
Sep 18 20:21:59.530556 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:21:59.530457    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-5284dc-r2v5g"
Sep 18 20:21:59.810115 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:21:59.810083    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-5284dc-r2v5g"]
Sep 18 20:21:59.811378 ip-10-0-140-178 kubenswrapper[2549]: W0918 20:21:59.811344    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9ea7fe1_104f_40e1_b269_889ce292658a.slice/crio-fc5e7131484341c177604e012f42c956aa98b3e01156b2bb7eaef22eff8d98f6 WatchSource:0}: Error finding container fc5e7131484341c177604e012f42c956aa98b3e01156b2bb7eaef22eff8d98f6: Status 404 returned error can't find the container with id fc5e7131484341c177604e012f42c956aa98b3e01156b2bb7eaef22eff8d98f6
Sep 18 20:22:00.379110 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:00.379060    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-5284dc-r2v5g" event={"ID":"e9ea7fe1-104f-40e1-b269-889ce292658a","Type":"ContainerStarted","Data":"63efd62c4502c7ec637dca9da28258a5bc64b404fbba78a6e2c666947e0878f3"}
Sep 18 20:22:00.379591 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:00.379134    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-5284dc-r2v5g" event={"ID":"e9ea7fe1-104f-40e1-b269-889ce292658a","Type":"ContainerStarted","Data":"fc5e7131484341c177604e012f42c956aa98b3e01156b2bb7eaef22eff8d98f6"}
Sep 18 20:22:01.387928 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:01.387891    2549 generic.go:356] "Generic (PLEG): container finished" podID="e9ea7fe1-104f-40e1-b269-889ce292658a" containerID="63efd62c4502c7ec637dca9da28258a5bc64b404fbba78a6e2c666947e0878f3" exitCode=0
Sep 18 20:22:01.388349 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:01.387962    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-5284dc-r2v5g" event={"ID":"e9ea7fe1-104f-40e1-b269-889ce292658a","Type":"ContainerDied","Data":"63efd62c4502c7ec637dca9da28258a5bc64b404fbba78a6e2c666947e0878f3"}
Sep 18 20:22:02.548183 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:02.548148    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-5284dc-r2v5g"
Sep 18 20:22:02.652791 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:02.652697    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9ea7fe1-104f-40e1-b269-889ce292658a-maas-api-service-ca\") pod \"e9ea7fe1-104f-40e1-b269-889ce292658a\" (UID: \"e9ea7fe1-104f-40e1-b269-889ce292658a\") "
Sep 18 20:22:02.653150 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:02.653114    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9ea7fe1-104f-40e1-b269-889ce292658a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e9ea7fe1-104f-40e1-b269-889ce292658a" (UID: "e9ea7fe1-104f-40e1-b269-889ce292658a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:22:02.653582 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:02.653562    2549 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9ea7fe1-104f-40e1-b269-889ce292658a-maas-api-service-ca\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:22:02.713505 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:02.713464    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:22:03.410931 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:03.410896    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-5284dc-r2v5g"
Sep 18 20:22:03.411124 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:03.410937    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-5284dc-r2v5g" event={"ID":"e9ea7fe1-104f-40e1-b269-889ce292658a","Type":"ContainerDied","Data":"fc5e7131484341c177604e012f42c956aa98b3e01156b2bb7eaef22eff8d98f6"}
Sep 18 20:22:03.411124 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:03.410985    2549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc5e7131484341c177604e012f42c956aa98b3e01156b2bb7eaef22eff8d98f6"
Sep 18 20:22:04.087451 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:04.087405    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-5284dc/simulator-w0-5284dc-kserve-84dc7dc7f-mpwht"]
Sep 18 20:22:15.143969 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:15.143920    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-5284dc-r2v5g"]
Sep 18 20:22:15.148446 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:15.148402    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-5284dc-r2v5g"]
Sep 18 20:22:15.522677 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:15.522589    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9ea7fe1-104f-40e1-b269-889ce292658a" path="/var/lib/kubelet/pods/e9ea7fe1-104f-40e1-b269-889ce292658a/volumes"
Sep 18 20:22:19.109428 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:19.109318    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-5284dc/simulator-w0-5284dc-kserve-84dc7dc7f-mpwht" podUID="daf3d095-d0b4-462f-b05d-0a699a8eb26c" containerName="main" containerID="cri-o://a66e0dd6332a112e76bcdcf3d5edc88fdcbc315b2fe24c835a9c3c2bfe2a1818" gracePeriod=45
Sep 18 20:22:19.381471 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:19.381442    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-5284dc/simulator-w0-5284dc-kserve-84dc7dc7f-mpwht"
Sep 18 20:22:19.436075 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:19.436038    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/daf3d095-d0b4-462f-b05d-0a699a8eb26c-tmp-dir\") pod \"daf3d095-d0b4-462f-b05d-0a699a8eb26c\" (UID: \"daf3d095-d0b4-462f-b05d-0a699a8eb26c\") "
Sep 18 20:22:19.436282 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:19.436094    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wwrkb\" (UniqueName: \"kubernetes.io/projected/daf3d095-d0b4-462f-b05d-0a699a8eb26c-kube-api-access-wwrkb\") pod \"daf3d095-d0b4-462f-b05d-0a699a8eb26c\" (UID: \"daf3d095-d0b4-462f-b05d-0a699a8eb26c\") "
Sep 18 20:22:19.436282 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:19.436132    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/daf3d095-d0b4-462f-b05d-0a699a8eb26c-home\") pod \"daf3d095-d0b4-462f-b05d-0a699a8eb26c\" (UID: \"daf3d095-d0b4-462f-b05d-0a699a8eb26c\") "
Sep 18 20:22:19.436282 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:19.436209    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/daf3d095-d0b4-462f-b05d-0a699a8eb26c-model-cache\") pod \"daf3d095-d0b4-462f-b05d-0a699a8eb26c\" (UID: \"daf3d095-d0b4-462f-b05d-0a699a8eb26c\") "
Sep 18 20:22:19.436499 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:19.436283    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/daf3d095-d0b4-462f-b05d-0a699a8eb26c-tls-certs\") pod \"daf3d095-d0b4-462f-b05d-0a699a8eb26c\" (UID: \"daf3d095-d0b4-462f-b05d-0a699a8eb26c\") "
Sep 18 20:22:19.436499 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:19.436328    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/daf3d095-d0b4-462f-b05d-0a699a8eb26c-dshm\") pod \"daf3d095-d0b4-462f-b05d-0a699a8eb26c\" (UID: \"daf3d095-d0b4-462f-b05d-0a699a8eb26c\") "
Sep 18 20:22:19.436499 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:19.436348    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daf3d095-d0b4-462f-b05d-0a699a8eb26c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "daf3d095-d0b4-462f-b05d-0a699a8eb26c" (UID: "daf3d095-d0b4-462f-b05d-0a699a8eb26c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:22:19.436647 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:19.436592    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daf3d095-d0b4-462f-b05d-0a699a8eb26c-model-cache" (OuterVolumeSpecName: "model-cache") pod "daf3d095-d0b4-462f-b05d-0a699a8eb26c" (UID: "daf3d095-d0b4-462f-b05d-0a699a8eb26c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:22:19.436780 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:19.436755    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daf3d095-d0b4-462f-b05d-0a699a8eb26c-home" (OuterVolumeSpecName: "home") pod "daf3d095-d0b4-462f-b05d-0a699a8eb26c" (UID: "daf3d095-d0b4-462f-b05d-0a699a8eb26c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:22:19.436780 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:19.436773    2549 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/daf3d095-d0b4-462f-b05d-0a699a8eb26c-model-cache\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:22:19.436886 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:19.436791    2549 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/daf3d095-d0b4-462f-b05d-0a699a8eb26c-tmp-dir\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:22:19.438747 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:19.438722    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/daf3d095-d0b4-462f-b05d-0a699a8eb26c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "daf3d095-d0b4-462f-b05d-0a699a8eb26c" (UID: "daf3d095-d0b4-462f-b05d-0a699a8eb26c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:22:19.438974 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:19.438949    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daf3d095-d0b4-462f-b05d-0a699a8eb26c-kube-api-access-wwrkb" (OuterVolumeSpecName: "kube-api-access-wwrkb") pod "daf3d095-d0b4-462f-b05d-0a699a8eb26c" (UID: "daf3d095-d0b4-462f-b05d-0a699a8eb26c"). InnerVolumeSpecName "kube-api-access-wwrkb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:22:19.439043 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:19.438949    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daf3d095-d0b4-462f-b05d-0a699a8eb26c-dshm" (OuterVolumeSpecName: "dshm") pod "daf3d095-d0b4-462f-b05d-0a699a8eb26c" (UID: "daf3d095-d0b4-462f-b05d-0a699a8eb26c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:22:19.513941 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:19.513900    2549 generic.go:356] "Generic (PLEG): container finished" podID="daf3d095-d0b4-462f-b05d-0a699a8eb26c" containerID="a66e0dd6332a112e76bcdcf3d5edc88fdcbc315b2fe24c835a9c3c2bfe2a1818" exitCode=0
Sep 18 20:22:19.513941 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:19.513946    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-5284dc/simulator-w0-5284dc-kserve-84dc7dc7f-mpwht"
Sep 18 20:22:19.514167 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:19.513988    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-5284dc/simulator-w0-5284dc-kserve-84dc7dc7f-mpwht" event={"ID":"daf3d095-d0b4-462f-b05d-0a699a8eb26c","Type":"ContainerDied","Data":"a66e0dd6332a112e76bcdcf3d5edc88fdcbc315b2fe24c835a9c3c2bfe2a1818"}
Sep 18 20:22:19.514167 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:19.514033    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-5284dc/simulator-w0-5284dc-kserve-84dc7dc7f-mpwht" event={"ID":"daf3d095-d0b4-462f-b05d-0a699a8eb26c","Type":"ContainerDied","Data":"4a1e984fd07f3cf59d5eff4946fefe9e8843788e707729101130fe2eeb1dfa30"}
Sep 18 20:22:19.514167 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:19.514053    2549 scope.go:117] "RemoveContainer" containerID="a66e0dd6332a112e76bcdcf3d5edc88fdcbc315b2fe24c835a9c3c2bfe2a1818"
Sep 18 20:22:19.531693 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:19.531661    2549 scope.go:117] "RemoveContainer" containerID="a66e0dd6332a112e76bcdcf3d5edc88fdcbc315b2fe24c835a9c3c2bfe2a1818"
Sep 18 20:22:19.532276 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:22:19.532244    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a66e0dd6332a112e76bcdcf3d5edc88fdcbc315b2fe24c835a9c3c2bfe2a1818\": container with ID starting with a66e0dd6332a112e76bcdcf3d5edc88fdcbc315b2fe24c835a9c3c2bfe2a1818 not found: ID does not exist" containerID="a66e0dd6332a112e76bcdcf3d5edc88fdcbc315b2fe24c835a9c3c2bfe2a1818"
Sep 18 20:22:19.532452 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:19.532287    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a66e0dd6332a112e76bcdcf3d5edc88fdcbc315b2fe24c835a9c3c2bfe2a1818"} err="failed to get container status \"a66e0dd6332a112e76bcdcf3d5edc88fdcbc315b2fe24c835a9c3c2bfe2a1818\": rpc error: code = NotFound desc = could not find container \"a66e0dd6332a112e76bcdcf3d5edc88fdcbc315b2fe24c835a9c3c2bfe2a1818\": container with ID starting with a66e0dd6332a112e76bcdcf3d5edc88fdcbc315b2fe24c835a9c3c2bfe2a1818 not found: ID does not exist"
Sep 18 20:22:19.538196 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:19.538162    2549 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/daf3d095-d0b4-462f-b05d-0a699a8eb26c-tls-certs\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:22:19.538196 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:19.538193    2549 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/daf3d095-d0b4-462f-b05d-0a699a8eb26c-dshm\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:22:19.538373 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:19.538205    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwrkb\" (UniqueName: \"kubernetes.io/projected/daf3d095-d0b4-462f-b05d-0a699a8eb26c-kube-api-access-wwrkb\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:22:19.538373 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:19.538218    2549 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/daf3d095-d0b4-462f-b05d-0a699a8eb26c-home\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:22:19.548176 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:19.548137    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-5284dc/simulator-w0-5284dc-kserve-84dc7dc7f-mpwht"]
Sep 18 20:22:19.555148 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:19.555108    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-5284dc/simulator-w0-5284dc-kserve-84dc7dc7f-mpwht"]
Sep 18 20:22:21.522147 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:21.522104    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="daf3d095-d0b4-462f-b05d-0a699a8eb26c" path="/var/lib/kubelet/pods/daf3d095-d0b4-462f-b05d-0a699a8eb26c/volumes"
Sep 18 20:22:22.316455 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:22.316414    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr"]
Sep 18 20:22:22.316682 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:22.316659    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr" podUID="374c8e70-b164-4b98-8c5d-f7994db126f2" containerName="istio-proxy" containerID="cri-o://92a6cfb5fdec34475d9f59d4357957edbc092711a9b5e350d2e82e3a3b91c183" gracePeriod=30
Sep 18 20:22:24.903172 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:24.903137    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4"]
Sep 18 20:22:24.903659 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:24.903402    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4" podUID="0658bf28-e97b-4d0d-945e-2f43253220b6" containerName="istio-proxy" containerID="cri-o://fbb6f9623fd80f82cc90e6bd50ac2aa9c5065a74fb4516a0cadcdbb31b21d02e" gracePeriod=30
Sep 18 20:22:25.524895 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:25.524844    2549 patch_prober.go:28] interesting pod/e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 18 20:22:25.525100 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:25.524924    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr" podUID="374c8e70-b164-4b98-8c5d-f7994db126f2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:22:25.815049 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:25.814941    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:22:27.580770 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:27.580731    2549 generic.go:356] "Generic (PLEG): container finished" podID="374c8e70-b164-4b98-8c5d-f7994db126f2" containerID="92a6cfb5fdec34475d9f59d4357957edbc092711a9b5e350d2e82e3a3b91c183" exitCode=0
Sep 18 20:22:27.581224 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:27.580802    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr" event={"ID":"374c8e70-b164-4b98-8c5d-f7994db126f2","Type":"ContainerDied","Data":"92a6cfb5fdec34475d9f59d4357957edbc092711a9b5e350d2e82e3a3b91c183"}
Sep 18 20:22:27.581224 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:27.580877    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr" event={"ID":"374c8e70-b164-4b98-8c5d-f7994db126f2","Type":"ContainerDied","Data":"0ea2e05bdda724e84ab7dd16702f8d4ef9508aed5079adbc9cb7f5afb6238d40"}
Sep 18 20:22:27.581224 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:27.580910    2549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ea2e05bdda724e84ab7dd16702f8d4ef9508aed5079adbc9cb7f5afb6238d40"
Sep 18 20:22:27.581224 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:27.580931    2549 scope.go:117] "RemoveContainer" containerID="72d061c1a843da03a5e6768c6bae567b761ff73cfe8163e9d982404a1b36b202"
Sep 18 20:22:27.595412 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:27.595368    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr"
Sep 18 20:22:27.725969 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:27.725923    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/374c8e70-b164-4b98-8c5d-f7994db126f2-istio-token\") pod \"374c8e70-b164-4b98-8c5d-f7994db126f2\" (UID: \"374c8e70-b164-4b98-8c5d-f7994db126f2\") "
Sep 18 20:22:27.725969 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:27.725975    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/374c8e70-b164-4b98-8c5d-f7994db126f2-istio-data\") pod \"374c8e70-b164-4b98-8c5d-f7994db126f2\" (UID: \"374c8e70-b164-4b98-8c5d-f7994db126f2\") "
Sep 18 20:22:27.726254 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:27.726017    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6cp6t\" (UniqueName: \"kubernetes.io/projected/374c8e70-b164-4b98-8c5d-f7994db126f2-kube-api-access-6cp6t\") pod \"374c8e70-b164-4b98-8c5d-f7994db126f2\" (UID: \"374c8e70-b164-4b98-8c5d-f7994db126f2\") "
Sep 18 20:22:27.726254 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:27.726039    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/374c8e70-b164-4b98-8c5d-f7994db126f2-istio-envoy\") pod \"374c8e70-b164-4b98-8c5d-f7994db126f2\" (UID: \"374c8e70-b164-4b98-8c5d-f7994db126f2\") "
Sep 18 20:22:27.726254 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:27.726136    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/374c8e70-b164-4b98-8c5d-f7994db126f2-istiod-ca-cert\") pod \"374c8e70-b164-4b98-8c5d-f7994db126f2\" (UID: \"374c8e70-b164-4b98-8c5d-f7994db126f2\") "
Sep 18 20:22:27.726254 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:27.726160    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/374c8e70-b164-4b98-8c5d-f7994db126f2-workload-socket\") pod \"374c8e70-b164-4b98-8c5d-f7994db126f2\" (UID: \"374c8e70-b164-4b98-8c5d-f7994db126f2\") "
Sep 18 20:22:27.726254 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:27.726185    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/374c8e70-b164-4b98-8c5d-f7994db126f2-workload-certs\") pod \"374c8e70-b164-4b98-8c5d-f7994db126f2\" (UID: \"374c8e70-b164-4b98-8c5d-f7994db126f2\") "
Sep 18 20:22:27.726254 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:27.726225    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/374c8e70-b164-4b98-8c5d-f7994db126f2-credential-socket\") pod \"374c8e70-b164-4b98-8c5d-f7994db126f2\" (UID: \"374c8e70-b164-4b98-8c5d-f7994db126f2\") "
Sep 18 20:22:27.726636 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:27.726279    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/374c8e70-b164-4b98-8c5d-f7994db126f2-istio-podinfo\") pod \"374c8e70-b164-4b98-8c5d-f7994db126f2\" (UID: \"374c8e70-b164-4b98-8c5d-f7994db126f2\") "
Sep 18 20:22:27.726636 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:27.726444    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/374c8e70-b164-4b98-8c5d-f7994db126f2-istio-data" (OuterVolumeSpecName: "istio-data") pod "374c8e70-b164-4b98-8c5d-f7994db126f2" (UID: "374c8e70-b164-4b98-8c5d-f7994db126f2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:22:27.726636 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:27.726515    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/374c8e70-b164-4b98-8c5d-f7994db126f2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "374c8e70-b164-4b98-8c5d-f7994db126f2" (UID: "374c8e70-b164-4b98-8c5d-f7994db126f2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:22:27.726636 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:27.726529    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/374c8e70-b164-4b98-8c5d-f7994db126f2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "374c8e70-b164-4b98-8c5d-f7994db126f2" (UID: "374c8e70-b164-4b98-8c5d-f7994db126f2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:22:27.726842 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:27.726672    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/374c8e70-b164-4b98-8c5d-f7994db126f2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "374c8e70-b164-4b98-8c5d-f7994db126f2" (UID: "374c8e70-b164-4b98-8c5d-f7994db126f2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:22:27.726894 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:27.726862    2549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/374c8e70-b164-4b98-8c5d-f7994db126f2-istio-data\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:22:27.726894 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:27.726883    2549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/374c8e70-b164-4b98-8c5d-f7994db126f2-workload-socket\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:22:27.726981 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:27.726897    2549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/374c8e70-b164-4b98-8c5d-f7994db126f2-workload-certs\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:22:27.726981 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:27.726911    2549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/374c8e70-b164-4b98-8c5d-f7994db126f2-credential-socket\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:22:27.726981 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:27.726923    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/374c8e70-b164-4b98-8c5d-f7994db126f2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "374c8e70-b164-4b98-8c5d-f7994db126f2" (UID: "374c8e70-b164-4b98-8c5d-f7994db126f2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:22:27.728444 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:27.728412    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/374c8e70-b164-4b98-8c5d-f7994db126f2-kube-api-access-6cp6t" (OuterVolumeSpecName: "kube-api-access-6cp6t") pod "374c8e70-b164-4b98-8c5d-f7994db126f2" (UID: "374c8e70-b164-4b98-8c5d-f7994db126f2"). InnerVolumeSpecName "kube-api-access-6cp6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:22:27.728609 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:27.728584    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/374c8e70-b164-4b98-8c5d-f7994db126f2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "374c8e70-b164-4b98-8c5d-f7994db126f2" (UID: "374c8e70-b164-4b98-8c5d-f7994db126f2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 20:22:27.728688 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:27.728671    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/374c8e70-b164-4b98-8c5d-f7994db126f2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "374c8e70-b164-4b98-8c5d-f7994db126f2" (UID: "374c8e70-b164-4b98-8c5d-f7994db126f2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:22:27.728736 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:27.728720    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/374c8e70-b164-4b98-8c5d-f7994db126f2-istio-token" (OuterVolumeSpecName: "istio-token") pod "374c8e70-b164-4b98-8c5d-f7994db126f2" (UID: "374c8e70-b164-4b98-8c5d-f7994db126f2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:22:27.828158 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:27.828097    2549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/374c8e70-b164-4b98-8c5d-f7994db126f2-istiod-ca-cert\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:22:27.828158 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:27.828150    2549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/374c8e70-b164-4b98-8c5d-f7994db126f2-istio-podinfo\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:22:27.828158 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:27.828161    2549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/374c8e70-b164-4b98-8c5d-f7994db126f2-istio-token\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:22:27.828158 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:27.828170    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cp6t\" (UniqueName: \"kubernetes.io/projected/374c8e70-b164-4b98-8c5d-f7994db126f2-kube-api-access-6cp6t\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:22:27.828158 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:27.828182    2549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/374c8e70-b164-4b98-8c5d-f7994db126f2-istio-envoy\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:22:28.588844 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:28.588810    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr"
Sep 18 20:22:28.658329 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:28.658286    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr"]
Sep 18 20:22:28.661214 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:28.661182    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-5284dc-openshift-default-6797c7d96-ljkmr"]
Sep 18 20:22:29.522399 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:29.522356    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="374c8e70-b164-4b98-8c5d-f7994db126f2" path="/var/lib/kubelet/pods/374c8e70-b164-4b98-8c5d-f7994db126f2/volumes"
Sep 18 20:22:30.183768 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:30.183692    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4"
Sep 18 20:22:30.251658 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:30.251622    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0658bf28-e97b-4d0d-945e-2f43253220b6-workload-socket\") pod \"0658bf28-e97b-4d0d-945e-2f43253220b6\" (UID: \"0658bf28-e97b-4d0d-945e-2f43253220b6\") "
Sep 18 20:22:30.251858 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:30.251789    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z6pgx\" (UniqueName: \"kubernetes.io/projected/0658bf28-e97b-4d0d-945e-2f43253220b6-kube-api-access-z6pgx\") pod \"0658bf28-e97b-4d0d-945e-2f43253220b6\" (UID: \"0658bf28-e97b-4d0d-945e-2f43253220b6\") "
Sep 18 20:22:30.251929 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:30.251856    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0658bf28-e97b-4d0d-945e-2f43253220b6-credential-socket\") pod \"0658bf28-e97b-4d0d-945e-2f43253220b6\" (UID: \"0658bf28-e97b-4d0d-945e-2f43253220b6\") "
Sep 18 20:22:30.251929 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:30.251873    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0658bf28-e97b-4d0d-945e-2f43253220b6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0658bf28-e97b-4d0d-945e-2f43253220b6" (UID: "0658bf28-e97b-4d0d-945e-2f43253220b6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:22:30.251929 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:30.251910    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0658bf28-e97b-4d0d-945e-2f43253220b6-istio-envoy\") pod \"0658bf28-e97b-4d0d-945e-2f43253220b6\" (UID: \"0658bf28-e97b-4d0d-945e-2f43253220b6\") "
Sep 18 20:22:30.252084 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:30.251964    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0658bf28-e97b-4d0d-945e-2f43253220b6-istiod-ca-cert\") pod \"0658bf28-e97b-4d0d-945e-2f43253220b6\" (UID: \"0658bf28-e97b-4d0d-945e-2f43253220b6\") "
Sep 18 20:22:30.252084 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:30.251996    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0658bf28-e97b-4d0d-945e-2f43253220b6-workload-certs\") pod \"0658bf28-e97b-4d0d-945e-2f43253220b6\" (UID: \"0658bf28-e97b-4d0d-945e-2f43253220b6\") "
Sep 18 20:22:30.252084 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:30.252026    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0658bf28-e97b-4d0d-945e-2f43253220b6-istio-podinfo\") pod \"0658bf28-e97b-4d0d-945e-2f43253220b6\" (UID: \"0658bf28-e97b-4d0d-945e-2f43253220b6\") "
Sep 18 20:22:30.252084 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:30.252061    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0658bf28-e97b-4d0d-945e-2f43253220b6-istio-data\") pod \"0658bf28-e97b-4d0d-945e-2f43253220b6\" (UID: \"0658bf28-e97b-4d0d-945e-2f43253220b6\") "
Sep 18 20:22:30.252284 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:30.252115    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0658bf28-e97b-4d0d-945e-2f43253220b6-istio-token\") pod \"0658bf28-e97b-4d0d-945e-2f43253220b6\" (UID: \"0658bf28-e97b-4d0d-945e-2f43253220b6\") "
Sep 18 20:22:30.252284 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:30.252181    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0658bf28-e97b-4d0d-945e-2f43253220b6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0658bf28-e97b-4d0d-945e-2f43253220b6" (UID: "0658bf28-e97b-4d0d-945e-2f43253220b6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:22:30.252513 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:30.252441    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0658bf28-e97b-4d0d-945e-2f43253220b6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0658bf28-e97b-4d0d-945e-2f43253220b6" (UID: "0658bf28-e97b-4d0d-945e-2f43253220b6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:22:30.252513 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:30.252480    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0658bf28-e97b-4d0d-945e-2f43253220b6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0658bf28-e97b-4d0d-945e-2f43253220b6" (UID: "0658bf28-e97b-4d0d-945e-2f43253220b6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:22:30.252728 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:30.252622    2549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0658bf28-e97b-4d0d-945e-2f43253220b6-workload-socket\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:22:30.252728 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:30.252642    2549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0658bf28-e97b-4d0d-945e-2f43253220b6-credential-socket\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:22:30.252728 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:30.252658    2549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0658bf28-e97b-4d0d-945e-2f43253220b6-istiod-ca-cert\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:22:30.252728 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:30.252674    2549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0658bf28-e97b-4d0d-945e-2f43253220b6-workload-certs\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:22:30.253009 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:30.252988    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0658bf28-e97b-4d0d-945e-2f43253220b6-istio-data" (OuterVolumeSpecName: "istio-data") pod "0658bf28-e97b-4d0d-945e-2f43253220b6" (UID: "0658bf28-e97b-4d0d-945e-2f43253220b6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:22:30.254671 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:30.254633    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0658bf28-e97b-4d0d-945e-2f43253220b6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0658bf28-e97b-4d0d-945e-2f43253220b6" (UID: "0658bf28-e97b-4d0d-945e-2f43253220b6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 20:22:30.254777 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:30.254704    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0658bf28-e97b-4d0d-945e-2f43253220b6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0658bf28-e97b-4d0d-945e-2f43253220b6" (UID: "0658bf28-e97b-4d0d-945e-2f43253220b6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:22:30.254777 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:30.254743    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0658bf28-e97b-4d0d-945e-2f43253220b6-kube-api-access-z6pgx" (OuterVolumeSpecName: "kube-api-access-z6pgx") pod "0658bf28-e97b-4d0d-945e-2f43253220b6" (UID: "0658bf28-e97b-4d0d-945e-2f43253220b6"). InnerVolumeSpecName "kube-api-access-z6pgx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:22:30.254915 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:30.254896    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0658bf28-e97b-4d0d-945e-2f43253220b6-istio-token" (OuterVolumeSpecName: "istio-token") pod "0658bf28-e97b-4d0d-945e-2f43253220b6" (UID: "0658bf28-e97b-4d0d-945e-2f43253220b6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:22:30.354232 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:30.354193    2549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0658bf28-e97b-4d0d-945e-2f43253220b6-istio-podinfo\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:22:30.354232 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:30.354224    2549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0658bf28-e97b-4d0d-945e-2f43253220b6-istio-data\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:22:30.354232 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:30.354236    2549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0658bf28-e97b-4d0d-945e-2f43253220b6-istio-token\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:22:30.354535 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:30.354249    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6pgx\" (UniqueName: \"kubernetes.io/projected/0658bf28-e97b-4d0d-945e-2f43253220b6-kube-api-access-z6pgx\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:22:30.354535 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:30.354261    2549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0658bf28-e97b-4d0d-945e-2f43253220b6-istio-envoy\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:22:30.604416 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:30.604363    2549 generic.go:356] "Generic (PLEG): container finished" podID="0658bf28-e97b-4d0d-945e-2f43253220b6" containerID="fbb6f9623fd80f82cc90e6bd50ac2aa9c5065a74fb4516a0cadcdbb31b21d02e" exitCode=0
Sep 18 20:22:30.604603 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:30.604419    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4" event={"ID":"0658bf28-e97b-4d0d-945e-2f43253220b6","Type":"ContainerDied","Data":"fbb6f9623fd80f82cc90e6bd50ac2aa9c5065a74fb4516a0cadcdbb31b21d02e"}
Sep 18 20:22:30.604603 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:30.604492    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4" event={"ID":"0658bf28-e97b-4d0d-945e-2f43253220b6","Type":"ContainerDied","Data":"0af0d94d14e5ae58b9a20c5e912201055d2cf5ba23072e9e5d291e5aeb169bb5"}
Sep 18 20:22:30.604603 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:30.604507    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4"
Sep 18 20:22:30.604603 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:30.604518    2549 scope.go:117] "RemoveContainer" containerID="fbb6f9623fd80f82cc90e6bd50ac2aa9c5065a74fb4516a0cadcdbb31b21d02e"
Sep 18 20:22:30.615786 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:30.615764    2549 scope.go:117] "RemoveContainer" containerID="c3e7cd13d085eddc5f8e3ccacbb7de0601518bcc35bae99b48523cc0f261f96b"
Sep 18 20:22:30.625563 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:30.625538    2549 scope.go:117] "RemoveContainer" containerID="fbb6f9623fd80f82cc90e6bd50ac2aa9c5065a74fb4516a0cadcdbb31b21d02e"
Sep 18 20:22:30.625912 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:22:30.625892    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbb6f9623fd80f82cc90e6bd50ac2aa9c5065a74fb4516a0cadcdbb31b21d02e\": container with ID starting with fbb6f9623fd80f82cc90e6bd50ac2aa9c5065a74fb4516a0cadcdbb31b21d02e not found: ID does not exist" containerID="fbb6f9623fd80f82cc90e6bd50ac2aa9c5065a74fb4516a0cadcdbb31b21d02e"
Sep 18 20:22:30.625958 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:30.625922    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbb6f9623fd80f82cc90e6bd50ac2aa9c5065a74fb4516a0cadcdbb31b21d02e"} err="failed to get container status \"fbb6f9623fd80f82cc90e6bd50ac2aa9c5065a74fb4516a0cadcdbb31b21d02e\": rpc error: code = NotFound desc = could not find container \"fbb6f9623fd80f82cc90e6bd50ac2aa9c5065a74fb4516a0cadcdbb31b21d02e\": container with ID starting with fbb6f9623fd80f82cc90e6bd50ac2aa9c5065a74fb4516a0cadcdbb31b21d02e not found: ID does not exist"
Sep 18 20:22:30.625958 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:30.625940    2549 scope.go:117] "RemoveContainer" containerID="c3e7cd13d085eddc5f8e3ccacbb7de0601518bcc35bae99b48523cc0f261f96b"
Sep 18 20:22:30.626210 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:22:30.626194    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3e7cd13d085eddc5f8e3ccacbb7de0601518bcc35bae99b48523cc0f261f96b\": container with ID starting with c3e7cd13d085eddc5f8e3ccacbb7de0601518bcc35bae99b48523cc0f261f96b not found: ID does not exist" containerID="c3e7cd13d085eddc5f8e3ccacbb7de0601518bcc35bae99b48523cc0f261f96b"
Sep 18 20:22:30.626247 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:30.626215    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3e7cd13d085eddc5f8e3ccacbb7de0601518bcc35bae99b48523cc0f261f96b"} err="failed to get container status \"c3e7cd13d085eddc5f8e3ccacbb7de0601518bcc35bae99b48523cc0f261f96b\": rpc error: code = NotFound desc = could not find container \"c3e7cd13d085eddc5f8e3ccacbb7de0601518bcc35bae99b48523cc0f261f96b\": container with ID starting with c3e7cd13d085eddc5f8e3ccacbb7de0601518bcc35bae99b48523cc0f261f96b not found: ID does not exist"
Sep 18 20:22:30.631427 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:30.631366    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4"]
Sep 18 20:22:30.635678 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:30.635648    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-7d49a30d-openshift-default-644f67b664-7f6c4"]
Sep 18 20:22:31.522531 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:31.522494    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0658bf28-e97b-4d0d-945e-2f43253220b6" path="/var/lib/kubelet/pods/0658bf28-e97b-4d0d-945e-2f43253220b6/volumes"
Sep 18 20:22:52.175160 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:52.175114    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:22:55.834671 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:55.834635    2549 scope.go:117] "RemoveContainer" containerID="480fff18c6b4d1ed2e127d546c0f286a113ca41b678bb4fd5117ba181fbcae78"
Sep 18 20:22:55.843935 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:22:55.843913    2549 scope.go:117] "RemoveContainer" containerID="92a6cfb5fdec34475d9f59d4357957edbc092711a9b5e350d2e82e3a3b91c183"
Sep 18 20:23:07.107370 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:07.107319    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:23:21.426310 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:21.426270    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-69f1ba-openshift-default-569c8d9dd6-wn99g"]
Sep 18 20:23:21.426972 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:21.426948    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0658bf28-e97b-4d0d-945e-2f43253220b6" containerName="istio-proxy"
Sep 18 20:23:21.427079 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:21.426974    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="0658bf28-e97b-4d0d-945e-2f43253220b6" containerName="istio-proxy"
Sep 18 20:23:21.427079 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:21.426986    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9ea7fe1-104f-40e1-b269-889ce292658a" containerName="revoke-keys"
Sep 18 20:23:21.427079 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:21.426994    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9ea7fe1-104f-40e1-b269-889ce292658a" containerName="revoke-keys"
Sep 18 20:23:21.427079 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:21.427007    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="374c8e70-b164-4b98-8c5d-f7994db126f2" containerName="istio-proxy"
Sep 18 20:23:21.427079 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:21.427015    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="374c8e70-b164-4b98-8c5d-f7994db126f2" containerName="istio-proxy"
Sep 18 20:23:21.427079 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:21.427026    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0658bf28-e97b-4d0d-945e-2f43253220b6" containerName="istio-proxy"
Sep 18 20:23:21.427079 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:21.427034    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="0658bf28-e97b-4d0d-945e-2f43253220b6" containerName="istio-proxy"
Sep 18 20:23:21.427079 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:21.427052    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="daf3d095-d0b4-462f-b05d-0a699a8eb26c" containerName="main"
Sep 18 20:23:21.427079 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:21.427060    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="daf3d095-d0b4-462f-b05d-0a699a8eb26c" containerName="main"
Sep 18 20:23:21.427544 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:21.427180    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="374c8e70-b164-4b98-8c5d-f7994db126f2" containerName="istio-proxy"
Sep 18 20:23:21.427544 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:21.427191    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="0658bf28-e97b-4d0d-945e-2f43253220b6" containerName="istio-proxy"
Sep 18 20:23:21.427544 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:21.427201    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="374c8e70-b164-4b98-8c5d-f7994db126f2" containerName="istio-proxy"
Sep 18 20:23:21.427544 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:21.427209    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9ea7fe1-104f-40e1-b269-889ce292658a" containerName="revoke-keys"
Sep 18 20:23:21.427544 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:21.427219    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="daf3d095-d0b4-462f-b05d-0a699a8eb26c" containerName="main"
Sep 18 20:23:21.427544 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:21.427228    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="0658bf28-e97b-4d0d-945e-2f43253220b6" containerName="istio-proxy"
Sep 18 20:23:21.430828 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:21.430803    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-69f1ba-openshift-default-569c8d9dd6-wn99g"
Sep 18 20:23:21.434210 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:21.434182    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-69f1ba-openshift-default-dockercfg-25q4n\""
Sep 18 20:23:21.443261 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:21.443231    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-69f1ba-openshift-default-569c8d9dd6-wn99g"]
Sep 18 20:23:21.482814 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:21.482777    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4652b23-aa53-474e-afdf-6984f3fb1b63-workload-socket\") pod \"e2e-api-a-69f1ba-openshift-default-569c8d9dd6-wn99g\" (UID: \"e4652b23-aa53-474e-afdf-6984f3fb1b63\") " pod="openshift-ingress/e2e-api-a-69f1ba-openshift-default-569c8d9dd6-wn99g"
Sep 18 20:23:21.483037 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:21.482823    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4652b23-aa53-474e-afdf-6984f3fb1b63-istio-podinfo\") pod \"e2e-api-a-69f1ba-openshift-default-569c8d9dd6-wn99g\" (UID: \"e4652b23-aa53-474e-afdf-6984f3fb1b63\") " pod="openshift-ingress/e2e-api-a-69f1ba-openshift-default-569c8d9dd6-wn99g"
Sep 18 20:23:21.483037 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:21.482842    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4652b23-aa53-474e-afdf-6984f3fb1b63-istio-envoy\") pod \"e2e-api-a-69f1ba-openshift-default-569c8d9dd6-wn99g\" (UID: \"e4652b23-aa53-474e-afdf-6984f3fb1b63\") " pod="openshift-ingress/e2e-api-a-69f1ba-openshift-default-569c8d9dd6-wn99g"
Sep 18 20:23:21.483037 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:21.482861    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4652b23-aa53-474e-afdf-6984f3fb1b63-workload-certs\") pod \"e2e-api-a-69f1ba-openshift-default-569c8d9dd6-wn99g\" (UID: \"e4652b23-aa53-474e-afdf-6984f3fb1b63\") " pod="openshift-ingress/e2e-api-a-69f1ba-openshift-default-569c8d9dd6-wn99g"
Sep 18 20:23:21.483037 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:21.482945    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqr9f\" (UniqueName: \"kubernetes.io/projected/e4652b23-aa53-474e-afdf-6984f3fb1b63-kube-api-access-bqr9f\") pod \"e2e-api-a-69f1ba-openshift-default-569c8d9dd6-wn99g\" (UID: \"e4652b23-aa53-474e-afdf-6984f3fb1b63\") " pod="openshift-ingress/e2e-api-a-69f1ba-openshift-default-569c8d9dd6-wn99g"
Sep 18 20:23:21.483037 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:21.482999    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4652b23-aa53-474e-afdf-6984f3fb1b63-istio-data\") pod \"e2e-api-a-69f1ba-openshift-default-569c8d9dd6-wn99g\" (UID: \"e4652b23-aa53-474e-afdf-6984f3fb1b63\") " pod="openshift-ingress/e2e-api-a-69f1ba-openshift-default-569c8d9dd6-wn99g"
Sep 18 20:23:21.483272 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:21.483069    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4652b23-aa53-474e-afdf-6984f3fb1b63-istiod-ca-cert\") pod \"e2e-api-a-69f1ba-openshift-default-569c8d9dd6-wn99g\" (UID: \"e4652b23-aa53-474e-afdf-6984f3fb1b63\") " pod="openshift-ingress/e2e-api-a-69f1ba-openshift-default-569c8d9dd6-wn99g"
Sep 18 20:23:21.483272 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:21.483104    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4652b23-aa53-474e-afdf-6984f3fb1b63-credential-socket\") pod \"e2e-api-a-69f1ba-openshift-default-569c8d9dd6-wn99g\" (UID: \"e4652b23-aa53-474e-afdf-6984f3fb1b63\") " pod="openshift-ingress/e2e-api-a-69f1ba-openshift-default-569c8d9dd6-wn99g"
Sep 18 20:23:21.483272 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:21.483169    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4652b23-aa53-474e-afdf-6984f3fb1b63-istio-token\") pod \"e2e-api-a-69f1ba-openshift-default-569c8d9dd6-wn99g\" (UID: \"e4652b23-aa53-474e-afdf-6984f3fb1b63\") " pod="openshift-ingress/e2e-api-a-69f1ba-openshift-default-569c8d9dd6-wn99g"
Sep 18 20:23:21.584698 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:21.584637    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4652b23-aa53-474e-afdf-6984f3fb1b63-workload-socket\") pod \"e2e-api-a-69f1ba-openshift-default-569c8d9dd6-wn99g\" (UID: \"e4652b23-aa53-474e-afdf-6984f3fb1b63\") " pod="openshift-ingress/e2e-api-a-69f1ba-openshift-default-569c8d9dd6-wn99g"
Sep 18 20:23:21.584698 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:21.584706    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4652b23-aa53-474e-afdf-6984f3fb1b63-istio-podinfo\") pod \"e2e-api-a-69f1ba-openshift-default-569c8d9dd6-wn99g\" (UID: \"e4652b23-aa53-474e-afdf-6984f3fb1b63\") " pod="openshift-ingress/e2e-api-a-69f1ba-openshift-default-569c8d9dd6-wn99g"
Sep 18 20:23:21.584992 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:21.584734    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4652b23-aa53-474e-afdf-6984f3fb1b63-istio-envoy\") pod \"e2e-api-a-69f1ba-openshift-default-569c8d9dd6-wn99g\" (UID: \"e4652b23-aa53-474e-afdf-6984f3fb1b63\") " pod="openshift-ingress/e2e-api-a-69f1ba-openshift-default-569c8d9dd6-wn99g"
Sep 18 20:23:21.584992 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:21.584752    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4652b23-aa53-474e-afdf-6984f3fb1b63-workload-certs\") pod \"e2e-api-a-69f1ba-openshift-default-569c8d9dd6-wn99g\" (UID: \"e4652b23-aa53-474e-afdf-6984f3fb1b63\") " pod="openshift-ingress/e2e-api-a-69f1ba-openshift-default-569c8d9dd6-wn99g"
Sep 18 20:23:21.584992 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:21.584780    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqr9f\" (UniqueName: \"kubernetes.io/projected/e4652b23-aa53-474e-afdf-6984f3fb1b63-kube-api-access-bqr9f\") pod \"e2e-api-a-69f1ba-openshift-default-569c8d9dd6-wn99g\" (UID: \"e4652b23-aa53-474e-afdf-6984f3fb1b63\") " pod="openshift-ingress/e2e-api-a-69f1ba-openshift-default-569c8d9dd6-wn99g"
Sep 18 20:23:21.585160 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:21.585015    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4652b23-aa53-474e-afdf-6984f3fb1b63-istio-data\") pod \"e2e-api-a-69f1ba-openshift-default-569c8d9dd6-wn99g\" (UID: \"e4652b23-aa53-474e-afdf-6984f3fb1b63\") " pod="openshift-ingress/e2e-api-a-69f1ba-openshift-default-569c8d9dd6-wn99g"
Sep 18 20:23:21.585160 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:21.585069    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4652b23-aa53-474e-afdf-6984f3fb1b63-workload-socket\") pod \"e2e-api-a-69f1ba-openshift-default-569c8d9dd6-wn99g\" (UID: \"e4652b23-aa53-474e-afdf-6984f3fb1b63\") " pod="openshift-ingress/e2e-api-a-69f1ba-openshift-default-569c8d9dd6-wn99g"
Sep 18 20:23:21.585278 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:21.585165    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4652b23-aa53-474e-afdf-6984f3fb1b63-istiod-ca-cert\") pod \"e2e-api-a-69f1ba-openshift-default-569c8d9dd6-wn99g\" (UID: \"e4652b23-aa53-474e-afdf-6984f3fb1b63\") " pod="openshift-ingress/e2e-api-a-69f1ba-openshift-default-569c8d9dd6-wn99g"
Sep 18 20:23:21.585278 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:21.585220    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4652b23-aa53-474e-afdf-6984f3fb1b63-credential-socket\") pod \"e2e-api-a-69f1ba-openshift-default-569c8d9dd6-wn99g\" (UID: \"e4652b23-aa53-474e-afdf-6984f3fb1b63\") " pod="openshift-ingress/e2e-api-a-69f1ba-openshift-default-569c8d9dd6-wn99g"
Sep 18 20:23:21.585377 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:21.585281    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4652b23-aa53-474e-afdf-6984f3fb1b63-istio-token\") pod \"e2e-api-a-69f1ba-openshift-default-569c8d9dd6-wn99g\" (UID: \"e4652b23-aa53-474e-afdf-6984f3fb1b63\") " pod="openshift-ingress/e2e-api-a-69f1ba-openshift-default-569c8d9dd6-wn99g"
Sep 18 20:23:21.585377 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:21.585315    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4652b23-aa53-474e-afdf-6984f3fb1b63-istio-data\") pod \"e2e-api-a-69f1ba-openshift-default-569c8d9dd6-wn99g\" (UID: \"e4652b23-aa53-474e-afdf-6984f3fb1b63\") " pod="openshift-ingress/e2e-api-a-69f1ba-openshift-default-569c8d9dd6-wn99g"
Sep 18 20:23:21.585377 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:21.585371    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4652b23-aa53-474e-afdf-6984f3fb1b63-workload-certs\") pod \"e2e-api-a-69f1ba-openshift-default-569c8d9dd6-wn99g\" (UID: \"e4652b23-aa53-474e-afdf-6984f3fb1b63\") " pod="openshift-ingress/e2e-api-a-69f1ba-openshift-default-569c8d9dd6-wn99g"
Sep 18 20:23:21.585646 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:21.585621    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4652b23-aa53-474e-afdf-6984f3fb1b63-credential-socket\") pod \"e2e-api-a-69f1ba-openshift-default-569c8d9dd6-wn99g\" (UID: \"e4652b23-aa53-474e-afdf-6984f3fb1b63\") " pod="openshift-ingress/e2e-api-a-69f1ba-openshift-default-569c8d9dd6-wn99g"
Sep 18 20:23:21.586117 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:21.586095    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4652b23-aa53-474e-afdf-6984f3fb1b63-istiod-ca-cert\") pod \"e2e-api-a-69f1ba-openshift-default-569c8d9dd6-wn99g\" (UID: \"e4652b23-aa53-474e-afdf-6984f3fb1b63\") " pod="openshift-ingress/e2e-api-a-69f1ba-openshift-default-569c8d9dd6-wn99g"
Sep 18 20:23:21.587297 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:21.587275    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4652b23-aa53-474e-afdf-6984f3fb1b63-istio-envoy\") pod \"e2e-api-a-69f1ba-openshift-default-569c8d9dd6-wn99g\" (UID: \"e4652b23-aa53-474e-afdf-6984f3fb1b63\") " pod="openshift-ingress/e2e-api-a-69f1ba-openshift-default-569c8d9dd6-wn99g"
Sep 18 20:23:21.587527 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:21.587508    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4652b23-aa53-474e-afdf-6984f3fb1b63-istio-podinfo\") pod \"e2e-api-a-69f1ba-openshift-default-569c8d9dd6-wn99g\" (UID: \"e4652b23-aa53-474e-afdf-6984f3fb1b63\") " pod="openshift-ingress/e2e-api-a-69f1ba-openshift-default-569c8d9dd6-wn99g"
Sep 18 20:23:21.595969 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:21.595940    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4652b23-aa53-474e-afdf-6984f3fb1b63-istio-token\") pod \"e2e-api-a-69f1ba-openshift-default-569c8d9dd6-wn99g\" (UID: \"e4652b23-aa53-474e-afdf-6984f3fb1b63\") " pod="openshift-ingress/e2e-api-a-69f1ba-openshift-default-569c8d9dd6-wn99g"
Sep 18 20:23:21.596208 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:21.596181    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqr9f\" (UniqueName: \"kubernetes.io/projected/e4652b23-aa53-474e-afdf-6984f3fb1b63-kube-api-access-bqr9f\") pod \"e2e-api-a-69f1ba-openshift-default-569c8d9dd6-wn99g\" (UID: \"e4652b23-aa53-474e-afdf-6984f3fb1b63\") " pod="openshift-ingress/e2e-api-a-69f1ba-openshift-default-569c8d9dd6-wn99g"
Sep 18 20:23:21.742685 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:21.742582    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-69f1ba-openshift-default-569c8d9dd6-wn99g"
Sep 18 20:23:22.041949 ip-10-0-140-178 kubenswrapper[2549]: W0918 20:23:22.041905    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4652b23_aa53_474e_afdf_6984f3fb1b63.slice/crio-852438eacc0bc3de1dd86acb7a15489e946e69e638dc7aa11241eb6ecb867719 WatchSource:0}: Error finding container 852438eacc0bc3de1dd86acb7a15489e946e69e638dc7aa11241eb6ecb867719: Status 404 returned error can't find the container with id 852438eacc0bc3de1dd86acb7a15489e946e69e638dc7aa11241eb6ecb867719
Sep 18 20:23:22.043734 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:22.043705    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-69f1ba-openshift-default-569c8d9dd6-wn99g"]
Sep 18 20:23:22.044766 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:22.044729    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:23:22.044847 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:22.044799    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:23:22.044847 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:22.044827    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:23:22.944239 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:22.944194    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-69f1ba-openshift-default-569c8d9dd6-wn99g" event={"ID":"e4652b23-aa53-474e-afdf-6984f3fb1b63","Type":"ContainerStarted","Data":"0bbb01dc316b53c46e7843ad2513a970f76c6f3c113a2b81f2457802b6699fcc"}
Sep 18 20:23:22.944689 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:22.944258    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-69f1ba-openshift-default-569c8d9dd6-wn99g" event={"ID":"e4652b23-aa53-474e-afdf-6984f3fb1b63","Type":"ContainerStarted","Data":"852438eacc0bc3de1dd86acb7a15489e946e69e638dc7aa11241eb6ecb867719"}
Sep 18 20:23:22.971157 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:22.971096    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-69f1ba-openshift-default-569c8d9dd6-wn99g" podStartSLOduration=1.971078155 podStartE2EDuration="1.971078155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:23:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:23:22.969323931 +0000 UTC m=+1587.923065109" watchObservedRunningTime="2026-09-18 20:23:22.971078155 +0000 UTC m=+1587.924819320"
Sep 18 20:23:23.743508 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:23.743461    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-69f1ba-openshift-default-569c8d9dd6-wn99g"
Sep 18 20:23:23.749325 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:23.749293    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-69f1ba-openshift-default-569c8d9dd6-wn99g"
Sep 18 20:23:23.952730 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:23.952687    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-69f1ba-openshift-default-569c8d9dd6-wn99g"
Sep 18 20:23:23.954004 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:23.953966    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-69f1ba-openshift-default-569c8d9dd6-wn99g"
Sep 18 20:23:25.353448 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:25.353402    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:23:47.116333 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:47.116293    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:23:47.120549 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:47.120518    2549 generic.go:356] "Generic (PLEG): container finished" podID="e4652b23-aa53-474e-afdf-6984f3fb1b63" containerID="0bbb01dc316b53c46e7843ad2513a970f76c6f3c113a2b81f2457802b6699fcc" exitCode=0
Sep 18 20:23:47.120719 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:47.120592    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-69f1ba-openshift-default-569c8d9dd6-wn99g" event={"ID":"e4652b23-aa53-474e-afdf-6984f3fb1b63","Type":"ContainerDied","Data":"0bbb01dc316b53c46e7843ad2513a970f76c6f3c113a2b81f2457802b6699fcc"}
Sep 18 20:23:47.121028 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:47.121012    2549 scope.go:117] "RemoveContainer" containerID="0bbb01dc316b53c46e7843ad2513a970f76c6f3c113a2b81f2457802b6699fcc"
Sep 18 20:23:47.122248 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:47.122217    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:23:47.122315 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:47.122292    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:23:47.122351 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:47.122323    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:23:48.130592 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:48.130491    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-69f1ba-openshift-default-569c8d9dd6-wn99g" event={"ID":"e4652b23-aa53-474e-afdf-6984f3fb1b63","Type":"ContainerStarted","Data":"bd5d0c51eefc0d6fe6119d633bc0bff9c9e2a5818dd6e34ec4c0a19d3df8624b"}
Sep 18 20:23:48.743367 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:48.743321    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-69f1ba-openshift-default-569c8d9dd6-wn99g"
Sep 18 20:23:48.743649 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:48.743608    2549 patch_prober.go:28] interesting pod/e2e-api-a-69f1ba-openshift-default-569c8d9dd6-wn99g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Sep 18 20:23:48.743774 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:48.743676    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-69f1ba-openshift-default-569c8d9dd6-wn99g" podUID="e4652b23-aa53-474e-afdf-6984f3fb1b63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Sep 18 20:23:49.743669 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:49.743629    2549 patch_prober.go:28] interesting pod/e2e-api-a-69f1ba-openshift-default-569c8d9dd6-wn99g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Sep 18 20:23:49.744077 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:49.743708    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-69f1ba-openshift-default-569c8d9dd6-wn99g" podUID="e4652b23-aa53-474e-afdf-6984f3fb1b63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Sep 18 20:23:50.743841 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:50.743796    2549 patch_prober.go:28] interesting pod/e2e-api-a-69f1ba-openshift-default-569c8d9dd6-wn99g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Sep 18 20:23:50.744262 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:50.743858    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-69f1ba-openshift-default-569c8d9dd6-wn99g" podUID="e4652b23-aa53-474e-afdf-6984f3fb1b63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Sep 18 20:23:51.743651 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:51.743613    2549 patch_prober.go:28] interesting pod/e2e-api-a-69f1ba-openshift-default-569c8d9dd6-wn99g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Sep 18 20:23:51.743843 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:51.743676    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-69f1ba-openshift-default-569c8d9dd6-wn99g" podUID="e4652b23-aa53-474e-afdf-6984f3fb1b63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Sep 18 20:23:52.743352 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:52.743312    2549 patch_prober.go:28] interesting pod/e2e-api-a-69f1ba-openshift-default-569c8d9dd6-wn99g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Sep 18 20:23:52.743811 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:52.743414    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-69f1ba-openshift-default-569c8d9dd6-wn99g" podUID="e4652b23-aa53-474e-afdf-6984f3fb1b63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Sep 18 20:23:53.748014 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:53.747972    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-69f1ba-openshift-default-569c8d9dd6-wn99g"
Sep 18 20:23:53.748518 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:53.748246    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-69f1ba-openshift-default-569c8d9dd6-wn99g"
Sep 18 20:23:53.749081 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:53.749062    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-69f1ba-openshift-default-569c8d9dd6-wn99g"
Sep 18 20:23:55.811894 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:55.811863    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526"]
Sep 18 20:23:55.812363 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:55.812348    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="374c8e70-b164-4b98-8c5d-f7994db126f2" containerName="istio-proxy"
Sep 18 20:23:55.812456 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:55.812365    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="374c8e70-b164-4b98-8c5d-f7994db126f2" containerName="istio-proxy"
Sep 18 20:23:55.815838 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:55.815817    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526"
Sep 18 20:23:55.819001 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:55.818971    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-4cd712-openshift-default-dockercfg-55st7\""
Sep 18 20:23:55.831542 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:55.831510    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526"]
Sep 18 20:23:55.857305 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:55.857260    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1994521d-e96d-4507-b0a4-d841c5545735-istio-podinfo\") pod \"e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526\" (UID: \"1994521d-e96d-4507-b0a4-d841c5545735\") " pod="openshift-ingress/e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526"
Sep 18 20:23:55.857305 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:55.857306    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1994521d-e96d-4507-b0a4-d841c5545735-istio-token\") pod \"e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526\" (UID: \"1994521d-e96d-4507-b0a4-d841c5545735\") " pod="openshift-ingress/e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526"
Sep 18 20:23:55.857589 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:55.857342    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1994521d-e96d-4507-b0a4-d841c5545735-istio-envoy\") pod \"e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526\" (UID: \"1994521d-e96d-4507-b0a4-d841c5545735\") " pod="openshift-ingress/e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526"
Sep 18 20:23:55.857589 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:55.857369    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1994521d-e96d-4507-b0a4-d841c5545735-istio-data\") pod \"e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526\" (UID: \"1994521d-e96d-4507-b0a4-d841c5545735\") " pod="openshift-ingress/e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526"
Sep 18 20:23:55.857589 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:55.857420    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1994521d-e96d-4507-b0a4-d841c5545735-istiod-ca-cert\") pod \"e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526\" (UID: \"1994521d-e96d-4507-b0a4-d841c5545735\") " pod="openshift-ingress/e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526"
Sep 18 20:23:55.857589 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:55.857514    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1994521d-e96d-4507-b0a4-d841c5545735-workload-socket\") pod \"e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526\" (UID: \"1994521d-e96d-4507-b0a4-d841c5545735\") " pod="openshift-ingress/e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526"
Sep 18 20:23:55.857589 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:55.857550    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1994521d-e96d-4507-b0a4-d841c5545735-credential-socket\") pod \"e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526\" (UID: \"1994521d-e96d-4507-b0a4-d841c5545735\") " pod="openshift-ingress/e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526"
Sep 18 20:23:55.857791 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:55.857652    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrdnl\" (UniqueName: \"kubernetes.io/projected/1994521d-e96d-4507-b0a4-d841c5545735-kube-api-access-hrdnl\") pod \"e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526\" (UID: \"1994521d-e96d-4507-b0a4-d841c5545735\") " pod="openshift-ingress/e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526"
Sep 18 20:23:55.857791 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:55.857696    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1994521d-e96d-4507-b0a4-d841c5545735-workload-certs\") pod \"e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526\" (UID: \"1994521d-e96d-4507-b0a4-d841c5545735\") " pod="openshift-ingress/e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526"
Sep 18 20:23:55.881871 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:55.881836    2549 scope.go:117] "RemoveContainer" containerID="76c62a9017d4252d8e9e763c23ccf8fedab9c5756ee6140731ca025743af5b11"
Sep 18 20:23:55.891327 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:55.891303    2549 scope.go:117] "RemoveContainer" containerID="5332398872c709c4074fcca2c90250ae4140755588b12553b706a5659480d0f7"
Sep 18 20:23:55.958518 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:55.958466    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1994521d-e96d-4507-b0a4-d841c5545735-workload-socket\") pod \"e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526\" (UID: \"1994521d-e96d-4507-b0a4-d841c5545735\") " pod="openshift-ingress/e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526"
Sep 18 20:23:55.958743 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:55.958543    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1994521d-e96d-4507-b0a4-d841c5545735-credential-socket\") pod \"e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526\" (UID: \"1994521d-e96d-4507-b0a4-d841c5545735\") " pod="openshift-ingress/e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526"
Sep 18 20:23:55.958743 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:55.958632    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrdnl\" (UniqueName: \"kubernetes.io/projected/1994521d-e96d-4507-b0a4-d841c5545735-kube-api-access-hrdnl\") pod \"e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526\" (UID: \"1994521d-e96d-4507-b0a4-d841c5545735\") " pod="openshift-ingress/e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526"
Sep 18 20:23:55.958743 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:55.958679    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1994521d-e96d-4507-b0a4-d841c5545735-workload-certs\") pod \"e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526\" (UID: \"1994521d-e96d-4507-b0a4-d841c5545735\") " pod="openshift-ingress/e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526"
Sep 18 20:23:55.958912 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:55.958752    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1994521d-e96d-4507-b0a4-d841c5545735-istio-podinfo\") pod \"e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526\" (UID: \"1994521d-e96d-4507-b0a4-d841c5545735\") " pod="openshift-ingress/e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526"
Sep 18 20:23:55.958912 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:55.958777    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1994521d-e96d-4507-b0a4-d841c5545735-istio-token\") pod \"e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526\" (UID: \"1994521d-e96d-4507-b0a4-d841c5545735\") " pod="openshift-ingress/e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526"
Sep 18 20:23:55.958912 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:55.958803    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1994521d-e96d-4507-b0a4-d841c5545735-istio-envoy\") pod \"e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526\" (UID: \"1994521d-e96d-4507-b0a4-d841c5545735\") " pod="openshift-ingress/e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526"
Sep 18 20:23:55.958912 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:55.958827    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1994521d-e96d-4507-b0a4-d841c5545735-istio-data\") pod \"e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526\" (UID: \"1994521d-e96d-4507-b0a4-d841c5545735\") " pod="openshift-ingress/e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526"
Sep 18 20:23:55.958912 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:55.958861    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1994521d-e96d-4507-b0a4-d841c5545735-istiod-ca-cert\") pod \"e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526\" (UID: \"1994521d-e96d-4507-b0a4-d841c5545735\") " pod="openshift-ingress/e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526"
Sep 18 20:23:55.959178 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:55.958961    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1994521d-e96d-4507-b0a4-d841c5545735-workload-socket\") pod \"e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526\" (UID: \"1994521d-e96d-4507-b0a4-d841c5545735\") " pod="openshift-ingress/e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526"
Sep 18 20:23:55.959707 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:55.959676    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1994521d-e96d-4507-b0a4-d841c5545735-istio-data\") pod \"e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526\" (UID: \"1994521d-e96d-4507-b0a4-d841c5545735\") " pod="openshift-ingress/e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526"
Sep 18 20:23:55.959905 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:55.959692    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1994521d-e96d-4507-b0a4-d841c5545735-istiod-ca-cert\") pod \"e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526\" (UID: \"1994521d-e96d-4507-b0a4-d841c5545735\") " pod="openshift-ingress/e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526"
Sep 18 20:23:55.960027 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:55.959964    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1994521d-e96d-4507-b0a4-d841c5545735-workload-certs\") pod \"e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526\" (UID: \"1994521d-e96d-4507-b0a4-d841c5545735\") " pod="openshift-ingress/e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526"
Sep 18 20:23:55.963405 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:55.960188    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1994521d-e96d-4507-b0a4-d841c5545735-credential-socket\") pod \"e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526\" (UID: \"1994521d-e96d-4507-b0a4-d841c5545735\") " pod="openshift-ingress/e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526"
Sep 18 20:23:55.963405 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:55.962718    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1994521d-e96d-4507-b0a4-d841c5545735-istio-envoy\") pod \"e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526\" (UID: \"1994521d-e96d-4507-b0a4-d841c5545735\") " pod="openshift-ingress/e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526"
Sep 18 20:23:55.963405 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:55.962912    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1994521d-e96d-4507-b0a4-d841c5545735-istio-podinfo\") pod \"e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526\" (UID: \"1994521d-e96d-4507-b0a4-d841c5545735\") " pod="openshift-ingress/e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526"
Sep 18 20:23:55.970266 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:55.970234    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrdnl\" (UniqueName: \"kubernetes.io/projected/1994521d-e96d-4507-b0a4-d841c5545735-kube-api-access-hrdnl\") pod \"e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526\" (UID: \"1994521d-e96d-4507-b0a4-d841c5545735\") " pod="openshift-ingress/e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526"
Sep 18 20:23:55.978241 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:55.978173    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1994521d-e96d-4507-b0a4-d841c5545735-istio-token\") pod \"e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526\" (UID: \"1994521d-e96d-4507-b0a4-d841c5545735\") " pod="openshift-ingress/e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526"
Sep 18 20:23:56.127093 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:56.127039    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526"
Sep 18 20:23:56.414933 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:56.414902    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526"]
Sep 18 20:23:56.416617 ip-10-0-140-178 kubenswrapper[2549]: W0918 20:23:56.416565    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1994521d_e96d_4507_b0a4_d841c5545735.slice/crio-b42ab4a4ee7559061ab4e44e930bf606a30fac08fb3a4295a610c1f584fe6ca8 WatchSource:0}: Error finding container b42ab4a4ee7559061ab4e44e930bf606a30fac08fb3a4295a610c1f584fe6ca8: Status 404 returned error can't find the container with id b42ab4a4ee7559061ab4e44e930bf606a30fac08fb3a4295a610c1f584fe6ca8
Sep 18 20:23:56.419096 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:56.419060    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:23:56.419192 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:56.419131    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:23:56.419192 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:56.419160    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:23:57.205827 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:57.205780    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526" event={"ID":"1994521d-e96d-4507-b0a4-d841c5545735","Type":"ContainerStarted","Data":"723501b3501ceb59e617c1423c1d2ba60699b340de98a90ab474413ac4af2227"}
Sep 18 20:23:57.206270 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:57.205852    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526" event={"ID":"1994521d-e96d-4507-b0a4-d841c5545735","Type":"ContainerStarted","Data":"b42ab4a4ee7559061ab4e44e930bf606a30fac08fb3a4295a610c1f584fe6ca8"}
Sep 18 20:23:57.236248 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:57.236186    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526" podStartSLOduration=2.236167208 podStartE2EDuration="2.236167208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:23:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:23:57.234643979 +0000 UTC m=+1622.188385149" watchObservedRunningTime="2026-09-18 20:23:57.236167208 +0000 UTC m=+1622.189908376"
Sep 18 20:23:58.127313 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:58.127271    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526"
Sep 18 20:23:58.132447 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:58.132416    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526"
Sep 18 20:23:58.212605 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:58.212562    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526"
Sep 18 20:23:58.213661 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:23:58.213635    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526"
Sep 18 20:24:03.619073 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:03.619028    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:24:23.376056 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:23.376006    2549 generic.go:356] "Generic (PLEG): container finished" podID="1994521d-e96d-4507-b0a4-d841c5545735" containerID="723501b3501ceb59e617c1423c1d2ba60699b340de98a90ab474413ac4af2227" exitCode=0
Sep 18 20:24:23.376619 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:23.376068    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526" event={"ID":"1994521d-e96d-4507-b0a4-d841c5545735","Type":"ContainerDied","Data":"723501b3501ceb59e617c1423c1d2ba60699b340de98a90ab474413ac4af2227"}
Sep 18 20:24:23.376619 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:23.376586    2549 scope.go:117] "RemoveContainer" containerID="723501b3501ceb59e617c1423c1d2ba60699b340de98a90ab474413ac4af2227"
Sep 18 20:24:23.377954 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:23.377919    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:24:23.378084 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:23.378006    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:24:23.378084 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:23.378039    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:24:24.388306 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:24.388270    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526" event={"ID":"1994521d-e96d-4507-b0a4-d841c5545735","Type":"ContainerStarted","Data":"b96a04fd7a22e01135a8d75ec655ec9d7270cad2d7925a7da66e01b886d0b47d"}
Sep 18 20:24:25.127776 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:25.127733    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526"
Sep 18 20:24:25.128876 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:25.128846    2549 patch_prober.go:28] interesting pod/e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Sep 18 20:24:25.129040 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:25.128909    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526" podUID="1994521d-e96d-4507-b0a4-d841c5545735" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Sep 18 20:24:26.108682 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:26.108644    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:24:26.128539 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:26.128502    2549 patch_prober.go:28] interesting pod/e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Sep 18 20:24:26.128706 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:26.128559    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526" podUID="1994521d-e96d-4507-b0a4-d841c5545735" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Sep 18 20:24:27.128591 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:27.128547    2549 patch_prober.go:28] interesting pod/e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Sep 18 20:24:27.129131 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:27.128620    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526" podUID="1994521d-e96d-4507-b0a4-d841c5545735" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Sep 18 20:24:28.127709 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:28.127663    2549 patch_prober.go:28] interesting pod/e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Sep 18 20:24:28.127912 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:28.127730    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526" podUID="1994521d-e96d-4507-b0a4-d841c5545735" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Sep 18 20:24:28.212834 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:28.212787    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526"
Sep 18 20:24:29.127825 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:29.127774    2549 patch_prober.go:28] interesting pod/e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Sep 18 20:24:29.128021 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:29.127839    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526" podUID="1994521d-e96d-4507-b0a4-d841c5545735" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Sep 18 20:24:30.128537 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:30.128495    2549 patch_prober.go:28] interesting pod/e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Sep 18 20:24:30.128955 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:30.128575    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526" podUID="1994521d-e96d-4507-b0a4-d841c5545735" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Sep 18 20:24:30.182847 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:30.182808    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7741-ab5743"]
Sep 18 20:24:30.185801 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:30.185777    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7741-ab5743"
Sep 18 20:24:30.189217 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:30.189183    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-jcdsw\""
Sep 18 20:24:30.194906 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:30.194878    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7741-ab5743"]
Sep 18 20:24:30.237768 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:30.237723    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97v7x\" (UniqueName: \"kubernetes.io/projected/88fd4c9d-852a-4b49-98b6-62b7a424ac02-kube-api-access-97v7x\") pod \"test-curl-7741-ab5743\" (UID: \"88fd4c9d-852a-4b49-98b6-62b7a424ac02\") " pod="opendatahub/test-curl-7741-ab5743"
Sep 18 20:24:30.339400 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:30.339346    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97v7x\" (UniqueName: \"kubernetes.io/projected/88fd4c9d-852a-4b49-98b6-62b7a424ac02-kube-api-access-97v7x\") pod \"test-curl-7741-ab5743\" (UID: \"88fd4c9d-852a-4b49-98b6-62b7a424ac02\") " pod="opendatahub/test-curl-7741-ab5743"
Sep 18 20:24:30.359138 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:30.359093    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97v7x\" (UniqueName: \"kubernetes.io/projected/88fd4c9d-852a-4b49-98b6-62b7a424ac02-kube-api-access-97v7x\") pod \"test-curl-7741-ab5743\" (UID: \"88fd4c9d-852a-4b49-98b6-62b7a424ac02\") " pod="opendatahub/test-curl-7741-ab5743"
Sep 18 20:24:30.497487 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:30.497366    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7741-ab5743"
Sep 18 20:24:30.772341 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:30.772309    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7741-ab5743"]
Sep 18 20:24:30.774746 ip-10-0-140-178 kubenswrapper[2549]: W0918 20:24:30.774693    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod88fd4c9d_852a_4b49_98b6_62b7a424ac02.slice/crio-132800c635999777b1fbbb47bb8f0d63067db2f3e7bb807d7dd4d0886b60f3cf WatchSource:0}: Error finding container 132800c635999777b1fbbb47bb8f0d63067db2f3e7bb807d7dd4d0886b60f3cf: Status 404 returned error can't find the container with id 132800c635999777b1fbbb47bb8f0d63067db2f3e7bb807d7dd4d0886b60f3cf
Sep 18 20:24:31.132845 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:31.132811    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526"
Sep 18 20:24:31.134057 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:31.134034    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526"
Sep 18 20:24:31.433914 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:31.433807    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7741-ab5743" event={"ID":"88fd4c9d-852a-4b49-98b6-62b7a424ac02","Type":"ContainerStarted","Data":"c851a39b859abf3fcf30e96a318d7f3323104a4bcdb3ee76624b1ce78f4d4653"}
Sep 18 20:24:31.433914 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:31.433861    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7741-ab5743" event={"ID":"88fd4c9d-852a-4b49-98b6-62b7a424ac02","Type":"ContainerStarted","Data":"132800c635999777b1fbbb47bb8f0d63067db2f3e7bb807d7dd4d0886b60f3cf"}
Sep 18 20:24:31.454009 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:31.453928    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7741-ab5743" podStartSLOduration=1.453901988 podStartE2EDuration="1.453901988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:24:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:24:31.453102278 +0000 UTC m=+1656.406843446" watchObservedRunningTime="2026-09-18 20:24:31.453901988 +0000 UTC m=+1656.407643157"
Sep 18 20:24:32.064660 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:32.064605    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7741-ab5743"]
Sep 18 20:24:32.070183 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:32.070140    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7741-ab5743"]
Sep 18 20:24:32.234109 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:32.233994    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-69f1ba-vphnh"]
Sep 18 20:24:32.234632 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:32.234514    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88fd4c9d-852a-4b49-98b6-62b7a424ac02" containerName="test-curl-7741-ab5743"
Sep 18 20:24:32.234632 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:32.234528    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="88fd4c9d-852a-4b49-98b6-62b7a424ac02" containerName="test-curl-7741-ab5743"
Sep 18 20:24:32.234632 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:32.234633    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="88fd4c9d-852a-4b49-98b6-62b7a424ac02" containerName="test-curl-7741-ab5743"
Sep 18 20:24:32.236885 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:32.236865    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-69f1ba-vphnh"
Sep 18 20:24:32.243194 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:32.243168    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bhssz\""
Sep 18 20:24:32.244424 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:32.244397    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 20:24:32.255292 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:32.255258    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-69f1ba-vphnh"]
Sep 18 20:24:32.362130 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:32.362089    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/586e3f0c-8db8-4147-849c-0ce53ad249bb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-69f1ba-vphnh\" (UID: \"586e3f0c-8db8-4147-849c-0ce53ad249bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-69f1ba-vphnh"
Sep 18 20:24:32.440090 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:24:32.440056    2549 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7741-ab5743" container="test-curl-7741-ab5743"
Sep 18 20:24:32.463105 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:32.463058    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/586e3f0c-8db8-4147-849c-0ce53ad249bb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-69f1ba-vphnh\" (UID: \"586e3f0c-8db8-4147-849c-0ce53ad249bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-69f1ba-vphnh"
Sep 18 20:24:32.463743 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:32.463723    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/586e3f0c-8db8-4147-849c-0ce53ad249bb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-69f1ba-vphnh\" (UID: \"586e3f0c-8db8-4147-849c-0ce53ad249bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-69f1ba-vphnh"
Sep 18 20:24:32.548439 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:32.548335    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-69f1ba-vphnh"
Sep 18 20:24:32.830322 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:32.830288    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-69f1ba-vphnh"]
Sep 18 20:24:32.831249 ip-10-0-140-178 kubenswrapper[2549]: W0918 20:24:32.831218    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod586e3f0c_8db8_4147_849c_0ce53ad249bb.slice/crio-7c26f823c8554974ba63125ab625f1b8fe8150ecc94dc714b6c99c0285bcbe10 WatchSource:0}: Error finding container 7c26f823c8554974ba63125ab625f1b8fe8150ecc94dc714b6c99c0285bcbe10: Status 404 returned error can't find the container with id 7c26f823c8554974ba63125ab625f1b8fe8150ecc94dc714b6c99c0285bcbe10
Sep 18 20:24:33.458534 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:33.458424    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-69f1ba-vphnh" event={"ID":"586e3f0c-8db8-4147-849c-0ce53ad249bb","Type":"ContainerStarted","Data":"feedefd960317cfc09be5828022e273cf18cf249912685fddb7d5206d29ae5c3"}
Sep 18 20:24:33.458534 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:33.458514    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-69f1ba-vphnh" event={"ID":"586e3f0c-8db8-4147-849c-0ce53ad249bb","Type":"ContainerStarted","Data":"7c26f823c8554974ba63125ab625f1b8fe8150ecc94dc714b6c99c0285bcbe10"}
Sep 18 20:24:33.459051 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:33.458770    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7741-ab5743" podUID="88fd4c9d-852a-4b49-98b6-62b7a424ac02" containerName="test-curl-7741-ab5743" containerID="cri-o://c851a39b859abf3fcf30e96a318d7f3323104a4bcdb3ee76624b1ce78f4d4653" gracePeriod=2
Sep 18 20:24:33.478647 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:24:33.478594    2549 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7741-ab5743\" is forbidden: User \"system:node:ip-10-0-140-178.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-178.ec2.internal' and this object" podUID="88fd4c9d-852a-4b49-98b6-62b7a424ac02" pod="opendatahub/test-curl-7741-ab5743"
Sep 18 20:24:33.479000 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:33.478624    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-69f1ba-vphnh" podStartSLOduration=1.478607765 podStartE2EDuration="1.478607765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:24:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:24:33.475866154 +0000 UTC m=+1658.429607324" watchObservedRunningTime="2026-09-18 20:24:33.478607765 +0000 UTC m=+1658.432348932"
Sep 18 20:24:33.513532 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:33.513491    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-40a1a5/auth-test-model-40a1a5-kserve-78fdfcfdc5-6f279"]
Sep 18 20:24:33.537018 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:24:33.536977    2549 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7741-ab5743\" is forbidden: User \"system:node:ip-10-0-140-178.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-178.ec2.internal' and this object" podUID="88fd4c9d-852a-4b49-98b6-62b7a424ac02" pod="opendatahub/test-curl-7741-ab5743"
Sep 18 20:24:34.467227 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:34.467190    2549 generic.go:356] "Generic (PLEG): container finished" podID="586e3f0c-8db8-4147-849c-0ce53ad249bb" containerID="feedefd960317cfc09be5828022e273cf18cf249912685fddb7d5206d29ae5c3" exitCode=0
Sep 18 20:24:34.467791 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:34.467314    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-69f1ba-vphnh" event={"ID":"586e3f0c-8db8-4147-849c-0ce53ad249bb","Type":"ContainerDied","Data":"feedefd960317cfc09be5828022e273cf18cf249912685fddb7d5206d29ae5c3"}
Sep 18 20:24:35.634891 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:35.634862    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-69f1ba-vphnh"
Sep 18 20:24:35.678371 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:35.678341    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7741-ab5743"
Sep 18 20:24:35.681846 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:24:35.681808    2549 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7741-ab5743\" is forbidden: User \"system:node:ip-10-0-140-178.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-178.ec2.internal' and this object" podUID="88fd4c9d-852a-4b49-98b6-62b7a424ac02" pod="opendatahub/test-curl-7741-ab5743"
Sep 18 20:24:35.699238 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:35.699205    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/586e3f0c-8db8-4147-849c-0ce53ad249bb-maas-api-service-ca\") pod \"586e3f0c-8db8-4147-849c-0ce53ad249bb\" (UID: \"586e3f0c-8db8-4147-849c-0ce53ad249bb\") "
Sep 18 20:24:35.699617 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:35.699591    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/586e3f0c-8db8-4147-849c-0ce53ad249bb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "586e3f0c-8db8-4147-849c-0ce53ad249bb" (UID: "586e3f0c-8db8-4147-849c-0ce53ad249bb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:24:35.699689 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:35.699678    2549 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/586e3f0c-8db8-4147-849c-0ce53ad249bb-maas-api-service-ca\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:24:35.800612 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:35.800511    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-97v7x\" (UniqueName: \"kubernetes.io/projected/88fd4c9d-852a-4b49-98b6-62b7a424ac02-kube-api-access-97v7x\") pod \"88fd4c9d-852a-4b49-98b6-62b7a424ac02\" (UID: \"88fd4c9d-852a-4b49-98b6-62b7a424ac02\") "
Sep 18 20:24:35.802792 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:35.802766    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88fd4c9d-852a-4b49-98b6-62b7a424ac02-kube-api-access-97v7x" (OuterVolumeSpecName: "kube-api-access-97v7x") pod "88fd4c9d-852a-4b49-98b6-62b7a424ac02" (UID: "88fd4c9d-852a-4b49-98b6-62b7a424ac02"). InnerVolumeSpecName "kube-api-access-97v7x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:24:35.902456 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:35.902372    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97v7x\" (UniqueName: \"kubernetes.io/projected/88fd4c9d-852a-4b49-98b6-62b7a424ac02-kube-api-access-97v7x\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:24:36.482605 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:36.482572    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-69f1ba-vphnh"
Sep 18 20:24:36.482806 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:36.482601    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-69f1ba-vphnh" event={"ID":"586e3f0c-8db8-4147-849c-0ce53ad249bb","Type":"ContainerDied","Data":"7c26f823c8554974ba63125ab625f1b8fe8150ecc94dc714b6c99c0285bcbe10"}
Sep 18 20:24:36.482806 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:36.482661    2549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c26f823c8554974ba63125ab625f1b8fe8150ecc94dc714b6c99c0285bcbe10"
Sep 18 20:24:36.484236 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:36.484208    2549 generic.go:356] "Generic (PLEG): container finished" podID="88fd4c9d-852a-4b49-98b6-62b7a424ac02" containerID="c851a39b859abf3fcf30e96a318d7f3323104a4bcdb3ee76624b1ce78f4d4653" exitCode=137
Sep 18 20:24:36.484359 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:36.484246    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7741-ab5743"
Sep 18 20:24:36.484359 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:36.484259    2549 scope.go:117] "RemoveContainer" containerID="c851a39b859abf3fcf30e96a318d7f3323104a4bcdb3ee76624b1ce78f4d4653"
Sep 18 20:24:36.487492 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:24:36.487461    2549 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7741-ab5743\" is forbidden: User \"system:node:ip-10-0-140-178.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-178.ec2.internal' and this object" podUID="88fd4c9d-852a-4b49-98b6-62b7a424ac02" pod="opendatahub/test-curl-7741-ab5743"
Sep 18 20:24:36.494524 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:36.494503    2549 scope.go:117] "RemoveContainer" containerID="c851a39b859abf3fcf30e96a318d7f3323104a4bcdb3ee76624b1ce78f4d4653"
Sep 18 20:24:36.494834 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:24:36.494817    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c851a39b859abf3fcf30e96a318d7f3323104a4bcdb3ee76624b1ce78f4d4653\": container with ID starting with c851a39b859abf3fcf30e96a318d7f3323104a4bcdb3ee76624b1ce78f4d4653 not found: ID does not exist" containerID="c851a39b859abf3fcf30e96a318d7f3323104a4bcdb3ee76624b1ce78f4d4653"
Sep 18 20:24:36.494877 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:36.494854    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c851a39b859abf3fcf30e96a318d7f3323104a4bcdb3ee76624b1ce78f4d4653"} err="failed to get container status \"c851a39b859abf3fcf30e96a318d7f3323104a4bcdb3ee76624b1ce78f4d4653\": rpc error: code = NotFound desc = could not find container \"c851a39b859abf3fcf30e96a318d7f3323104a4bcdb3ee76624b1ce78f4d4653\": container with ID starting with c851a39b859abf3fcf30e96a318d7f3323104a4bcdb3ee76624b1ce78f4d4653 not found: ID does not exist"
Sep 18 20:24:36.501568 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:24:36.501535    2549 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7741-ab5743\" is forbidden: User \"system:node:ip-10-0-140-178.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-178.ec2.internal' and this object" podUID="88fd4c9d-852a-4b49-98b6-62b7a424ac02" pod="opendatahub/test-curl-7741-ab5743"
Sep 18 20:24:36.503858 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:24:36.503827    2549 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7741-ab5743\" is forbidden: User \"system:node:ip-10-0-140-178.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-178.ec2.internal' and this object" podUID="88fd4c9d-852a-4b49-98b6-62b7a424ac02" pod="opendatahub/test-curl-7741-ab5743"
Sep 18 20:24:37.523144 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:37.523102    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88fd4c9d-852a-4b49-98b6-62b7a424ac02" path="/var/lib/kubelet/pods/88fd4c9d-852a-4b49-98b6-62b7a424ac02/volumes"
Sep 18 20:24:42.111348 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:42.111311    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-40a1a5/sub-test-model-40a1a5-kserve-cbb55bd7-6zjdf"]
Sep 18 20:24:42.111977 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:42.111955    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="586e3f0c-8db8-4147-849c-0ce53ad249bb" containerName="revoke-keys"
Sep 18 20:24:42.112043 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:42.111981    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="586e3f0c-8db8-4147-849c-0ce53ad249bb" containerName="revoke-keys"
Sep 18 20:24:42.112077 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:42.112067    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="586e3f0c-8db8-4147-849c-0ce53ad249bb" containerName="revoke-keys"
Sep 18 20:24:42.114976 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:42.114955    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-40a1a5/sub-test-model-40a1a5-kserve-cbb55bd7-6zjdf"
Sep 18 20:24:42.121674 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:42.121642    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-40a1a5\"/\"sub-test-model-40a1a5-kserve-self-signed-certs\""
Sep 18 20:24:42.128912 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:42.128875    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-40a1a5/sub-test-model-40a1a5-kserve-cbb55bd7-6zjdf"]
Sep 18 20:24:42.274095 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:42.274052    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ed6c4d4-2a7f-40d1-8cbb-7784adf5b134-dshm\") pod \"sub-test-model-40a1a5-kserve-cbb55bd7-6zjdf\" (UID: \"2ed6c4d4-2a7f-40d1-8cbb-7784adf5b134\") " pod="ai-tenant-e2e-shared-a-w4-40a1a5/sub-test-model-40a1a5-kserve-cbb55bd7-6zjdf"
Sep 18 20:24:42.274290 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:42.274108    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ed6c4d4-2a7f-40d1-8cbb-7784adf5b134-tls-certs\") pod \"sub-test-model-40a1a5-kserve-cbb55bd7-6zjdf\" (UID: \"2ed6c4d4-2a7f-40d1-8cbb-7784adf5b134\") " pod="ai-tenant-e2e-shared-a-w4-40a1a5/sub-test-model-40a1a5-kserve-cbb55bd7-6zjdf"
Sep 18 20:24:42.274290 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:42.274141    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg9n2\" (UniqueName: \"kubernetes.io/projected/2ed6c4d4-2a7f-40d1-8cbb-7784adf5b134-kube-api-access-vg9n2\") pod \"sub-test-model-40a1a5-kserve-cbb55bd7-6zjdf\" (UID: \"2ed6c4d4-2a7f-40d1-8cbb-7784adf5b134\") " pod="ai-tenant-e2e-shared-a-w4-40a1a5/sub-test-model-40a1a5-kserve-cbb55bd7-6zjdf"
Sep 18 20:24:42.274290 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:42.274205    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ed6c4d4-2a7f-40d1-8cbb-7784adf5b134-model-cache\") pod \"sub-test-model-40a1a5-kserve-cbb55bd7-6zjdf\" (UID: \"2ed6c4d4-2a7f-40d1-8cbb-7784adf5b134\") " pod="ai-tenant-e2e-shared-a-w4-40a1a5/sub-test-model-40a1a5-kserve-cbb55bd7-6zjdf"
Sep 18 20:24:42.274440 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:42.274291    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ed6c4d4-2a7f-40d1-8cbb-7784adf5b134-tmp-dir\") pod \"sub-test-model-40a1a5-kserve-cbb55bd7-6zjdf\" (UID: \"2ed6c4d4-2a7f-40d1-8cbb-7784adf5b134\") " pod="ai-tenant-e2e-shared-a-w4-40a1a5/sub-test-model-40a1a5-kserve-cbb55bd7-6zjdf"
Sep 18 20:24:42.274440 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:42.274321    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ed6c4d4-2a7f-40d1-8cbb-7784adf5b134-home\") pod \"sub-test-model-40a1a5-kserve-cbb55bd7-6zjdf\" (UID: \"2ed6c4d4-2a7f-40d1-8cbb-7784adf5b134\") " pod="ai-tenant-e2e-shared-a-w4-40a1a5/sub-test-model-40a1a5-kserve-cbb55bd7-6zjdf"
Sep 18 20:24:42.375888 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:42.375779    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ed6c4d4-2a7f-40d1-8cbb-7784adf5b134-model-cache\") pod \"sub-test-model-40a1a5-kserve-cbb55bd7-6zjdf\" (UID: \"2ed6c4d4-2a7f-40d1-8cbb-7784adf5b134\") " pod="ai-tenant-e2e-shared-a-w4-40a1a5/sub-test-model-40a1a5-kserve-cbb55bd7-6zjdf"
Sep 18 20:24:42.376047 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:42.375902    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ed6c4d4-2a7f-40d1-8cbb-7784adf5b134-tmp-dir\") pod \"sub-test-model-40a1a5-kserve-cbb55bd7-6zjdf\" (UID: \"2ed6c4d4-2a7f-40d1-8cbb-7784adf5b134\") " pod="ai-tenant-e2e-shared-a-w4-40a1a5/sub-test-model-40a1a5-kserve-cbb55bd7-6zjdf"
Sep 18 20:24:42.376047 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:42.375937    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ed6c4d4-2a7f-40d1-8cbb-7784adf5b134-home\") pod \"sub-test-model-40a1a5-kserve-cbb55bd7-6zjdf\" (UID: \"2ed6c4d4-2a7f-40d1-8cbb-7784adf5b134\") " pod="ai-tenant-e2e-shared-a-w4-40a1a5/sub-test-model-40a1a5-kserve-cbb55bd7-6zjdf"
Sep 18 20:24:42.376047 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:42.375961    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ed6c4d4-2a7f-40d1-8cbb-7784adf5b134-dshm\") pod \"sub-test-model-40a1a5-kserve-cbb55bd7-6zjdf\" (UID: \"2ed6c4d4-2a7f-40d1-8cbb-7784adf5b134\") " pod="ai-tenant-e2e-shared-a-w4-40a1a5/sub-test-model-40a1a5-kserve-cbb55bd7-6zjdf"
Sep 18 20:24:42.376047 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:42.376009    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ed6c4d4-2a7f-40d1-8cbb-7784adf5b134-tls-certs\") pod \"sub-test-model-40a1a5-kserve-cbb55bd7-6zjdf\" (UID: \"2ed6c4d4-2a7f-40d1-8cbb-7784adf5b134\") " pod="ai-tenant-e2e-shared-a-w4-40a1a5/sub-test-model-40a1a5-kserve-cbb55bd7-6zjdf"
Sep 18 20:24:42.376047 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:42.376034    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg9n2\" (UniqueName: \"kubernetes.io/projected/2ed6c4d4-2a7f-40d1-8cbb-7784adf5b134-kube-api-access-vg9n2\") pod \"sub-test-model-40a1a5-kserve-cbb55bd7-6zjdf\" (UID: \"2ed6c4d4-2a7f-40d1-8cbb-7784adf5b134\") " pod="ai-tenant-e2e-shared-a-w4-40a1a5/sub-test-model-40a1a5-kserve-cbb55bd7-6zjdf"
Sep 18 20:24:42.376302 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:42.376249    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ed6c4d4-2a7f-40d1-8cbb-7784adf5b134-model-cache\") pod \"sub-test-model-40a1a5-kserve-cbb55bd7-6zjdf\" (UID: \"2ed6c4d4-2a7f-40d1-8cbb-7784adf5b134\") " pod="ai-tenant-e2e-shared-a-w4-40a1a5/sub-test-model-40a1a5-kserve-cbb55bd7-6zjdf"
Sep 18 20:24:42.376362 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:42.376315    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ed6c4d4-2a7f-40d1-8cbb-7784adf5b134-home\") pod \"sub-test-model-40a1a5-kserve-cbb55bd7-6zjdf\" (UID: \"2ed6c4d4-2a7f-40d1-8cbb-7784adf5b134\") " pod="ai-tenant-e2e-shared-a-w4-40a1a5/sub-test-model-40a1a5-kserve-cbb55bd7-6zjdf"
Sep 18 20:24:42.376489 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:42.376459    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ed6c4d4-2a7f-40d1-8cbb-7784adf5b134-tmp-dir\") pod \"sub-test-model-40a1a5-kserve-cbb55bd7-6zjdf\" (UID: \"2ed6c4d4-2a7f-40d1-8cbb-7784adf5b134\") " pod="ai-tenant-e2e-shared-a-w4-40a1a5/sub-test-model-40a1a5-kserve-cbb55bd7-6zjdf"
Sep 18 20:24:42.378434 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:42.378369    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ed6c4d4-2a7f-40d1-8cbb-7784adf5b134-dshm\") pod \"sub-test-model-40a1a5-kserve-cbb55bd7-6zjdf\" (UID: \"2ed6c4d4-2a7f-40d1-8cbb-7784adf5b134\") " pod="ai-tenant-e2e-shared-a-w4-40a1a5/sub-test-model-40a1a5-kserve-cbb55bd7-6zjdf"
Sep 18 20:24:42.378568 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:42.378551    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ed6c4d4-2a7f-40d1-8cbb-7784adf5b134-tls-certs\") pod \"sub-test-model-40a1a5-kserve-cbb55bd7-6zjdf\" (UID: \"2ed6c4d4-2a7f-40d1-8cbb-7784adf5b134\") " pod="ai-tenant-e2e-shared-a-w4-40a1a5/sub-test-model-40a1a5-kserve-cbb55bd7-6zjdf"
Sep 18 20:24:42.386124 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:42.386092    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg9n2\" (UniqueName: \"kubernetes.io/projected/2ed6c4d4-2a7f-40d1-8cbb-7784adf5b134-kube-api-access-vg9n2\") pod \"sub-test-model-40a1a5-kserve-cbb55bd7-6zjdf\" (UID: \"2ed6c4d4-2a7f-40d1-8cbb-7784adf5b134\") " pod="ai-tenant-e2e-shared-a-w4-40a1a5/sub-test-model-40a1a5-kserve-cbb55bd7-6zjdf"
Sep 18 20:24:42.427341 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:42.427299    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-40a1a5/sub-test-model-40a1a5-kserve-cbb55bd7-6zjdf"
Sep 18 20:24:42.785085 ip-10-0-140-178 kubenswrapper[2549]: W0918 20:24:42.785049    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2ed6c4d4_2a7f_40d1_8cbb_7784adf5b134.slice/crio-c41bdd4f383e3739012f12c6bd2625be762584f0dd993257acc0ffe3ef28db0b WatchSource:0}: Error finding container c41bdd4f383e3739012f12c6bd2625be762584f0dd993257acc0ffe3ef28db0b: Status 404 returned error can't find the container with id c41bdd4f383e3739012f12c6bd2625be762584f0dd993257acc0ffe3ef28db0b
Sep 18 20:24:42.785247 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:42.785164    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-40a1a5/sub-test-model-40a1a5-kserve-cbb55bd7-6zjdf"]
Sep 18 20:24:43.536261 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:43.536194    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-40a1a5/sub-test-model-40a1a5-kserve-cbb55bd7-6zjdf" event={"ID":"2ed6c4d4-2a7f-40d1-8cbb-7784adf5b134","Type":"ContainerStarted","Data":"5033b1b96a20814eb570cce1eaa19a960091ccf30a5bd96a91494084a6057850"}
Sep 18 20:24:43.536794 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:43.536284    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-40a1a5/sub-test-model-40a1a5-kserve-cbb55bd7-6zjdf" event={"ID":"2ed6c4d4-2a7f-40d1-8cbb-7784adf5b134","Type":"ContainerStarted","Data":"c41bdd4f383e3739012f12c6bd2625be762584f0dd993257acc0ffe3ef28db0b"}
Sep 18 20:24:43.562361 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:43.562282    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-40a1a5/sub-test-model-40a1a5-kserve-cbb55bd7-6zjdf" podStartSLOduration=1.562260079 podStartE2EDuration="1.562260079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:24:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:24:43.559528555 +0000 UTC m=+1668.513269723" watchObservedRunningTime="2026-09-18 20:24:43.562260079 +0000 UTC m=+1668.516001247"
Sep 18 20:24:45.130450 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:45.130408    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:24:48.534855 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:48.534805    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-40a1a5/auth-test-model-40a1a5-kserve-78fdfcfdc5-6f279" podUID="d4b5a5a7-c19d-45eb-b615-5fec088e0610" containerName="main" containerID="cri-o://bdfee2be0eb4780f32cfbecd76d9efd0e6041edb7cfb7f70c949adacdb27e60f" gracePeriod=45
Sep 18 20:24:48.799579 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:48.799506    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-40a1a5/auth-test-model-40a1a5-kserve-78fdfcfdc5-6f279"
Sep 18 20:24:48.950847 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:48.950808    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4b5a5a7-c19d-45eb-b615-5fec088e0610-home\") pod \"d4b5a5a7-c19d-45eb-b615-5fec088e0610\" (UID: \"d4b5a5a7-c19d-45eb-b615-5fec088e0610\") "
Sep 18 20:24:48.951049 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:48.950888    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9jslc\" (UniqueName: \"kubernetes.io/projected/d4b5a5a7-c19d-45eb-b615-5fec088e0610-kube-api-access-9jslc\") pod \"d4b5a5a7-c19d-45eb-b615-5fec088e0610\" (UID: \"d4b5a5a7-c19d-45eb-b615-5fec088e0610\") "
Sep 18 20:24:48.951049 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:48.950931    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4b5a5a7-c19d-45eb-b615-5fec088e0610-tls-certs\") pod \"d4b5a5a7-c19d-45eb-b615-5fec088e0610\" (UID: \"d4b5a5a7-c19d-45eb-b615-5fec088e0610\") "
Sep 18 20:24:48.951049 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:48.951040    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4b5a5a7-c19d-45eb-b615-5fec088e0610-dshm\") pod \"d4b5a5a7-c19d-45eb-b615-5fec088e0610\" (UID: \"d4b5a5a7-c19d-45eb-b615-5fec088e0610\") "
Sep 18 20:24:48.951234 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:48.951073    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4b5a5a7-c19d-45eb-b615-5fec088e0610-tmp-dir\") pod \"d4b5a5a7-c19d-45eb-b615-5fec088e0610\" (UID: \"d4b5a5a7-c19d-45eb-b615-5fec088e0610\") "
Sep 18 20:24:48.951234 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:48.951117    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4b5a5a7-c19d-45eb-b615-5fec088e0610-model-cache\") pod \"d4b5a5a7-c19d-45eb-b615-5fec088e0610\" (UID: \"d4b5a5a7-c19d-45eb-b615-5fec088e0610\") "
Sep 18 20:24:48.951234 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:48.951213    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4b5a5a7-c19d-45eb-b615-5fec088e0610-home" (OuterVolumeSpecName: "home") pod "d4b5a5a7-c19d-45eb-b615-5fec088e0610" (UID: "d4b5a5a7-c19d-45eb-b615-5fec088e0610"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:24:48.951733 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:48.951452    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4b5a5a7-c19d-45eb-b615-5fec088e0610-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d4b5a5a7-c19d-45eb-b615-5fec088e0610" (UID: "d4b5a5a7-c19d-45eb-b615-5fec088e0610"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:24:48.951733 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:48.951572    2549 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4b5a5a7-c19d-45eb-b615-5fec088e0610-tmp-dir\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:24:48.951733 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:48.951594    2549 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4b5a5a7-c19d-45eb-b615-5fec088e0610-home\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:24:48.951733 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:48.951690    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4b5a5a7-c19d-45eb-b615-5fec088e0610-model-cache" (OuterVolumeSpecName: "model-cache") pod "d4b5a5a7-c19d-45eb-b615-5fec088e0610" (UID: "d4b5a5a7-c19d-45eb-b615-5fec088e0610"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:24:48.953829 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:48.953784    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4b5a5a7-c19d-45eb-b615-5fec088e0610-dshm" (OuterVolumeSpecName: "dshm") pod "d4b5a5a7-c19d-45eb-b615-5fec088e0610" (UID: "d4b5a5a7-c19d-45eb-b615-5fec088e0610"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:24:48.953951 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:48.953905    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4b5a5a7-c19d-45eb-b615-5fec088e0610-kube-api-access-9jslc" (OuterVolumeSpecName: "kube-api-access-9jslc") pod "d4b5a5a7-c19d-45eb-b615-5fec088e0610" (UID: "d4b5a5a7-c19d-45eb-b615-5fec088e0610"). InnerVolumeSpecName "kube-api-access-9jslc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:24:48.954022 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:48.953950    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4b5a5a7-c19d-45eb-b615-5fec088e0610-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d4b5a5a7-c19d-45eb-b615-5fec088e0610" (UID: "d4b5a5a7-c19d-45eb-b615-5fec088e0610"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:24:49.052901 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:49.052780    2549 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4b5a5a7-c19d-45eb-b615-5fec088e0610-dshm\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:24:49.052901 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:49.052823    2549 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4b5a5a7-c19d-45eb-b615-5fec088e0610-model-cache\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:24:49.052901 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:49.052838    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jslc\" (UniqueName: \"kubernetes.io/projected/d4b5a5a7-c19d-45eb-b615-5fec088e0610-kube-api-access-9jslc\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:24:49.052901 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:49.052850    2549 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4b5a5a7-c19d-45eb-b615-5fec088e0610-tls-certs\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:24:49.292359 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:49.292317    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-69f1ba-vphnh"]
Sep 18 20:24:49.296636 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:49.296598    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-69f1ba-vphnh"]
Sep 18 20:24:49.523063 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:49.523025    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="586e3f0c-8db8-4147-849c-0ce53ad249bb" path="/var/lib/kubelet/pods/586e3f0c-8db8-4147-849c-0ce53ad249bb/volumes"
Sep 18 20:24:49.583240 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:49.583193    2549 generic.go:356] "Generic (PLEG): container finished" podID="d4b5a5a7-c19d-45eb-b615-5fec088e0610" containerID="bdfee2be0eb4780f32cfbecd76d9efd0e6041edb7cfb7f70c949adacdb27e60f" exitCode=0
Sep 18 20:24:49.583757 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:49.583254    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-40a1a5/auth-test-model-40a1a5-kserve-78fdfcfdc5-6f279"
Sep 18 20:24:49.583757 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:49.583265    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-40a1a5/auth-test-model-40a1a5-kserve-78fdfcfdc5-6f279" event={"ID":"d4b5a5a7-c19d-45eb-b615-5fec088e0610","Type":"ContainerDied","Data":"bdfee2be0eb4780f32cfbecd76d9efd0e6041edb7cfb7f70c949adacdb27e60f"}
Sep 18 20:24:49.583757 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:49.583314    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-40a1a5/auth-test-model-40a1a5-kserve-78fdfcfdc5-6f279" event={"ID":"d4b5a5a7-c19d-45eb-b615-5fec088e0610","Type":"ContainerDied","Data":"d0f02fa4e5526ccb49231d5f064a83f510280710632f37738c96487bbce4ca28"}
Sep 18 20:24:49.583757 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:49.583332    2549 scope.go:117] "RemoveContainer" containerID="bdfee2be0eb4780f32cfbecd76d9efd0e6041edb7cfb7f70c949adacdb27e60f"
Sep 18 20:24:49.593857 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:49.593835    2549 scope.go:117] "RemoveContainer" containerID="bdfee2be0eb4780f32cfbecd76d9efd0e6041edb7cfb7f70c949adacdb27e60f"
Sep 18 20:24:49.594233 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:24:49.594204    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdfee2be0eb4780f32cfbecd76d9efd0e6041edb7cfb7f70c949adacdb27e60f\": container with ID starting with bdfee2be0eb4780f32cfbecd76d9efd0e6041edb7cfb7f70c949adacdb27e60f not found: ID does not exist" containerID="bdfee2be0eb4780f32cfbecd76d9efd0e6041edb7cfb7f70c949adacdb27e60f"
Sep 18 20:24:49.594310 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:49.594246    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdfee2be0eb4780f32cfbecd76d9efd0e6041edb7cfb7f70c949adacdb27e60f"} err="failed to get container status \"bdfee2be0eb4780f32cfbecd76d9efd0e6041edb7cfb7f70c949adacdb27e60f\": rpc error: code = NotFound desc = could not find container \"bdfee2be0eb4780f32cfbecd76d9efd0e6041edb7cfb7f70c949adacdb27e60f\": container with ID starting with bdfee2be0eb4780f32cfbecd76d9efd0e6041edb7cfb7f70c949adacdb27e60f not found: ID does not exist"
Sep 18 20:24:49.603337 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:49.603297    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-40a1a5/auth-test-model-40a1a5-kserve-78fdfcfdc5-6f279"]
Sep 18 20:24:49.608557 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:49.608527    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-40a1a5/auth-test-model-40a1a5-kserve-78fdfcfdc5-6f279"]
Sep 18 20:24:51.522405 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:51.522348    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4b5a5a7-c19d-45eb-b615-5fec088e0610" path="/var/lib/kubelet/pods/d4b5a5a7-c19d-45eb-b615-5fec088e0610/volumes"
Sep 18 20:24:52.427917 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:52.427873    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-40a1a5/sub-test-model-40a1a5-kserve-cbb55bd7-6zjdf"
Sep 18 20:24:52.428113 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:52.427937    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-40a1a5/sub-test-model-40a1a5-kserve-cbb55bd7-6zjdf"
Sep 18 20:24:52.440750 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:52.440719    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-40a1a5/sub-test-model-40a1a5-kserve-cbb55bd7-6zjdf"
Sep 18 20:24:52.616625 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:52.616590    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-40a1a5/sub-test-model-40a1a5-kserve-cbb55bd7-6zjdf"
Sep 18 20:24:55.919897 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:55.919847    2549 scope.go:117] "RemoveContainer" containerID="57c31bd7525fa1dcae02846285c858deba96e873551a16273c1e0cb74d1540a8"
Sep 18 20:24:55.984055 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:55.984012    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-69f1ba-openshift-default-569c8d9dd6-wn99g"]
Sep 18 20:24:55.984363 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:55.984314    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-69f1ba-openshift-default-569c8d9dd6-wn99g" podUID="e4652b23-aa53-474e-afdf-6984f3fb1b63" containerName="istio-proxy" containerID="cri-o://bd5d0c51eefc0d6fe6119d633bc0bff9c9e2a5818dd6e34ec4c0a19d3df8624b" gracePeriod=30
Sep 18 20:24:56.362101 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:56.362058    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4cd712-68vkn"]
Sep 18 20:24:56.362668 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:56.362646    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4b5a5a7-c19d-45eb-b615-5fec088e0610" containerName="main"
Sep 18 20:24:56.362668 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:56.362669    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4b5a5a7-c19d-45eb-b615-5fec088e0610" containerName="main"
Sep 18 20:24:56.362856 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:56.362780    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4b5a5a7-c19d-45eb-b615-5fec088e0610" containerName="main"
Sep 18 20:24:56.366230 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:56.366203    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4cd712-68vkn"
Sep 18 20:24:56.369660 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:56.369636    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 20:24:56.370277 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:56.370262    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bhssz\""
Sep 18 20:24:56.376674 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:56.376637    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4cd712-68vkn"]
Sep 18 20:24:56.427247 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:56.427204    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cad5e488-0c69-40da-80d6-a1a6b0825235-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4cd712-68vkn\" (UID: \"cad5e488-0c69-40da-80d6-a1a6b0825235\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4cd712-68vkn"
Sep 18 20:24:56.527974 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:56.527924    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cad5e488-0c69-40da-80d6-a1a6b0825235-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4cd712-68vkn\" (UID: \"cad5e488-0c69-40da-80d6-a1a6b0825235\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4cd712-68vkn"
Sep 18 20:24:56.528609 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:56.528590    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cad5e488-0c69-40da-80d6-a1a6b0825235-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4cd712-68vkn\" (UID: \"cad5e488-0c69-40da-80d6-a1a6b0825235\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4cd712-68vkn"
Sep 18 20:24:56.678836 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:56.678735    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4cd712-68vkn"
Sep 18 20:24:56.957211 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:56.957176    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4cd712-68vkn"]
Sep 18 20:24:56.959447 ip-10-0-140-178 kubenswrapper[2549]: W0918 20:24:56.959415    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcad5e488_0c69_40da_80d6_a1a6b0825235.slice/crio-6308d4727252d8b8e928c56d83a6bc29c76bf93a575f87e8432201e3add13832 WatchSource:0}: Error finding container 6308d4727252d8b8e928c56d83a6bc29c76bf93a575f87e8432201e3add13832: Status 404 returned error can't find the container with id 6308d4727252d8b8e928c56d83a6bc29c76bf93a575f87e8432201e3add13832
Sep 18 20:24:57.639748 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:57.639705    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4cd712-68vkn" event={"ID":"cad5e488-0c69-40da-80d6-a1a6b0825235","Type":"ContainerStarted","Data":"9ea1b06fe957dc13b428598f64215b5b50f5c47eec26063276114ec1d69fb279"}
Sep 18 20:24:57.639948 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:57.639766    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4cd712-68vkn" event={"ID":"cad5e488-0c69-40da-80d6-a1a6b0825235","Type":"ContainerStarted","Data":"6308d4727252d8b8e928c56d83a6bc29c76bf93a575f87e8432201e3add13832"}
Sep 18 20:24:57.663128 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:57.663076    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4cd712-68vkn" podStartSLOduration=1.663057127 podStartE2EDuration="1.663057127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:24:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:24:57.658165619 +0000 UTC m=+1682.611906786" watchObservedRunningTime="2026-09-18 20:24:57.663057127 +0000 UTC m=+1682.616798294"
Sep 18 20:24:58.648351 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:58.648313    2549 generic.go:356] "Generic (PLEG): container finished" podID="cad5e488-0c69-40da-80d6-a1a6b0825235" containerID="9ea1b06fe957dc13b428598f64215b5b50f5c47eec26063276114ec1d69fb279" exitCode=0
Sep 18 20:24:58.648808 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:58.648434    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4cd712-68vkn" event={"ID":"cad5e488-0c69-40da-80d6-a1a6b0825235","Type":"ContainerDied","Data":"9ea1b06fe957dc13b428598f64215b5b50f5c47eec26063276114ec1d69fb279"}
Sep 18 20:24:59.806981 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:59.806950    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4cd712-68vkn"
Sep 18 20:24:59.867884 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:59.867774    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cad5e488-0c69-40da-80d6-a1a6b0825235-maas-api-service-ca\") pod \"cad5e488-0c69-40da-80d6-a1a6b0825235\" (UID: \"cad5e488-0c69-40da-80d6-a1a6b0825235\") "
Sep 18 20:24:59.868136 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:59.868112    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cad5e488-0c69-40da-80d6-a1a6b0825235-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cad5e488-0c69-40da-80d6-a1a6b0825235" (UID: "cad5e488-0c69-40da-80d6-a1a6b0825235"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:24:59.868292 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:24:59.868276    2549 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cad5e488-0c69-40da-80d6-a1a6b0825235-maas-api-service-ca\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:25:00.661817 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:00.661780    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4cd712-68vkn"
Sep 18 20:25:00.662055 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:00.661811    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4cd712-68vkn" event={"ID":"cad5e488-0c69-40da-80d6-a1a6b0825235","Type":"ContainerDied","Data":"6308d4727252d8b8e928c56d83a6bc29c76bf93a575f87e8432201e3add13832"}
Sep 18 20:25:00.662055 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:00.661864    2549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6308d4727252d8b8e928c56d83a6bc29c76bf93a575f87e8432201e3add13832"
Sep 18 20:25:01.252614 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:01.252535    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-69f1ba-openshift-default-569c8d9dd6-wn99g"
Sep 18 20:25:01.284532 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:01.284500    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4652b23-aa53-474e-afdf-6984f3fb1b63-istio-data\") pod \"e4652b23-aa53-474e-afdf-6984f3fb1b63\" (UID: \"e4652b23-aa53-474e-afdf-6984f3fb1b63\") "
Sep 18 20:25:01.284739 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:01.284560    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bqr9f\" (UniqueName: \"kubernetes.io/projected/e4652b23-aa53-474e-afdf-6984f3fb1b63-kube-api-access-bqr9f\") pod \"e4652b23-aa53-474e-afdf-6984f3fb1b63\" (UID: \"e4652b23-aa53-474e-afdf-6984f3fb1b63\") "
Sep 18 20:25:01.284739 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:01.284632    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4652b23-aa53-474e-afdf-6984f3fb1b63-istio-envoy\") pod \"e4652b23-aa53-474e-afdf-6984f3fb1b63\" (UID: \"e4652b23-aa53-474e-afdf-6984f3fb1b63\") "
Sep 18 20:25:01.284739 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:01.284676    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4652b23-aa53-474e-afdf-6984f3fb1b63-workload-socket\") pod \"e4652b23-aa53-474e-afdf-6984f3fb1b63\" (UID: \"e4652b23-aa53-474e-afdf-6984f3fb1b63\") "
Sep 18 20:25:01.284739 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:01.284737    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4652b23-aa53-474e-afdf-6984f3fb1b63-istio-token\") pod \"e4652b23-aa53-474e-afdf-6984f3fb1b63\" (UID: \"e4652b23-aa53-474e-afdf-6984f3fb1b63\") "
Sep 18 20:25:01.284963 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:01.284786    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4652b23-aa53-474e-afdf-6984f3fb1b63-istiod-ca-cert\") pod \"e4652b23-aa53-474e-afdf-6984f3fb1b63\" (UID: \"e4652b23-aa53-474e-afdf-6984f3fb1b63\") "
Sep 18 20:25:01.284963 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:01.284827    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4652b23-aa53-474e-afdf-6984f3fb1b63-credential-socket\") pod \"e4652b23-aa53-474e-afdf-6984f3fb1b63\" (UID: \"e4652b23-aa53-474e-afdf-6984f3fb1b63\") "
Sep 18 20:25:01.284963 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:01.284839    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4652b23-aa53-474e-afdf-6984f3fb1b63-istio-data" (OuterVolumeSpecName: "istio-data") pod "e4652b23-aa53-474e-afdf-6984f3fb1b63" (UID: "e4652b23-aa53-474e-afdf-6984f3fb1b63"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:25:01.284963 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:01.284899    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4652b23-aa53-474e-afdf-6984f3fb1b63-istio-podinfo\") pod \"e4652b23-aa53-474e-afdf-6984f3fb1b63\" (UID: \"e4652b23-aa53-474e-afdf-6984f3fb1b63\") "
Sep 18 20:25:01.284963 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:01.284925    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4652b23-aa53-474e-afdf-6984f3fb1b63-workload-certs\") pod \"e4652b23-aa53-474e-afdf-6984f3fb1b63\" (UID: \"e4652b23-aa53-474e-afdf-6984f3fb1b63\") "
Sep 18 20:25:01.285223 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:01.284992    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4652b23-aa53-474e-afdf-6984f3fb1b63-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e4652b23-aa53-474e-afdf-6984f3fb1b63" (UID: "e4652b23-aa53-474e-afdf-6984f3fb1b63"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:25:01.285223 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:01.285061    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4652b23-aa53-474e-afdf-6984f3fb1b63-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e4652b23-aa53-474e-afdf-6984f3fb1b63" (UID: "e4652b23-aa53-474e-afdf-6984f3fb1b63"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:25:01.285223 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:01.285150    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4652b23-aa53-474e-afdf-6984f3fb1b63-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e4652b23-aa53-474e-afdf-6984f3fb1b63" (UID: "e4652b23-aa53-474e-afdf-6984f3fb1b63"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:25:01.285412 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:01.285230    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4652b23-aa53-474e-afdf-6984f3fb1b63-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e4652b23-aa53-474e-afdf-6984f3fb1b63" (UID: "e4652b23-aa53-474e-afdf-6984f3fb1b63"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:25:01.285505 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:01.285490    2549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4652b23-aa53-474e-afdf-6984f3fb1b63-istiod-ca-cert\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:25:01.285560 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:01.285512    2549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4652b23-aa53-474e-afdf-6984f3fb1b63-credential-socket\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:25:01.285560 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:01.285528    2549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4652b23-aa53-474e-afdf-6984f3fb1b63-workload-certs\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:25:01.285560 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:01.285541    2549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4652b23-aa53-474e-afdf-6984f3fb1b63-istio-data\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:25:01.285560 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:01.285554    2549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4652b23-aa53-474e-afdf-6984f3fb1b63-workload-socket\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:25:01.287198 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:01.287164    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e4652b23-aa53-474e-afdf-6984f3fb1b63-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e4652b23-aa53-474e-afdf-6984f3fb1b63" (UID: "e4652b23-aa53-474e-afdf-6984f3fb1b63"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 20:25:01.287519 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:01.287486    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4652b23-aa53-474e-afdf-6984f3fb1b63-istio-token" (OuterVolumeSpecName: "istio-token") pod "e4652b23-aa53-474e-afdf-6984f3fb1b63" (UID: "e4652b23-aa53-474e-afdf-6984f3fb1b63"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:25:01.287623 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:01.287562    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4652b23-aa53-474e-afdf-6984f3fb1b63-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e4652b23-aa53-474e-afdf-6984f3fb1b63" (UID: "e4652b23-aa53-474e-afdf-6984f3fb1b63"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:25:01.287623 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:01.287588    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4652b23-aa53-474e-afdf-6984f3fb1b63-kube-api-access-bqr9f" (OuterVolumeSpecName: "kube-api-access-bqr9f") pod "e4652b23-aa53-474e-afdf-6984f3fb1b63" (UID: "e4652b23-aa53-474e-afdf-6984f3fb1b63"). InnerVolumeSpecName "kube-api-access-bqr9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:25:01.386899 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:01.386853    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqr9f\" (UniqueName: \"kubernetes.io/projected/e4652b23-aa53-474e-afdf-6984f3fb1b63-kube-api-access-bqr9f\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:25:01.386899 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:01.386892    2549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4652b23-aa53-474e-afdf-6984f3fb1b63-istio-envoy\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:25:01.386899 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:01.386903    2549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4652b23-aa53-474e-afdf-6984f3fb1b63-istio-token\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:25:01.386899 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:01.386912    2549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4652b23-aa53-474e-afdf-6984f3fb1b63-istio-podinfo\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:25:01.670995 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:01.670954    2549 generic.go:356] "Generic (PLEG): container finished" podID="e4652b23-aa53-474e-afdf-6984f3fb1b63" containerID="bd5d0c51eefc0d6fe6119d633bc0bff9c9e2a5818dd6e34ec4c0a19d3df8624b" exitCode=0
Sep 18 20:25:01.671228 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:01.671002    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-69f1ba-openshift-default-569c8d9dd6-wn99g"
Sep 18 20:25:01.671228 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:01.671039    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-69f1ba-openshift-default-569c8d9dd6-wn99g" event={"ID":"e4652b23-aa53-474e-afdf-6984f3fb1b63","Type":"ContainerDied","Data":"bd5d0c51eefc0d6fe6119d633bc0bff9c9e2a5818dd6e34ec4c0a19d3df8624b"}
Sep 18 20:25:01.671228 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:01.671102    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-69f1ba-openshift-default-569c8d9dd6-wn99g" event={"ID":"e4652b23-aa53-474e-afdf-6984f3fb1b63","Type":"ContainerDied","Data":"852438eacc0bc3de1dd86acb7a15489e946e69e638dc7aa11241eb6ecb867719"}
Sep 18 20:25:01.671228 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:01.671127    2549 scope.go:117] "RemoveContainer" containerID="bd5d0c51eefc0d6fe6119d633bc0bff9c9e2a5818dd6e34ec4c0a19d3df8624b"
Sep 18 20:25:01.681234 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:01.681212    2549 scope.go:117] "RemoveContainer" containerID="0bbb01dc316b53c46e7843ad2513a970f76c6f3c113a2b81f2457802b6699fcc"
Sep 18 20:25:01.691339 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:01.691315    2549 scope.go:117] "RemoveContainer" containerID="bd5d0c51eefc0d6fe6119d633bc0bff9c9e2a5818dd6e34ec4c0a19d3df8624b"
Sep 18 20:25:01.691782 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:25:01.691757    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd5d0c51eefc0d6fe6119d633bc0bff9c9e2a5818dd6e34ec4c0a19d3df8624b\": container with ID starting with bd5d0c51eefc0d6fe6119d633bc0bff9c9e2a5818dd6e34ec4c0a19d3df8624b not found: ID does not exist" containerID="bd5d0c51eefc0d6fe6119d633bc0bff9c9e2a5818dd6e34ec4c0a19d3df8624b"
Sep 18 20:25:01.691861 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:01.691794    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd5d0c51eefc0d6fe6119d633bc0bff9c9e2a5818dd6e34ec4c0a19d3df8624b"} err="failed to get container status \"bd5d0c51eefc0d6fe6119d633bc0bff9c9e2a5818dd6e34ec4c0a19d3df8624b\": rpc error: code = NotFound desc = could not find container \"bd5d0c51eefc0d6fe6119d633bc0bff9c9e2a5818dd6e34ec4c0a19d3df8624b\": container with ID starting with bd5d0c51eefc0d6fe6119d633bc0bff9c9e2a5818dd6e34ec4c0a19d3df8624b not found: ID does not exist"
Sep 18 20:25:01.691861 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:01.691816    2549 scope.go:117] "RemoveContainer" containerID="0bbb01dc316b53c46e7843ad2513a970f76c6f3c113a2b81f2457802b6699fcc"
Sep 18 20:25:01.692151 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:25:01.692130    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bbb01dc316b53c46e7843ad2513a970f76c6f3c113a2b81f2457802b6699fcc\": container with ID starting with 0bbb01dc316b53c46e7843ad2513a970f76c6f3c113a2b81f2457802b6699fcc not found: ID does not exist" containerID="0bbb01dc316b53c46e7843ad2513a970f76c6f3c113a2b81f2457802b6699fcc"
Sep 18 20:25:01.692217 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:01.692160    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bbb01dc316b53c46e7843ad2513a970f76c6f3c113a2b81f2457802b6699fcc"} err="failed to get container status \"0bbb01dc316b53c46e7843ad2513a970f76c6f3c113a2b81f2457802b6699fcc\": rpc error: code = NotFound desc = could not find container \"0bbb01dc316b53c46e7843ad2513a970f76c6f3c113a2b81f2457802b6699fcc\": container with ID starting with 0bbb01dc316b53c46e7843ad2513a970f76c6f3c113a2b81f2457802b6699fcc not found: ID does not exist"
Sep 18 20:25:01.693583 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:01.693559    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-69f1ba-openshift-default-569c8d9dd6-wn99g"]
Sep 18 20:25:01.699192 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:01.699164    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-69f1ba-openshift-default-569c8d9dd6-wn99g"]
Sep 18 20:25:01.816805 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:01.816686    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:25:03.529255 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:03.529214    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4652b23-aa53-474e-afdf-6984f3fb1b63" path="/var/lib/kubelet/pods/e4652b23-aa53-474e-afdf-6984f3fb1b63/volumes"
Sep 18 20:25:12.197151 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:12.197103    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4cd712-68vkn"]
Sep 18 20:25:12.201113 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:12.201079    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4cd712-68vkn"]
Sep 18 20:25:13.521957 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:13.521920    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cad5e488-0c69-40da-80d6-a1a6b0825235" path="/var/lib/kubelet/pods/cad5e488-0c69-40da-80d6-a1a6b0825235/volumes"
Sep 18 20:25:19.377806 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:19.377711    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526"]
Sep 18 20:25:19.378250 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:19.377956    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526" podUID="1994521d-e96d-4507-b0a4-d841c5545735" containerName="istio-proxy" containerID="cri-o://b96a04fd7a22e01135a8d75ec655ec9d7270cad2d7925a7da66e01b886d0b47d" gracePeriod=30
Sep 18 20:25:19.523817 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:19.523785    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:25:19.716160 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:19.716071    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5319c6-bncqn"]
Sep 18 20:25:19.716543 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:19.716530    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4652b23-aa53-474e-afdf-6984f3fb1b63" containerName="istio-proxy"
Sep 18 20:25:19.716602 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:19.716545    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4652b23-aa53-474e-afdf-6984f3fb1b63" containerName="istio-proxy"
Sep 18 20:25:19.716602 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:19.716556    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4652b23-aa53-474e-afdf-6984f3fb1b63" containerName="istio-proxy"
Sep 18 20:25:19.716602 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:19.716563    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4652b23-aa53-474e-afdf-6984f3fb1b63" containerName="istio-proxy"
Sep 18 20:25:19.716602 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:19.716573    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cad5e488-0c69-40da-80d6-a1a6b0825235" containerName="revoke-keys"
Sep 18 20:25:19.716602 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:19.716578    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="cad5e488-0c69-40da-80d6-a1a6b0825235" containerName="revoke-keys"
Sep 18 20:25:19.716749 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:19.716643    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4652b23-aa53-474e-afdf-6984f3fb1b63" containerName="istio-proxy"
Sep 18 20:25:19.716749 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:19.716651    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="cad5e488-0c69-40da-80d6-a1a6b0825235" containerName="revoke-keys"
Sep 18 20:25:19.721570 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:19.721535    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5319c6-bncqn"
Sep 18 20:25:19.724512 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:19.724482    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 20:25:19.724512 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:19.724501    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bhssz\""
Sep 18 20:25:19.730258 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:19.729598    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5319c6-bncqn"]
Sep 18 20:25:19.894421 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:19.894365    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3413651-bb79-40a0-961a-e97312ec2278-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-5319c6-bncqn\" (UID: \"b3413651-bb79-40a0-961a-e97312ec2278\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5319c6-bncqn"
Sep 18 20:25:19.995943 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:19.995828    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3413651-bb79-40a0-961a-e97312ec2278-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-5319c6-bncqn\" (UID: \"b3413651-bb79-40a0-961a-e97312ec2278\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5319c6-bncqn"
Sep 18 20:25:19.996549 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:19.996527    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3413651-bb79-40a0-961a-e97312ec2278-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-5319c6-bncqn\" (UID: \"b3413651-bb79-40a0-961a-e97312ec2278\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5319c6-bncqn"
Sep 18 20:25:20.034797 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:20.034761    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5319c6-bncqn"
Sep 18 20:25:20.317058 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:20.317021    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5319c6-bncqn"]
Sep 18 20:25:20.318934 ip-10-0-140-178 kubenswrapper[2549]: W0918 20:25:20.318898    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3413651_bb79_40a0_961a_e97312ec2278.slice/crio-e5eed67e57211f3ccdb8ef7884a832612c808da935c03f821c5bec71883f95f0 WatchSource:0}: Error finding container e5eed67e57211f3ccdb8ef7884a832612c808da935c03f821c5bec71883f95f0: Status 404 returned error can't find the container with id e5eed67e57211f3ccdb8ef7884a832612c808da935c03f821c5bec71883f95f0
Sep 18 20:25:20.795156 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:20.795118    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5319c6-bncqn" event={"ID":"b3413651-bb79-40a0-961a-e97312ec2278","Type":"ContainerStarted","Data":"87ad04e7a988f6da60ae638f7beacdebc75ef008fc0c7e3d2c5e4c4468543722"}
Sep 18 20:25:20.795602 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:20.795180    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5319c6-bncqn" event={"ID":"b3413651-bb79-40a0-961a-e97312ec2278","Type":"ContainerStarted","Data":"e5eed67e57211f3ccdb8ef7884a832612c808da935c03f821c5bec71883f95f0"}
Sep 18 20:25:20.813320 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:20.813261    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5319c6-bncqn" podStartSLOduration=1.813241456 podStartE2EDuration="1.813241456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:25:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:25:20.811479929 +0000 UTC m=+1705.765221098" watchObservedRunningTime="2026-09-18 20:25:20.813241456 +0000 UTC m=+1705.766982656"
Sep 18 20:25:21.198638 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:25:21.198593    2549 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3413651_bb79_40a0_961a_e97312ec2278.slice/crio-conmon-87ad04e7a988f6da60ae638f7beacdebc75ef008fc0c7e3d2c5e4c4468543722.scope\": RecentStats: unable to find data in memory cache]"
Sep 18 20:25:21.804165 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:21.804127    2549 generic.go:356] "Generic (PLEG): container finished" podID="b3413651-bb79-40a0-961a-e97312ec2278" containerID="87ad04e7a988f6da60ae638f7beacdebc75ef008fc0c7e3d2c5e4c4468543722" exitCode=0
Sep 18 20:25:21.804619 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:21.804204    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5319c6-bncqn" event={"ID":"b3413651-bb79-40a0-961a-e97312ec2278","Type":"ContainerDied","Data":"87ad04e7a988f6da60ae638f7beacdebc75ef008fc0c7e3d2c5e4c4468543722"}
Sep 18 20:25:22.964739 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:22.964703    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5319c6-bncqn"
Sep 18 20:25:23.129880 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:23.129840    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3413651-bb79-40a0-961a-e97312ec2278-maas-api-service-ca\") pod \"b3413651-bb79-40a0-961a-e97312ec2278\" (UID: \"b3413651-bb79-40a0-961a-e97312ec2278\") "
Sep 18 20:25:23.130257 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:23.130230    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3413651-bb79-40a0-961a-e97312ec2278-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b3413651-bb79-40a0-961a-e97312ec2278" (UID: "b3413651-bb79-40a0-961a-e97312ec2278"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:25:23.231610 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:23.231569    2549 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3413651-bb79-40a0-961a-e97312ec2278-maas-api-service-ca\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:25:23.817518 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:23.817417    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5319c6-bncqn" event={"ID":"b3413651-bb79-40a0-961a-e97312ec2278","Type":"ContainerDied","Data":"e5eed67e57211f3ccdb8ef7884a832612c808da935c03f821c5bec71883f95f0"}
Sep 18 20:25:23.817518 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:23.817491    2549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e5eed67e57211f3ccdb8ef7884a832612c808da935c03f821c5bec71883f95f0"
Sep 18 20:25:23.817518 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:23.817431    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5319c6-bncqn"
Sep 18 20:25:24.648302 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:24.648224    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526"
Sep 18 20:25:24.747239 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:24.747192    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hrdnl\" (UniqueName: \"kubernetes.io/projected/1994521d-e96d-4507-b0a4-d841c5545735-kube-api-access-hrdnl\") pod \"1994521d-e96d-4507-b0a4-d841c5545735\" (UID: \"1994521d-e96d-4507-b0a4-d841c5545735\") "
Sep 18 20:25:24.747497 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:24.747294    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1994521d-e96d-4507-b0a4-d841c5545735-istio-data\") pod \"1994521d-e96d-4507-b0a4-d841c5545735\" (UID: \"1994521d-e96d-4507-b0a4-d841c5545735\") "
Sep 18 20:25:24.747497 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:24.747397    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1994521d-e96d-4507-b0a4-d841c5545735-istio-token\") pod \"1994521d-e96d-4507-b0a4-d841c5545735\" (UID: \"1994521d-e96d-4507-b0a4-d841c5545735\") "
Sep 18 20:25:24.747497 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:24.747442    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1994521d-e96d-4507-b0a4-d841c5545735-credential-socket\") pod \"1994521d-e96d-4507-b0a4-d841c5545735\" (UID: \"1994521d-e96d-4507-b0a4-d841c5545735\") "
Sep 18 20:25:24.747685 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:24.747609    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1994521d-e96d-4507-b0a4-d841c5545735-istio-data" (OuterVolumeSpecName: "istio-data") pod "1994521d-e96d-4507-b0a4-d841c5545735" (UID: "1994521d-e96d-4507-b0a4-d841c5545735"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:25:24.747744 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:24.747721    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1994521d-e96d-4507-b0a4-d841c5545735-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1994521d-e96d-4507-b0a4-d841c5545735" (UID: "1994521d-e96d-4507-b0a4-d841c5545735"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:25:24.747867 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:24.747848    2549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1994521d-e96d-4507-b0a4-d841c5545735-credential-socket\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:25:24.747930 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:24.747871    2549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1994521d-e96d-4507-b0a4-d841c5545735-istio-data\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:25:24.749655 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:24.749623    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1994521d-e96d-4507-b0a4-d841c5545735-istio-token" (OuterVolumeSpecName: "istio-token") pod "1994521d-e96d-4507-b0a4-d841c5545735" (UID: "1994521d-e96d-4507-b0a4-d841c5545735"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:25:24.749784 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:24.749683    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1994521d-e96d-4507-b0a4-d841c5545735-kube-api-access-hrdnl" (OuterVolumeSpecName: "kube-api-access-hrdnl") pod "1994521d-e96d-4507-b0a4-d841c5545735" (UID: "1994521d-e96d-4507-b0a4-d841c5545735"). InnerVolumeSpecName "kube-api-access-hrdnl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:25:24.825682 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:24.825647    2549 generic.go:356] "Generic (PLEG): container finished" podID="1994521d-e96d-4507-b0a4-d841c5545735" containerID="b96a04fd7a22e01135a8d75ec655ec9d7270cad2d7925a7da66e01b886d0b47d" exitCode=0
Sep 18 20:25:24.825872 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:24.825707    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526"
Sep 18 20:25:24.825872 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:24.825732    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526" event={"ID":"1994521d-e96d-4507-b0a4-d841c5545735","Type":"ContainerDied","Data":"b96a04fd7a22e01135a8d75ec655ec9d7270cad2d7925a7da66e01b886d0b47d"}
Sep 18 20:25:24.825872 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:24.825794    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526" event={"ID":"1994521d-e96d-4507-b0a4-d841c5545735","Type":"ContainerDied","Data":"b42ab4a4ee7559061ab4e44e930bf606a30fac08fb3a4295a610c1f584fe6ca8"}
Sep 18 20:25:24.825872 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:24.825830    2549 scope.go:117] "RemoveContainer" containerID="b96a04fd7a22e01135a8d75ec655ec9d7270cad2d7925a7da66e01b886d0b47d"
Sep 18 20:25:24.838855 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:24.838830    2549 scope.go:117] "RemoveContainer" containerID="723501b3501ceb59e617c1423c1d2ba60699b340de98a90ab474413ac4af2227"
Sep 18 20:25:24.848349 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:24.848281    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1994521d-e96d-4507-b0a4-d841c5545735-istiod-ca-cert\") pod \"1994521d-e96d-4507-b0a4-d841c5545735\" (UID: \"1994521d-e96d-4507-b0a4-d841c5545735\") "
Sep 18 20:25:24.848349 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:24.848325    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1994521d-e96d-4507-b0a4-d841c5545735-workload-certs\") pod \"1994521d-e96d-4507-b0a4-d841c5545735\" (UID: \"1994521d-e96d-4507-b0a4-d841c5545735\") "
Sep 18 20:25:24.848651 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:24.848348    2549 scope.go:117] "RemoveContainer" containerID="b96a04fd7a22e01135a8d75ec655ec9d7270cad2d7925a7da66e01b886d0b47d"
Sep 18 20:25:24.848651 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:24.848358    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1994521d-e96d-4507-b0a4-d841c5545735-workload-socket\") pod \"1994521d-e96d-4507-b0a4-d841c5545735\" (UID: \"1994521d-e96d-4507-b0a4-d841c5545735\") "
Sep 18 20:25:24.848651 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:24.848555    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1994521d-e96d-4507-b0a4-d841c5545735-istio-envoy\") pod \"1994521d-e96d-4507-b0a4-d841c5545735\" (UID: \"1994521d-e96d-4507-b0a4-d841c5545735\") "
Sep 18 20:25:24.848651 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:24.848598    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1994521d-e96d-4507-b0a4-d841c5545735-istio-podinfo\") pod \"1994521d-e96d-4507-b0a4-d841c5545735\" (UID: \"1994521d-e96d-4507-b0a4-d841c5545735\") "
Sep 18 20:25:24.848651 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:24.848606    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1994521d-e96d-4507-b0a4-d841c5545735-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1994521d-e96d-4507-b0a4-d841c5545735" (UID: "1994521d-e96d-4507-b0a4-d841c5545735"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:25:24.848909 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:24.848710    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1994521d-e96d-4507-b0a4-d841c5545735-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1994521d-e96d-4507-b0a4-d841c5545735" (UID: "1994521d-e96d-4507-b0a4-d841c5545735"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:25:24.848969 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:24.848933    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1994521d-e96d-4507-b0a4-d841c5545735-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1994521d-e96d-4507-b0a4-d841c5545735" (UID: "1994521d-e96d-4507-b0a4-d841c5545735"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:25:24.849087 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:24.849067    2549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1994521d-e96d-4507-b0a4-d841c5545735-istiod-ca-cert\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:25:24.849168 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:24.849092    2549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1994521d-e96d-4507-b0a4-d841c5545735-workload-certs\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:25:24.849168 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:24.849106    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrdnl\" (UniqueName: \"kubernetes.io/projected/1994521d-e96d-4507-b0a4-d841c5545735-kube-api-access-hrdnl\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:25:24.849168 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:24.849120    2549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1994521d-e96d-4507-b0a4-d841c5545735-workload-socket\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:25:24.849168 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:25:24.849119    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b96a04fd7a22e01135a8d75ec655ec9d7270cad2d7925a7da66e01b886d0b47d\": container with ID starting with b96a04fd7a22e01135a8d75ec655ec9d7270cad2d7925a7da66e01b886d0b47d not found: ID does not exist" containerID="b96a04fd7a22e01135a8d75ec655ec9d7270cad2d7925a7da66e01b886d0b47d"
Sep 18 20:25:24.849168 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:24.849136    2549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1994521d-e96d-4507-b0a4-d841c5545735-istio-token\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:25:24.849168 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:24.849156    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b96a04fd7a22e01135a8d75ec655ec9d7270cad2d7925a7da66e01b886d0b47d"} err="failed to get container status \"b96a04fd7a22e01135a8d75ec655ec9d7270cad2d7925a7da66e01b886d0b47d\": rpc error: code = NotFound desc = could not find container \"b96a04fd7a22e01135a8d75ec655ec9d7270cad2d7925a7da66e01b886d0b47d\": container with ID starting with b96a04fd7a22e01135a8d75ec655ec9d7270cad2d7925a7da66e01b886d0b47d not found: ID does not exist"
Sep 18 20:25:24.849501 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:24.849181    2549 scope.go:117] "RemoveContainer" containerID="723501b3501ceb59e617c1423c1d2ba60699b340de98a90ab474413ac4af2227"
Sep 18 20:25:24.849664 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:25:24.849628    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"723501b3501ceb59e617c1423c1d2ba60699b340de98a90ab474413ac4af2227\": container with ID starting with 723501b3501ceb59e617c1423c1d2ba60699b340de98a90ab474413ac4af2227 not found: ID does not exist" containerID="723501b3501ceb59e617c1423c1d2ba60699b340de98a90ab474413ac4af2227"
Sep 18 20:25:24.849809 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:24.849674    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"723501b3501ceb59e617c1423c1d2ba60699b340de98a90ab474413ac4af2227"} err="failed to get container status \"723501b3501ceb59e617c1423c1d2ba60699b340de98a90ab474413ac4af2227\": rpc error: code = NotFound desc = could not find container \"723501b3501ceb59e617c1423c1d2ba60699b340de98a90ab474413ac4af2227\": container with ID starting with 723501b3501ceb59e617c1423c1d2ba60699b340de98a90ab474413ac4af2227 not found: ID does not exist"
Sep 18 20:25:24.850965 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:24.850942    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1994521d-e96d-4507-b0a4-d841c5545735-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1994521d-e96d-4507-b0a4-d841c5545735" (UID: "1994521d-e96d-4507-b0a4-d841c5545735"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 20:25:24.851059 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:24.850995    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1994521d-e96d-4507-b0a4-d841c5545735-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1994521d-e96d-4507-b0a4-d841c5545735" (UID: "1994521d-e96d-4507-b0a4-d841c5545735"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:25:24.949946 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:24.949851    2549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1994521d-e96d-4507-b0a4-d841c5545735-istio-envoy\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:25:24.949946 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:24.949883    2549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1994521d-e96d-4507-b0a4-d841c5545735-istio-podinfo\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:25:25.151506 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:25.151466    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526"]
Sep 18 20:25:25.156453 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:25.156415    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-4cd712-openshift-default-85df66dcb4-p8526"]
Sep 18 20:25:25.523634 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:25.523598    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1994521d-e96d-4507-b0a4-d841c5545735" path="/var/lib/kubelet/pods/1994521d-e96d-4507-b0a4-d841c5545735/volumes"
Sep 18 20:25:30.706068 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:30.706019    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5319c6-bncqn"]
Sep 18 20:25:30.708745 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:30.708719    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5319c6-bncqn"]
Sep 18 20:25:31.523316 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:31.523274    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3413651-bb79-40a0-961a-e97312ec2278" path="/var/lib/kubelet/pods/b3413651-bb79-40a0-961a-e97312ec2278/volumes"
Sep 18 20:25:36.223619 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:36.223578    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:25:37.337967 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:37.337918    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x"]
Sep 18 20:25:37.338468 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:37.338228    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x" podUID="7a68b8bc-a551-4666-98ce-909b507edd2d" containerName="istio-proxy" containerID="cri-o://0e4dc9e0562572920eff16d4bb3f3371a3f9f7f0fa4188cba4c03c8838dd52ff" gracePeriod=30
Sep 18 20:25:37.691043 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:37.690939    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c5206e-wrxwj"]
Sep 18 20:25:37.691416 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:37.691404    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1994521d-e96d-4507-b0a4-d841c5545735" containerName="istio-proxy"
Sep 18 20:25:37.691470 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:37.691418    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="1994521d-e96d-4507-b0a4-d841c5545735" containerName="istio-proxy"
Sep 18 20:25:37.691470 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:37.691431    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1994521d-e96d-4507-b0a4-d841c5545735" containerName="istio-proxy"
Sep 18 20:25:37.691470 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:37.691437    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="1994521d-e96d-4507-b0a4-d841c5545735" containerName="istio-proxy"
Sep 18 20:25:37.691470 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:37.691449    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3413651-bb79-40a0-961a-e97312ec2278" containerName="revoke-keys"
Sep 18 20:25:37.691470 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:37.691455    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3413651-bb79-40a0-961a-e97312ec2278" containerName="revoke-keys"
Sep 18 20:25:37.691622 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:37.691520    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3413651-bb79-40a0-961a-e97312ec2278" containerName="revoke-keys"
Sep 18 20:25:37.691622 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:37.691531    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="1994521d-e96d-4507-b0a4-d841c5545735" containerName="istio-proxy"
Sep 18 20:25:37.691622 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:37.691541    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4652b23-aa53-474e-afdf-6984f3fb1b63" containerName="istio-proxy"
Sep 18 20:25:37.695107 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:37.695087    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c5206e-wrxwj"
Sep 18 20:25:37.698273 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:37.698246    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 20:25:37.698877 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:37.698852    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bhssz\""
Sep 18 20:25:37.704330 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:37.704299    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c5206e-wrxwj"]
Sep 18 20:25:37.799152 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:37.799111    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b3c948e-a535-4b43-9026-05f4cbbb2847-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-c5206e-wrxwj\" (UID: \"2b3c948e-a535-4b43-9026-05f4cbbb2847\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c5206e-wrxwj"
Sep 18 20:25:37.900645 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:37.900591    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b3c948e-a535-4b43-9026-05f4cbbb2847-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-c5206e-wrxwj\" (UID: \"2b3c948e-a535-4b43-9026-05f4cbbb2847\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c5206e-wrxwj"
Sep 18 20:25:37.901254 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:37.901231    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b3c948e-a535-4b43-9026-05f4cbbb2847-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-c5206e-wrxwj\" (UID: \"2b3c948e-a535-4b43-9026-05f4cbbb2847\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c5206e-wrxwj"
Sep 18 20:25:38.007814 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:38.007718    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c5206e-wrxwj"
Sep 18 20:25:38.289704 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:38.289666    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c5206e-wrxwj"]
Sep 18 20:25:38.291948 ip-10-0-140-178 kubenswrapper[2549]: W0918 20:25:38.291915    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b3c948e_a535_4b43_9026_05f4cbbb2847.slice/crio-a850c23252f268047abad61659d590c1b34c578e3b60d5231cb926f2d38c7c1a WatchSource:0}: Error finding container a850c23252f268047abad61659d590c1b34c578e3b60d5231cb926f2d38c7c1a: Status 404 returned error can't find the container with id a850c23252f268047abad61659d590c1b34c578e3b60d5231cb926f2d38c7c1a
Sep 18 20:25:38.917319 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:38.917280    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c5206e-wrxwj" event={"ID":"2b3c948e-a535-4b43-9026-05f4cbbb2847","Type":"ContainerStarted","Data":"b0bfab383108f8cfa7e6d268d9bc5030a8ea708078b0d4925597cc0bbc7dd862"}
Sep 18 20:25:38.917781 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:38.917337    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c5206e-wrxwj" event={"ID":"2b3c948e-a535-4b43-9026-05f4cbbb2847","Type":"ContainerStarted","Data":"a850c23252f268047abad61659d590c1b34c578e3b60d5231cb926f2d38c7c1a"}
Sep 18 20:25:38.953666 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:38.953600    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c5206e-wrxwj" podStartSLOduration=1.953581658 podStartE2EDuration="1.953581658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:25:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:25:38.952005435 +0000 UTC m=+1723.905746603" watchObservedRunningTime="2026-09-18 20:25:38.953581658 +0000 UTC m=+1723.907322825"
Sep 18 20:25:39.925507 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:39.925468    2549 generic.go:356] "Generic (PLEG): container finished" podID="2b3c948e-a535-4b43-9026-05f4cbbb2847" containerID="b0bfab383108f8cfa7e6d268d9bc5030a8ea708078b0d4925597cc0bbc7dd862" exitCode=0
Sep 18 20:25:39.925938 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:39.925555    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c5206e-wrxwj" event={"ID":"2b3c948e-a535-4b43-9026-05f4cbbb2847","Type":"ContainerDied","Data":"b0bfab383108f8cfa7e6d268d9bc5030a8ea708078b0d4925597cc0bbc7dd862"}
Sep 18 20:25:40.217558 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:40.217459    2549 patch_prober.go:28] interesting pod/e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 18 20:25:40.217558 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:40.217538    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x" podUID="7a68b8bc-a551-4666-98ce-909b507edd2d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:25:41.079948 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:41.079908    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c5206e-wrxwj"
Sep 18 20:25:41.237814 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:41.237715    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b3c948e-a535-4b43-9026-05f4cbbb2847-maas-api-service-ca\") pod \"2b3c948e-a535-4b43-9026-05f4cbbb2847\" (UID: \"2b3c948e-a535-4b43-9026-05f4cbbb2847\") "
Sep 18 20:25:41.238112 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:41.238086    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b3c948e-a535-4b43-9026-05f4cbbb2847-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2b3c948e-a535-4b43-9026-05f4cbbb2847" (UID: "2b3c948e-a535-4b43-9026-05f4cbbb2847"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:25:41.339344 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:41.339294    2549 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b3c948e-a535-4b43-9026-05f4cbbb2847-maas-api-service-ca\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:25:41.942231 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:41.942200    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c5206e-wrxwj"
Sep 18 20:25:41.942447 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:41.942233    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c5206e-wrxwj" event={"ID":"2b3c948e-a535-4b43-9026-05f4cbbb2847","Type":"ContainerDied","Data":"a850c23252f268047abad61659d590c1b34c578e3b60d5231cb926f2d38c7c1a"}
Sep 18 20:25:41.942447 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:41.942283    2549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a850c23252f268047abad61659d590c1b34c578e3b60d5231cb926f2d38c7c1a"
Sep 18 20:25:42.606590 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:42.606503    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x"
Sep 18 20:25:42.753031 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:42.752989    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d2mkw\" (UniqueName: \"kubernetes.io/projected/7a68b8bc-a551-4666-98ce-909b507edd2d-kube-api-access-d2mkw\") pod \"7a68b8bc-a551-4666-98ce-909b507edd2d\" (UID: \"7a68b8bc-a551-4666-98ce-909b507edd2d\") "
Sep 18 20:25:42.753246 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:42.753099    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a68b8bc-a551-4666-98ce-909b507edd2d-istiod-ca-cert\") pod \"7a68b8bc-a551-4666-98ce-909b507edd2d\" (UID: \"7a68b8bc-a551-4666-98ce-909b507edd2d\") "
Sep 18 20:25:42.753246 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:42.753138    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a68b8bc-a551-4666-98ce-909b507edd2d-credential-socket\") pod \"7a68b8bc-a551-4666-98ce-909b507edd2d\" (UID: \"7a68b8bc-a551-4666-98ce-909b507edd2d\") "
Sep 18 20:25:42.753246 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:42.753160    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a68b8bc-a551-4666-98ce-909b507edd2d-istio-token\") pod \"7a68b8bc-a551-4666-98ce-909b507edd2d\" (UID: \"7a68b8bc-a551-4666-98ce-909b507edd2d\") "
Sep 18 20:25:42.753472 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:42.753248    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a68b8bc-a551-4666-98ce-909b507edd2d-istio-data\") pod \"7a68b8bc-a551-4666-98ce-909b507edd2d\" (UID: \"7a68b8bc-a551-4666-98ce-909b507edd2d\") "
Sep 18 20:25:42.753472 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:42.753299    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a68b8bc-a551-4666-98ce-909b507edd2d-workload-socket\") pod \"7a68b8bc-a551-4666-98ce-909b507edd2d\" (UID: \"7a68b8bc-a551-4666-98ce-909b507edd2d\") "
Sep 18 20:25:42.753472 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:42.753341    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a68b8bc-a551-4666-98ce-909b507edd2d-istio-envoy\") pod \"7a68b8bc-a551-4666-98ce-909b507edd2d\" (UID: \"7a68b8bc-a551-4666-98ce-909b507edd2d\") "
Sep 18 20:25:42.753472 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:42.753422    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a68b8bc-a551-4666-98ce-909b507edd2d-istio-podinfo\") pod \"7a68b8bc-a551-4666-98ce-909b507edd2d\" (UID: \"7a68b8bc-a551-4666-98ce-909b507edd2d\") "
Sep 18 20:25:42.753472 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:42.753457    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a68b8bc-a551-4666-98ce-909b507edd2d-workload-certs\") pod \"7a68b8bc-a551-4666-98ce-909b507edd2d\" (UID: \"7a68b8bc-a551-4666-98ce-909b507edd2d\") "
Sep 18 20:25:42.753709 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:42.753592    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a68b8bc-a551-4666-98ce-909b507edd2d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7a68b8bc-a551-4666-98ce-909b507edd2d" (UID: "7a68b8bc-a551-4666-98ce-909b507edd2d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:25:42.753709 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:42.753674    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a68b8bc-a551-4666-98ce-909b507edd2d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7a68b8bc-a551-4666-98ce-909b507edd2d" (UID: "7a68b8bc-a551-4666-98ce-909b507edd2d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:25:42.753943 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:42.753848    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a68b8bc-a551-4666-98ce-909b507edd2d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7a68b8bc-a551-4666-98ce-909b507edd2d" (UID: "7a68b8bc-a551-4666-98ce-909b507edd2d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:25:42.753943 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:42.753919    2549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a68b8bc-a551-4666-98ce-909b507edd2d-credential-socket\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:25:42.753943 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:42.753940    2549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a68b8bc-a551-4666-98ce-909b507edd2d-workload-socket\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:25:42.754177 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:42.753999    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a68b8bc-a551-4666-98ce-909b507edd2d-istio-data" (OuterVolumeSpecName: "istio-data") pod "7a68b8bc-a551-4666-98ce-909b507edd2d" (UID: "7a68b8bc-a551-4666-98ce-909b507edd2d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:25:42.754177 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:42.754090    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a68b8bc-a551-4666-98ce-909b507edd2d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7a68b8bc-a551-4666-98ce-909b507edd2d" (UID: "7a68b8bc-a551-4666-98ce-909b507edd2d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:25:42.756062 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:42.756032    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a68b8bc-a551-4666-98ce-909b507edd2d-kube-api-access-d2mkw" (OuterVolumeSpecName: "kube-api-access-d2mkw") pod "7a68b8bc-a551-4666-98ce-909b507edd2d" (UID: "7a68b8bc-a551-4666-98ce-909b507edd2d"). InnerVolumeSpecName "kube-api-access-d2mkw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:25:42.756309 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:42.756271    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7a68b8bc-a551-4666-98ce-909b507edd2d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7a68b8bc-a551-4666-98ce-909b507edd2d" (UID: "7a68b8bc-a551-4666-98ce-909b507edd2d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 20:25:42.756412 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:42.756364    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a68b8bc-a551-4666-98ce-909b507edd2d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7a68b8bc-a551-4666-98ce-909b507edd2d" (UID: "7a68b8bc-a551-4666-98ce-909b507edd2d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:25:42.756480 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:42.756396    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a68b8bc-a551-4666-98ce-909b507edd2d-istio-token" (OuterVolumeSpecName: "istio-token") pod "7a68b8bc-a551-4666-98ce-909b507edd2d" (UID: "7a68b8bc-a551-4666-98ce-909b507edd2d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:25:42.855164 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:42.855128    2549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a68b8bc-a551-4666-98ce-909b507edd2d-istio-podinfo\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:25:42.855164 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:42.855159    2549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a68b8bc-a551-4666-98ce-909b507edd2d-workload-certs\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:25:42.855164 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:42.855169    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2mkw\" (UniqueName: \"kubernetes.io/projected/7a68b8bc-a551-4666-98ce-909b507edd2d-kube-api-access-d2mkw\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:25:42.855481 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:42.855181    2549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a68b8bc-a551-4666-98ce-909b507edd2d-istiod-ca-cert\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:25:42.855481 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:42.855189    2549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a68b8bc-a551-4666-98ce-909b507edd2d-istio-token\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:25:42.855481 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:42.855196    2549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a68b8bc-a551-4666-98ce-909b507edd2d-istio-data\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:25:42.855481 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:42.855205    2549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a68b8bc-a551-4666-98ce-909b507edd2d-istio-envoy\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:25:42.952083 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:42.951977    2549 generic.go:356] "Generic (PLEG): container finished" podID="7a68b8bc-a551-4666-98ce-909b507edd2d" containerID="0e4dc9e0562572920eff16d4bb3f3371a3f9f7f0fa4188cba4c03c8838dd52ff" exitCode=0
Sep 18 20:25:42.952083 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:42.952028    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x" event={"ID":"7a68b8bc-a551-4666-98ce-909b507edd2d","Type":"ContainerDied","Data":"0e4dc9e0562572920eff16d4bb3f3371a3f9f7f0fa4188cba4c03c8838dd52ff"}
Sep 18 20:25:42.952328 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:42.952093    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x" event={"ID":"7a68b8bc-a551-4666-98ce-909b507edd2d","Type":"ContainerDied","Data":"ebf8228c1383cd476dd1cb5b57588d688b0c0bb5948d7baed869a63f0114aab1"}
Sep 18 20:25:42.952328 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:42.952037    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x"
Sep 18 20:25:42.952328 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:42.952117    2549 scope.go:117] "RemoveContainer" containerID="0e4dc9e0562572920eff16d4bb3f3371a3f9f7f0fa4188cba4c03c8838dd52ff"
Sep 18 20:25:42.963102 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:42.963078    2549 scope.go:117] "RemoveContainer" containerID="e66e5f718cf77246509150fcb67777489cc6f3bbe02afa754f9a6fd19c7d8d04"
Sep 18 20:25:42.972472 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:42.972451    2549 scope.go:117] "RemoveContainer" containerID="0e4dc9e0562572920eff16d4bb3f3371a3f9f7f0fa4188cba4c03c8838dd52ff"
Sep 18 20:25:42.972840 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:25:42.972818    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e4dc9e0562572920eff16d4bb3f3371a3f9f7f0fa4188cba4c03c8838dd52ff\": container with ID starting with 0e4dc9e0562572920eff16d4bb3f3371a3f9f7f0fa4188cba4c03c8838dd52ff not found: ID does not exist" containerID="0e4dc9e0562572920eff16d4bb3f3371a3f9f7f0fa4188cba4c03c8838dd52ff"
Sep 18 20:25:42.972930 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:42.972853    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e4dc9e0562572920eff16d4bb3f3371a3f9f7f0fa4188cba4c03c8838dd52ff"} err="failed to get container status \"0e4dc9e0562572920eff16d4bb3f3371a3f9f7f0fa4188cba4c03c8838dd52ff\": rpc error: code = NotFound desc = could not find container \"0e4dc9e0562572920eff16d4bb3f3371a3f9f7f0fa4188cba4c03c8838dd52ff\": container with ID starting with 0e4dc9e0562572920eff16d4bb3f3371a3f9f7f0fa4188cba4c03c8838dd52ff not found: ID does not exist"
Sep 18 20:25:42.972930 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:42.972879    2549 scope.go:117] "RemoveContainer" containerID="e66e5f718cf77246509150fcb67777489cc6f3bbe02afa754f9a6fd19c7d8d04"
Sep 18 20:25:42.973240 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:25:42.973212    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e66e5f718cf77246509150fcb67777489cc6f3bbe02afa754f9a6fd19c7d8d04\": container with ID starting with e66e5f718cf77246509150fcb67777489cc6f3bbe02afa754f9a6fd19c7d8d04 not found: ID does not exist" containerID="e66e5f718cf77246509150fcb67777489cc6f3bbe02afa754f9a6fd19c7d8d04"
Sep 18 20:25:42.973502 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:42.973252    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e66e5f718cf77246509150fcb67777489cc6f3bbe02afa754f9a6fd19c7d8d04"} err="failed to get container status \"e66e5f718cf77246509150fcb67777489cc6f3bbe02afa754f9a6fd19c7d8d04\": rpc error: code = NotFound desc = could not find container \"e66e5f718cf77246509150fcb67777489cc6f3bbe02afa754f9a6fd19c7d8d04\": container with ID starting with e66e5f718cf77246509150fcb67777489cc6f3bbe02afa754f9a6fd19c7d8d04 not found: ID does not exist"
Sep 18 20:25:42.976315 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:42.976289    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x"]
Sep 18 20:25:42.980822 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:42.980794    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-5319c6-openshift-default-59bffb687b-8m84x"]
Sep 18 20:25:43.522146 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:43.522110    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a68b8bc-a551-4666-98ce-909b507edd2d" path="/var/lib/kubelet/pods/7a68b8bc-a551-4666-98ce-909b507edd2d/volumes"
Sep 18 20:25:48.770766 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:48.770718    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c5206e-wrxwj"]
Sep 18 20:25:48.775118 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:48.775085    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c5206e-wrxwj"]
Sep 18 20:25:49.521716 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:49.521681    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b3c948e-a535-4b43-9026-05f4cbbb2847" path="/var/lib/kubelet/pods/2b3c948e-a535-4b43-9026-05f4cbbb2847/volumes"
Sep 18 20:25:51.232130 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:51.232093    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:25:55.273081 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:55.273043    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g"]
Sep 18 20:25:55.273623 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:55.273282    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g" podUID="a390c820-232c-4b8c-b4e4-711699f90d2b" containerName="istio-proxy" containerID="cri-o://774e37b5243071bde848d20fdd9c92ef4534f975e551ff4a5e8343118b09d395" gracePeriod=30
Sep 18 20:25:59.342056 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:59.342009    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7741-76f57f"]
Sep 18 20:25:59.342667 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:59.342645    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b3c948e-a535-4b43-9026-05f4cbbb2847" containerName="revoke-keys"
Sep 18 20:25:59.342758 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:59.342671    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b3c948e-a535-4b43-9026-05f4cbbb2847" containerName="revoke-keys"
Sep 18 20:25:59.342758 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:59.342684    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a68b8bc-a551-4666-98ce-909b507edd2d" containerName="istio-proxy"
Sep 18 20:25:59.342758 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:59.342692    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a68b8bc-a551-4666-98ce-909b507edd2d" containerName="istio-proxy"
Sep 18 20:25:59.342758 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:59.342714    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a68b8bc-a551-4666-98ce-909b507edd2d" containerName="istio-proxy"
Sep 18 20:25:59.342758 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:59.342722    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a68b8bc-a551-4666-98ce-909b507edd2d" containerName="istio-proxy"
Sep 18 20:25:59.343005 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:59.342844    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="1994521d-e96d-4507-b0a4-d841c5545735" containerName="istio-proxy"
Sep 18 20:25:59.343005 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:59.342858    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a68b8bc-a551-4666-98ce-909b507edd2d" containerName="istio-proxy"
Sep 18 20:25:59.343005 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:59.342868    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b3c948e-a535-4b43-9026-05f4cbbb2847" containerName="revoke-keys"
Sep 18 20:25:59.345843 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:59.345814    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7741-76f57f"
Sep 18 20:25:59.349803 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:59.349771    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-jcdsw\""
Sep 18 20:25:59.351545 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:59.351510    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7741-76f57f"]
Sep 18 20:25:59.417095 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:59.417038    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlwkb\" (UniqueName: \"kubernetes.io/projected/b0e692c7-53d5-4bfd-b9e4-12df9f17c1f5-kube-api-access-nlwkb\") pod \"test-curl-7741-76f57f\" (UID: \"b0e692c7-53d5-4bfd-b9e4-12df9f17c1f5\") " pod="opendatahub/test-curl-7741-76f57f"
Sep 18 20:25:59.517699 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:59.517668    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlwkb\" (UniqueName: \"kubernetes.io/projected/b0e692c7-53d5-4bfd-b9e4-12df9f17c1f5-kube-api-access-nlwkb\") pod \"test-curl-7741-76f57f\" (UID: \"b0e692c7-53d5-4bfd-b9e4-12df9f17c1f5\") " pod="opendatahub/test-curl-7741-76f57f"
Sep 18 20:25:59.527629 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:59.527598    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlwkb\" (UniqueName: \"kubernetes.io/projected/b0e692c7-53d5-4bfd-b9e4-12df9f17c1f5-kube-api-access-nlwkb\") pod \"test-curl-7741-76f57f\" (UID: \"b0e692c7-53d5-4bfd-b9e4-12df9f17c1f5\") " pod="opendatahub/test-curl-7741-76f57f"
Sep 18 20:25:59.658281 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:59.658181    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7741-76f57f"
Sep 18 20:25:59.935092 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:25:59.935041    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7741-76f57f"]
Sep 18 20:25:59.936817 ip-10-0-140-178 kubenswrapper[2549]: W0918 20:25:59.936787    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb0e692c7_53d5_4bfd_b9e4_12df9f17c1f5.slice/crio-b320361c0164090bb0aaba5cc46682cfc8bb44c0938974b88df6593c2e3f87c0 WatchSource:0}: Error finding container b320361c0164090bb0aaba5cc46682cfc8bb44c0938974b88df6593c2e3f87c0: Status 404 returned error can't find the container with id b320361c0164090bb0aaba5cc46682cfc8bb44c0938974b88df6593c2e3f87c0
Sep 18 20:26:00.060727 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:00.060688    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7741-76f57f" event={"ID":"b0e692c7-53d5-4bfd-b9e4-12df9f17c1f5","Type":"ContainerStarted","Data":"027bf6c92e5fb131b8eb844415febfb53a43e3d1132254dd17629da3864fdbf5"}
Sep 18 20:26:00.060727 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:00.060736    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7741-76f57f" event={"ID":"b0e692c7-53d5-4bfd-b9e4-12df9f17c1f5","Type":"ContainerStarted","Data":"b320361c0164090bb0aaba5cc46682cfc8bb44c0938974b88df6593c2e3f87c0"}
Sep 18 20:26:00.079318 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:00.079263    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7741-76f57f" podStartSLOduration=1.079243117 podStartE2EDuration="1.079243117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:25:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:26:00.077621749 +0000 UTC m=+1745.031362916" watchObservedRunningTime="2026-09-18 20:26:00.079243117 +0000 UTC m=+1745.032984348"
Sep 18 20:26:00.527882 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:00.527855    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g"
Sep 18 20:26:00.628063 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:00.628020    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a390c820-232c-4b8c-b4e4-711699f90d2b-istio-data\") pod \"a390c820-232c-4b8c-b4e4-711699f90d2b\" (UID: \"a390c820-232c-4b8c-b4e4-711699f90d2b\") "
Sep 18 20:26:00.628252 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:00.628083    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a390c820-232c-4b8c-b4e4-711699f90d2b-istio-token\") pod \"a390c820-232c-4b8c-b4e4-711699f90d2b\" (UID: \"a390c820-232c-4b8c-b4e4-711699f90d2b\") "
Sep 18 20:26:00.628252 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:00.628120    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a390c820-232c-4b8c-b4e4-711699f90d2b-credential-socket\") pod \"a390c820-232c-4b8c-b4e4-711699f90d2b\" (UID: \"a390c820-232c-4b8c-b4e4-711699f90d2b\") "
Sep 18 20:26:00.628252 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:00.628143    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a390c820-232c-4b8c-b4e4-711699f90d2b-workload-socket\") pod \"a390c820-232c-4b8c-b4e4-711699f90d2b\" (UID: \"a390c820-232c-4b8c-b4e4-711699f90d2b\") "
Sep 18 20:26:00.628252 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:00.628173    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a390c820-232c-4b8c-b4e4-711699f90d2b-istio-envoy\") pod \"a390c820-232c-4b8c-b4e4-711699f90d2b\" (UID: \"a390c820-232c-4b8c-b4e4-711699f90d2b\") "
Sep 18 20:26:00.628532 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:00.628249    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a390c820-232c-4b8c-b4e4-711699f90d2b-istiod-ca-cert\") pod \"a390c820-232c-4b8c-b4e4-711699f90d2b\" (UID: \"a390c820-232c-4b8c-b4e4-711699f90d2b\") "
Sep 18 20:26:00.628532 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:00.628315    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a390c820-232c-4b8c-b4e4-711699f90d2b-istio-podinfo\") pod \"a390c820-232c-4b8c-b4e4-711699f90d2b\" (UID: \"a390c820-232c-4b8c-b4e4-711699f90d2b\") "
Sep 18 20:26:00.628532 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:00.628475    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a390c820-232c-4b8c-b4e4-711699f90d2b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a390c820-232c-4b8c-b4e4-711699f90d2b" (UID: "a390c820-232c-4b8c-b4e4-711699f90d2b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:26:00.628532 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:00.628491    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a390c820-232c-4b8c-b4e4-711699f90d2b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a390c820-232c-4b8c-b4e4-711699f90d2b" (UID: "a390c820-232c-4b8c-b4e4-711699f90d2b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:26:00.628756 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:00.628471    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-plwgb\" (UniqueName: \"kubernetes.io/projected/a390c820-232c-4b8c-b4e4-711699f90d2b-kube-api-access-plwgb\") pod \"a390c820-232c-4b8c-b4e4-711699f90d2b\" (UID: \"a390c820-232c-4b8c-b4e4-711699f90d2b\") "
Sep 18 20:26:00.628756 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:00.628538    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a390c820-232c-4b8c-b4e4-711699f90d2b-istio-data" (OuterVolumeSpecName: "istio-data") pod "a390c820-232c-4b8c-b4e4-711699f90d2b" (UID: "a390c820-232c-4b8c-b4e4-711699f90d2b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:26:00.628756 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:00.628576    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a390c820-232c-4b8c-b4e4-711699f90d2b-workload-certs\") pod \"a390c820-232c-4b8c-b4e4-711699f90d2b\" (UID: \"a390c820-232c-4b8c-b4e4-711699f90d2b\") "
Sep 18 20:26:00.628756 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:00.628720    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a390c820-232c-4b8c-b4e4-711699f90d2b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a390c820-232c-4b8c-b4e4-711699f90d2b" (UID: "a390c820-232c-4b8c-b4e4-711699f90d2b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:26:00.628956 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:00.628917    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a390c820-232c-4b8c-b4e4-711699f90d2b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a390c820-232c-4b8c-b4e4-711699f90d2b" (UID: "a390c820-232c-4b8c-b4e4-711699f90d2b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:26:00.629059 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:00.629043    2549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a390c820-232c-4b8c-b4e4-711699f90d2b-credential-socket\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:26:00.629113 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:00.629060    2549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a390c820-232c-4b8c-b4e4-711699f90d2b-workload-socket\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:26:00.629113 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:00.629072    2549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a390c820-232c-4b8c-b4e4-711699f90d2b-istiod-ca-cert\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:26:00.629113 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:00.629081    2549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a390c820-232c-4b8c-b4e4-711699f90d2b-workload-certs\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:26:00.629113 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:00.629088    2549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a390c820-232c-4b8c-b4e4-711699f90d2b-istio-data\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:26:00.630645 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:00.630615    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a390c820-232c-4b8c-b4e4-711699f90d2b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a390c820-232c-4b8c-b4e4-711699f90d2b" (UID: "a390c820-232c-4b8c-b4e4-711699f90d2b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:26:00.630645 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:00.630633    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a390c820-232c-4b8c-b4e4-711699f90d2b-istio-token" (OuterVolumeSpecName: "istio-token") pod "a390c820-232c-4b8c-b4e4-711699f90d2b" (UID: "a390c820-232c-4b8c-b4e4-711699f90d2b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:26:00.630972 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:00.630948    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a390c820-232c-4b8c-b4e4-711699f90d2b-kube-api-access-plwgb" (OuterVolumeSpecName: "kube-api-access-plwgb") pod "a390c820-232c-4b8c-b4e4-711699f90d2b" (UID: "a390c820-232c-4b8c-b4e4-711699f90d2b"). InnerVolumeSpecName "kube-api-access-plwgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:26:00.631072 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:00.631018    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a390c820-232c-4b8c-b4e4-711699f90d2b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a390c820-232c-4b8c-b4e4-711699f90d2b" (UID: "a390c820-232c-4b8c-b4e4-711699f90d2b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 20:26:00.686308 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:00.686263    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7741-76f57f"]
Sep 18 20:26:00.691036 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:00.690999    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7741-76f57f"]
Sep 18 20:26:00.730101 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:00.730065    2549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a390c820-232c-4b8c-b4e4-711699f90d2b-istio-envoy\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:26:00.730101 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:00.730098    2549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a390c820-232c-4b8c-b4e4-711699f90d2b-istio-podinfo\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:26:00.730101 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:00.730108    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plwgb\" (UniqueName: \"kubernetes.io/projected/a390c820-232c-4b8c-b4e4-711699f90d2b-kube-api-access-plwgb\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:26:00.730358 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:00.730118    2549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a390c820-232c-4b8c-b4e4-711699f90d2b-istio-token\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:26:00.807301 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:00.807209    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7741-702e31"]
Sep 18 20:26:00.807638 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:00.807625    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a390c820-232c-4b8c-b4e4-711699f90d2b" containerName="istio-proxy"
Sep 18 20:26:00.807694 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:00.807640    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="a390c820-232c-4b8c-b4e4-711699f90d2b" containerName="istio-proxy"
Sep 18 20:26:00.807694 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:00.807651    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0e692c7-53d5-4bfd-b9e4-12df9f17c1f5" containerName="test-curl-7741-76f57f"
Sep 18 20:26:00.807694 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:00.807657    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0e692c7-53d5-4bfd-b9e4-12df9f17c1f5" containerName="test-curl-7741-76f57f"
Sep 18 20:26:00.807793 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:00.807735    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="a390c820-232c-4b8c-b4e4-711699f90d2b" containerName="istio-proxy"
Sep 18 20:26:00.807793 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:00.807745    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="a390c820-232c-4b8c-b4e4-711699f90d2b" containerName="istio-proxy"
Sep 18 20:26:00.807793 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:00.807751    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a68b8bc-a551-4666-98ce-909b507edd2d" containerName="istio-proxy"
Sep 18 20:26:00.807793 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:00.807760    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0e692c7-53d5-4bfd-b9e4-12df9f17c1f5" containerName="test-curl-7741-76f57f"
Sep 18 20:26:00.811115 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:00.811092    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7741-702e31"
Sep 18 20:26:00.819646 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:00.819529    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7741-702e31"]
Sep 18 20:26:00.830642 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:00.830604    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxm77\" (UniqueName: \"kubernetes.io/projected/d21052db-b539-41dc-ab2c-f77796d86070-kube-api-access-wxm77\") pod \"test-curl-7741-702e31\" (UID: \"d21052db-b539-41dc-ab2c-f77796d86070\") " pod="opendatahub/test-curl-7741-702e31"
Sep 18 20:26:00.931804 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:00.931764    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxm77\" (UniqueName: \"kubernetes.io/projected/d21052db-b539-41dc-ab2c-f77796d86070-kube-api-access-wxm77\") pod \"test-curl-7741-702e31\" (UID: \"d21052db-b539-41dc-ab2c-f77796d86070\") " pod="opendatahub/test-curl-7741-702e31"
Sep 18 20:26:00.945247 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:00.945213    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxm77\" (UniqueName: \"kubernetes.io/projected/d21052db-b539-41dc-ab2c-f77796d86070-kube-api-access-wxm77\") pod \"test-curl-7741-702e31\" (UID: \"d21052db-b539-41dc-ab2c-f77796d86070\") " pod="opendatahub/test-curl-7741-702e31"
Sep 18 20:26:01.009366 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:01.009320    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:26:01.069146 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:01.069036    2549 generic.go:356] "Generic (PLEG): container finished" podID="a390c820-232c-4b8c-b4e4-711699f90d2b" containerID="774e37b5243071bde848d20fdd9c92ef4534f975e551ff4a5e8343118b09d395" exitCode=0
Sep 18 20:26:01.069146 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:01.069092    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g"
Sep 18 20:26:01.069146 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:01.069120    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g" event={"ID":"a390c820-232c-4b8c-b4e4-711699f90d2b","Type":"ContainerDied","Data":"774e37b5243071bde848d20fdd9c92ef4534f975e551ff4a5e8343118b09d395"}
Sep 18 20:26:01.069474 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:01.069177    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g" event={"ID":"a390c820-232c-4b8c-b4e4-711699f90d2b","Type":"ContainerDied","Data":"e8065335562dd04210158558e6ca20a11bd0a1995669948e3d72ca85db731a8a"}
Sep 18 20:26:01.069474 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:01.069205    2549 scope.go:117] "RemoveContainer" containerID="774e37b5243071bde848d20fdd9c92ef4534f975e551ff4a5e8343118b09d395"
Sep 18 20:26:01.069592 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:26:01.069557    2549 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7741-76f57f" container="test-curl-7741-76f57f"
Sep 18 20:26:01.079403 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:01.079364    2549 scope.go:117] "RemoveContainer" containerID="7010c2fd412a05d5026549528e8b7e2cc026fcefacc2d6dec96f5c626d750063"
Sep 18 20:26:01.095936 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:01.095905    2549 scope.go:117] "RemoveContainer" containerID="774e37b5243071bde848d20fdd9c92ef4534f975e551ff4a5e8343118b09d395"
Sep 18 20:26:01.096293 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:26:01.096269    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"774e37b5243071bde848d20fdd9c92ef4534f975e551ff4a5e8343118b09d395\": container with ID starting with 774e37b5243071bde848d20fdd9c92ef4534f975e551ff4a5e8343118b09d395 not found: ID does not exist" containerID="774e37b5243071bde848d20fdd9c92ef4534f975e551ff4a5e8343118b09d395"
Sep 18 20:26:01.096352 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:01.096304    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"774e37b5243071bde848d20fdd9c92ef4534f975e551ff4a5e8343118b09d395"} err="failed to get container status \"774e37b5243071bde848d20fdd9c92ef4534f975e551ff4a5e8343118b09d395\": rpc error: code = NotFound desc = could not find container \"774e37b5243071bde848d20fdd9c92ef4534f975e551ff4a5e8343118b09d395\": container with ID starting with 774e37b5243071bde848d20fdd9c92ef4534f975e551ff4a5e8343118b09d395 not found: ID does not exist"
Sep 18 20:26:01.096352 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:01.096326    2549 scope.go:117] "RemoveContainer" containerID="7010c2fd412a05d5026549528e8b7e2cc026fcefacc2d6dec96f5c626d750063"
Sep 18 20:26:01.096620 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:26:01.096606    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7010c2fd412a05d5026549528e8b7e2cc026fcefacc2d6dec96f5c626d750063\": container with ID starting with 7010c2fd412a05d5026549528e8b7e2cc026fcefacc2d6dec96f5c626d750063 not found: ID does not exist" containerID="7010c2fd412a05d5026549528e8b7e2cc026fcefacc2d6dec96f5c626d750063"
Sep 18 20:26:01.096667 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:01.096624    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7010c2fd412a05d5026549528e8b7e2cc026fcefacc2d6dec96f5c626d750063"} err="failed to get container status \"7010c2fd412a05d5026549528e8b7e2cc026fcefacc2d6dec96f5c626d750063\": rpc error: code = NotFound desc = could not find container \"7010c2fd412a05d5026549528e8b7e2cc026fcefacc2d6dec96f5c626d750063\": container with ID starting with 7010c2fd412a05d5026549528e8b7e2cc026fcefacc2d6dec96f5c626d750063 not found: ID does not exist"
Sep 18 20:26:01.100720 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:01.100687    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g"]
Sep 18 20:26:01.104298 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:01.104268    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c5206e-openshift-default-7864cb5c8b-xkx5g"]
Sep 18 20:26:01.123030 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:01.122996    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7741-702e31"
Sep 18 20:26:01.401113 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:01.401083    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7741-702e31"]
Sep 18 20:26:01.403119 ip-10-0-140-178 kubenswrapper[2549]: W0918 20:26:01.403089    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd21052db_b539_41dc_ab2c_f77796d86070.slice/crio-8a79eb1c09684d7b033a15640bea4d729978d2036bbc809c2ee17370a95208dc WatchSource:0}: Error finding container 8a79eb1c09684d7b033a15640bea4d729978d2036bbc809c2ee17370a95208dc: Status 404 returned error can't find the container with id 8a79eb1c09684d7b033a15640bea4d729978d2036bbc809c2ee17370a95208dc
Sep 18 20:26:01.521961 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:01.521917    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a390c820-232c-4b8c-b4e4-711699f90d2b" path="/var/lib/kubelet/pods/a390c820-232c-4b8c-b4e4-711699f90d2b/volumes"
Sep 18 20:26:02.078622 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:02.078571    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7741-702e31" event={"ID":"d21052db-b539-41dc-ab2c-f77796d86070","Type":"ContainerStarted","Data":"5552a7c3d9f514a0cb29369d5d91c3a5023a04ad4b7acc62fff28f4f2613480a"}
Sep 18 20:26:02.079285 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:02.078636    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7741-702e31" event={"ID":"d21052db-b539-41dc-ab2c-f77796d86070","Type":"ContainerStarted","Data":"8a79eb1c09684d7b033a15640bea4d729978d2036bbc809c2ee17370a95208dc"}
Sep 18 20:26:02.079886 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:02.079860    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7741-76f57f" podUID="b0e692c7-53d5-4bfd-b9e4-12df9f17c1f5" containerName="test-curl-7741-76f57f" containerID="cri-o://027bf6c92e5fb131b8eb844415febfb53a43e3d1132254dd17629da3864fdbf5" gracePeriod=2
Sep 18 20:26:02.095839 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:02.095779    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7741-702e31" podStartSLOduration=2.095759038 podStartE2EDuration="2.095759038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:26:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:26:02.095048969 +0000 UTC m=+1747.048790136" watchObservedRunningTime="2026-09-18 20:26:02.095759038 +0000 UTC m=+1747.049500225"
Sep 18 20:26:02.688196 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:02.688157    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7741-702e31"]
Sep 18 20:26:02.693142 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:02.693106    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7741-702e31"]
Sep 18 20:26:03.086499 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:26:03.086461    2549 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7741-702e31" container="test-curl-7741-702e31"
Sep 18 20:26:04.092497 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:04.092459    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7741-702e31" podUID="d21052db-b539-41dc-ab2c-f77796d86070" containerName="test-curl-7741-702e31" containerID="cri-o://5552a7c3d9f514a0cb29369d5d91c3a5023a04ad4b7acc62fff28f4f2613480a" gracePeriod=2
Sep 18 20:26:04.279513 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:04.279485    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7741-76f57f"
Sep 18 20:26:04.365367 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:04.365266    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nlwkb\" (UniqueName: \"kubernetes.io/projected/b0e692c7-53d5-4bfd-b9e4-12df9f17c1f5-kube-api-access-nlwkb\") pod \"b0e692c7-53d5-4bfd-b9e4-12df9f17c1f5\" (UID: \"b0e692c7-53d5-4bfd-b9e4-12df9f17c1f5\") "
Sep 18 20:26:04.367578 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:04.367544    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0e692c7-53d5-4bfd-b9e4-12df9f17c1f5-kube-api-access-nlwkb" (OuterVolumeSpecName: "kube-api-access-nlwkb") pod "b0e692c7-53d5-4bfd-b9e4-12df9f17c1f5" (UID: "b0e692c7-53d5-4bfd-b9e4-12df9f17c1f5"). InnerVolumeSpecName "kube-api-access-nlwkb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:26:04.467139 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:04.467104    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlwkb\" (UniqueName: \"kubernetes.io/projected/b0e692c7-53d5-4bfd-b9e4-12df9f17c1f5-kube-api-access-nlwkb\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:26:04.865302 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:04.865257    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-40a1a5/sub-test-model-40a1a5-kserve-cbb55bd7-6zjdf"]
Sep 18 20:26:05.100061 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:05.100024    2549 generic.go:356] "Generic (PLEG): container finished" podID="b0e692c7-53d5-4bfd-b9e4-12df9f17c1f5" containerID="027bf6c92e5fb131b8eb844415febfb53a43e3d1132254dd17629da3864fdbf5" exitCode=137
Sep 18 20:26:05.100505 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:05.100069    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7741-76f57f"
Sep 18 20:26:05.100505 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:05.100124    2549 scope.go:117] "RemoveContainer" containerID="027bf6c92e5fb131b8eb844415febfb53a43e3d1132254dd17629da3864fdbf5"
Sep 18 20:26:05.110563 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:05.110540    2549 scope.go:117] "RemoveContainer" containerID="027bf6c92e5fb131b8eb844415febfb53a43e3d1132254dd17629da3864fdbf5"
Sep 18 20:26:05.110939 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:26:05.110919    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"027bf6c92e5fb131b8eb844415febfb53a43e3d1132254dd17629da3864fdbf5\": container with ID starting with 027bf6c92e5fb131b8eb844415febfb53a43e3d1132254dd17629da3864fdbf5 not found: ID does not exist" containerID="027bf6c92e5fb131b8eb844415febfb53a43e3d1132254dd17629da3864fdbf5"
Sep 18 20:26:05.110987 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:05.110950    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"027bf6c92e5fb131b8eb844415febfb53a43e3d1132254dd17629da3864fdbf5"} err="failed to get container status \"027bf6c92e5fb131b8eb844415febfb53a43e3d1132254dd17629da3864fdbf5\": rpc error: code = NotFound desc = could not find container \"027bf6c92e5fb131b8eb844415febfb53a43e3d1132254dd17629da3864fdbf5\": container with ID starting with 027bf6c92e5fb131b8eb844415febfb53a43e3d1132254dd17629da3864fdbf5 not found: ID does not exist"
Sep 18 20:26:05.112690 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:26:05.112662    2549 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7741-76f57f\" is forbidden: User \"system:node:ip-10-0-140-178.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-178.ec2.internal' and this object" podUID="b0e692c7-53d5-4bfd-b9e4-12df9f17c1f5" pod="opendatahub/test-curl-7741-76f57f"
Sep 18 20:26:05.519863 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:26:05.519817    2549 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7741-76f57f\" is forbidden: User \"system:node:ip-10-0-140-178.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-178.ec2.internal' and this object" podUID="b0e692c7-53d5-4bfd-b9e4-12df9f17c1f5" pod="opendatahub/test-curl-7741-76f57f"
Sep 18 20:26:05.521364 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:05.521338    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0e692c7-53d5-4bfd-b9e4-12df9f17c1f5" path="/var/lib/kubelet/pods/b0e692c7-53d5-4bfd-b9e4-12df9f17c1f5/volumes"
Sep 18 20:26:06.294447 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:06.294412    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7741-702e31"
Sep 18 20:26:06.387154 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:06.387055    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wxm77\" (UniqueName: \"kubernetes.io/projected/d21052db-b539-41dc-ab2c-f77796d86070-kube-api-access-wxm77\") pod \"d21052db-b539-41dc-ab2c-f77796d86070\" (UID: \"d21052db-b539-41dc-ab2c-f77796d86070\") "
Sep 18 20:26:06.389276 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:06.389244    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d21052db-b539-41dc-ab2c-f77796d86070-kube-api-access-wxm77" (OuterVolumeSpecName: "kube-api-access-wxm77") pod "d21052db-b539-41dc-ab2c-f77796d86070" (UID: "d21052db-b539-41dc-ab2c-f77796d86070"). InnerVolumeSpecName "kube-api-access-wxm77". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:26:06.488209 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:06.488159    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxm77\" (UniqueName: \"kubernetes.io/projected/d21052db-b539-41dc-ab2c-f77796d86070-kube-api-access-wxm77\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:26:07.116019 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:07.115983    2549 generic.go:356] "Generic (PLEG): container finished" podID="d21052db-b539-41dc-ab2c-f77796d86070" containerID="5552a7c3d9f514a0cb29369d5d91c3a5023a04ad4b7acc62fff28f4f2613480a" exitCode=137
Sep 18 20:26:07.116218 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:07.116028    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7741-702e31"
Sep 18 20:26:07.116218 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:07.116083    2549 scope.go:117] "RemoveContainer" containerID="5552a7c3d9f514a0cb29369d5d91c3a5023a04ad4b7acc62fff28f4f2613480a"
Sep 18 20:26:07.126301 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:07.126280    2549 scope.go:117] "RemoveContainer" containerID="5552a7c3d9f514a0cb29369d5d91c3a5023a04ad4b7acc62fff28f4f2613480a"
Sep 18 20:26:07.126657 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:26:07.126622    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5552a7c3d9f514a0cb29369d5d91c3a5023a04ad4b7acc62fff28f4f2613480a\": container with ID starting with 5552a7c3d9f514a0cb29369d5d91c3a5023a04ad4b7acc62fff28f4f2613480a not found: ID does not exist" containerID="5552a7c3d9f514a0cb29369d5d91c3a5023a04ad4b7acc62fff28f4f2613480a"
Sep 18 20:26:07.126763 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:07.126670    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5552a7c3d9f514a0cb29369d5d91c3a5023a04ad4b7acc62fff28f4f2613480a"} err="failed to get container status \"5552a7c3d9f514a0cb29369d5d91c3a5023a04ad4b7acc62fff28f4f2613480a\": rpc error: code = NotFound desc = could not find container \"5552a7c3d9f514a0cb29369d5d91c3a5023a04ad4b7acc62fff28f4f2613480a\": container with ID starting with 5552a7c3d9f514a0cb29369d5d91c3a5023a04ad4b7acc62fff28f4f2613480a not found: ID does not exist"
Sep 18 20:26:07.127846 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:26:07.127821    2549 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7741-702e31\" is forbidden: User \"system:node:ip-10-0-140-178.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-178.ec2.internal' and this object" podUID="d21052db-b539-41dc-ab2c-f77796d86070" pod="opendatahub/test-curl-7741-702e31"
Sep 18 20:26:07.521847 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:07.521747    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d21052db-b539-41dc-ab2c-f77796d86070" path="/var/lib/kubelet/pods/d21052db-b539-41dc-ab2c-f77796d86070/volumes"
Sep 18 20:26:13.412014 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:13.411963    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:26:19.885423 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:19.885357    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-40a1a5/sub-test-model-40a1a5-kserve-cbb55bd7-6zjdf" podUID="2ed6c4d4-2a7f-40d1-8cbb-7784adf5b134" containerName="main" containerID="cri-o://5033b1b96a20814eb570cce1eaa19a960091ccf30a5bd96a91494084a6057850" gracePeriod=45
Sep 18 20:26:20.159107 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:20.159075    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-40a1a5/sub-test-model-40a1a5-kserve-cbb55bd7-6zjdf"
Sep 18 20:26:20.197406 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:20.197360    2549 generic.go:356] "Generic (PLEG): container finished" podID="2ed6c4d4-2a7f-40d1-8cbb-7784adf5b134" containerID="5033b1b96a20814eb570cce1eaa19a960091ccf30a5bd96a91494084a6057850" exitCode=0
Sep 18 20:26:20.197598 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:20.197433    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-40a1a5/sub-test-model-40a1a5-kserve-cbb55bd7-6zjdf" event={"ID":"2ed6c4d4-2a7f-40d1-8cbb-7784adf5b134","Type":"ContainerDied","Data":"5033b1b96a20814eb570cce1eaa19a960091ccf30a5bd96a91494084a6057850"}
Sep 18 20:26:20.197598 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:20.197474    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-40a1a5/sub-test-model-40a1a5-kserve-cbb55bd7-6zjdf"
Sep 18 20:26:20.197598 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:20.197485    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-40a1a5/sub-test-model-40a1a5-kserve-cbb55bd7-6zjdf" event={"ID":"2ed6c4d4-2a7f-40d1-8cbb-7784adf5b134","Type":"ContainerDied","Data":"c41bdd4f383e3739012f12c6bd2625be762584f0dd993257acc0ffe3ef28db0b"}
Sep 18 20:26:20.197598 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:20.197509    2549 scope.go:117] "RemoveContainer" containerID="5033b1b96a20814eb570cce1eaa19a960091ccf30a5bd96a91494084a6057850"
Sep 18 20:26:20.207345 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:20.207319    2549 scope.go:117] "RemoveContainer" containerID="5033b1b96a20814eb570cce1eaa19a960091ccf30a5bd96a91494084a6057850"
Sep 18 20:26:20.207729 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:26:20.207708    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5033b1b96a20814eb570cce1eaa19a960091ccf30a5bd96a91494084a6057850\": container with ID starting with 5033b1b96a20814eb570cce1eaa19a960091ccf30a5bd96a91494084a6057850 not found: ID does not exist" containerID="5033b1b96a20814eb570cce1eaa19a960091ccf30a5bd96a91494084a6057850"
Sep 18 20:26:20.207819 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:20.207739    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5033b1b96a20814eb570cce1eaa19a960091ccf30a5bd96a91494084a6057850"} err="failed to get container status \"5033b1b96a20814eb570cce1eaa19a960091ccf30a5bd96a91494084a6057850\": rpc error: code = NotFound desc = could not find container \"5033b1b96a20814eb570cce1eaa19a960091ccf30a5bd96a91494084a6057850\": container with ID starting with 5033b1b96a20814eb570cce1eaa19a960091ccf30a5bd96a91494084a6057850 not found: ID does not exist"
Sep 18 20:26:20.325570 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:20.325528    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ed6c4d4-2a7f-40d1-8cbb-7784adf5b134-dshm\") pod \"2ed6c4d4-2a7f-40d1-8cbb-7784adf5b134\" (UID: \"2ed6c4d4-2a7f-40d1-8cbb-7784adf5b134\") "
Sep 18 20:26:20.325759 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:20.325657    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ed6c4d4-2a7f-40d1-8cbb-7784adf5b134-model-cache\") pod \"2ed6c4d4-2a7f-40d1-8cbb-7784adf5b134\" (UID: \"2ed6c4d4-2a7f-40d1-8cbb-7784adf5b134\") "
Sep 18 20:26:20.325759 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:20.325689    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ed6c4d4-2a7f-40d1-8cbb-7784adf5b134-tls-certs\") pod \"2ed6c4d4-2a7f-40d1-8cbb-7784adf5b134\" (UID: \"2ed6c4d4-2a7f-40d1-8cbb-7784adf5b134\") "
Sep 18 20:26:20.325759 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:20.325752    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vg9n2\" (UniqueName: \"kubernetes.io/projected/2ed6c4d4-2a7f-40d1-8cbb-7784adf5b134-kube-api-access-vg9n2\") pod \"2ed6c4d4-2a7f-40d1-8cbb-7784adf5b134\" (UID: \"2ed6c4d4-2a7f-40d1-8cbb-7784adf5b134\") "
Sep 18 20:26:20.325885 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:20.325780    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ed6c4d4-2a7f-40d1-8cbb-7784adf5b134-tmp-dir\") pod \"2ed6c4d4-2a7f-40d1-8cbb-7784adf5b134\" (UID: \"2ed6c4d4-2a7f-40d1-8cbb-7784adf5b134\") "
Sep 18 20:26:20.325885 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:20.325813    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ed6c4d4-2a7f-40d1-8cbb-7784adf5b134-home\") pod \"2ed6c4d4-2a7f-40d1-8cbb-7784adf5b134\" (UID: \"2ed6c4d4-2a7f-40d1-8cbb-7784adf5b134\") "
Sep 18 20:26:20.326040 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:20.325965    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ed6c4d4-2a7f-40d1-8cbb-7784adf5b134-model-cache" (OuterVolumeSpecName: "model-cache") pod "2ed6c4d4-2a7f-40d1-8cbb-7784adf5b134" (UID: "2ed6c4d4-2a7f-40d1-8cbb-7784adf5b134"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:26:20.326199 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:20.326088    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ed6c4d4-2a7f-40d1-8cbb-7784adf5b134-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2ed6c4d4-2a7f-40d1-8cbb-7784adf5b134" (UID: "2ed6c4d4-2a7f-40d1-8cbb-7784adf5b134"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:26:20.326199 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:20.326157    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ed6c4d4-2a7f-40d1-8cbb-7784adf5b134-home" (OuterVolumeSpecName: "home") pod "2ed6c4d4-2a7f-40d1-8cbb-7784adf5b134" (UID: "2ed6c4d4-2a7f-40d1-8cbb-7784adf5b134"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:26:20.327853 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:20.327828    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ed6c4d4-2a7f-40d1-8cbb-7784adf5b134-dshm" (OuterVolumeSpecName: "dshm") pod "2ed6c4d4-2a7f-40d1-8cbb-7784adf5b134" (UID: "2ed6c4d4-2a7f-40d1-8cbb-7784adf5b134"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:26:20.327970 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:20.327883    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ed6c4d4-2a7f-40d1-8cbb-7784adf5b134-kube-api-access-vg9n2" (OuterVolumeSpecName: "kube-api-access-vg9n2") pod "2ed6c4d4-2a7f-40d1-8cbb-7784adf5b134" (UID: "2ed6c4d4-2a7f-40d1-8cbb-7784adf5b134"). InnerVolumeSpecName "kube-api-access-vg9n2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:26:20.328010 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:20.327975    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ed6c4d4-2a7f-40d1-8cbb-7784adf5b134-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2ed6c4d4-2a7f-40d1-8cbb-7784adf5b134" (UID: "2ed6c4d4-2a7f-40d1-8cbb-7784adf5b134"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:26:20.426928 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:20.426822    2549 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ed6c4d4-2a7f-40d1-8cbb-7784adf5b134-tls-certs\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:26:20.426928 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:20.426862    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vg9n2\" (UniqueName: \"kubernetes.io/projected/2ed6c4d4-2a7f-40d1-8cbb-7784adf5b134-kube-api-access-vg9n2\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:26:20.426928 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:20.426896    2549 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ed6c4d4-2a7f-40d1-8cbb-7784adf5b134-tmp-dir\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:26:20.426928 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:20.426908    2549 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ed6c4d4-2a7f-40d1-8cbb-7784adf5b134-home\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:26:20.426928 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:20.426917    2549 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ed6c4d4-2a7f-40d1-8cbb-7784adf5b134-dshm\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:26:20.426928 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:20.426925    2549 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ed6c4d4-2a7f-40d1-8cbb-7784adf5b134-model-cache\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:26:20.524607 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:20.524568    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-40a1a5/sub-test-model-40a1a5-kserve-cbb55bd7-6zjdf"]
Sep 18 20:26:20.530954 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:20.530921    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-40a1a5/sub-test-model-40a1a5-kserve-cbb55bd7-6zjdf"]
Sep 18 20:26:21.522783 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:21.522740    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ed6c4d4-2a7f-40d1-8cbb-7784adf5b134" path="/var/lib/kubelet/pods/2ed6c4d4-2a7f-40d1-8cbb-7784adf5b134/volumes"
Sep 18 20:26:22.015549 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:22.015509    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:26:50.639902 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:50.639857    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-81f19d/rate-test-model-81f19d-kserve-6c5b976b67-njlhx"]
Sep 18 20:26:50.640405 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:50.640271    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ed6c4d4-2a7f-40d1-8cbb-7784adf5b134" containerName="main"
Sep 18 20:26:50.640405 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:50.640283    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ed6c4d4-2a7f-40d1-8cbb-7784adf5b134" containerName="main"
Sep 18 20:26:50.640405 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:50.640290    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a390c820-232c-4b8c-b4e4-711699f90d2b" containerName="istio-proxy"
Sep 18 20:26:50.640405 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:50.640296    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="a390c820-232c-4b8c-b4e4-711699f90d2b" containerName="istio-proxy"
Sep 18 20:26:50.640405 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:50.640306    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d21052db-b539-41dc-ab2c-f77796d86070" containerName="test-curl-7741-702e31"
Sep 18 20:26:50.640405 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:50.640312    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="d21052db-b539-41dc-ab2c-f77796d86070" containerName="test-curl-7741-702e31"
Sep 18 20:26:50.640405 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:50.640399    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ed6c4d4-2a7f-40d1-8cbb-7784adf5b134" containerName="main"
Sep 18 20:26:50.640405 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:50.640410    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="d21052db-b539-41dc-ab2c-f77796d86070" containerName="test-curl-7741-702e31"
Sep 18 20:26:50.643613 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:50.643590    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-81f19d/rate-test-model-81f19d-kserve-6c5b976b67-njlhx"
Sep 18 20:26:50.646984 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:50.646954    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-81f19d\"/\"kube-root-ca.crt\""
Sep 18 20:26:50.647120 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:50.646952    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-81f19d\"/\"rate-test-model-81f19d-kserve-self-signed-certs\""
Sep 18 20:26:50.648309 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:50.648282    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-81f19d\"/\"openshift-service-ca.crt\""
Sep 18 20:26:50.648426 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:50.648411    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-81f19d\"/\"default-dockercfg-kwxkf\""
Sep 18 20:26:50.655138 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:50.655107    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-81f19d/rate-test-model-81f19d-kserve-6c5b976b67-njlhx"]
Sep 18 20:26:50.700027 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:50.699990    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fbc6f93-c02b-4967-af97-57e896e8a96c-tls-certs\") pod \"rate-test-model-81f19d-kserve-6c5b976b67-njlhx\" (UID: \"6fbc6f93-c02b-4967-af97-57e896e8a96c\") " pod="ai-tenant-e2e-shared-b-w4-81f19d/rate-test-model-81f19d-kserve-6c5b976b67-njlhx"
Sep 18 20:26:50.700226 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:50.700040    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fbc6f93-c02b-4967-af97-57e896e8a96c-model-cache\") pod \"rate-test-model-81f19d-kserve-6c5b976b67-njlhx\" (UID: \"6fbc6f93-c02b-4967-af97-57e896e8a96c\") " pod="ai-tenant-e2e-shared-b-w4-81f19d/rate-test-model-81f19d-kserve-6c5b976b67-njlhx"
Sep 18 20:26:50.700226 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:50.700123    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fbc6f93-c02b-4967-af97-57e896e8a96c-dshm\") pod \"rate-test-model-81f19d-kserve-6c5b976b67-njlhx\" (UID: \"6fbc6f93-c02b-4967-af97-57e896e8a96c\") " pod="ai-tenant-e2e-shared-b-w4-81f19d/rate-test-model-81f19d-kserve-6c5b976b67-njlhx"
Sep 18 20:26:50.700226 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:50.700204    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fbc6f93-c02b-4967-af97-57e896e8a96c-tmp-dir\") pod \"rate-test-model-81f19d-kserve-6c5b976b67-njlhx\" (UID: \"6fbc6f93-c02b-4967-af97-57e896e8a96c\") " pod="ai-tenant-e2e-shared-b-w4-81f19d/rate-test-model-81f19d-kserve-6c5b976b67-njlhx"
Sep 18 20:26:50.700347 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:50.700259    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6fbc6f93-c02b-4967-af97-57e896e8a96c-home\") pod \"rate-test-model-81f19d-kserve-6c5b976b67-njlhx\" (UID: \"6fbc6f93-c02b-4967-af97-57e896e8a96c\") " pod="ai-tenant-e2e-shared-b-w4-81f19d/rate-test-model-81f19d-kserve-6c5b976b67-njlhx"
Sep 18 20:26:50.700347 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:50.700310    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqvxm\" (UniqueName: \"kubernetes.io/projected/6fbc6f93-c02b-4967-af97-57e896e8a96c-kube-api-access-vqvxm\") pod \"rate-test-model-81f19d-kserve-6c5b976b67-njlhx\" (UID: \"6fbc6f93-c02b-4967-af97-57e896e8a96c\") " pod="ai-tenant-e2e-shared-b-w4-81f19d/rate-test-model-81f19d-kserve-6c5b976b67-njlhx"
Sep 18 20:26:50.801225 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:50.801168    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fbc6f93-c02b-4967-af97-57e896e8a96c-tls-certs\") pod \"rate-test-model-81f19d-kserve-6c5b976b67-njlhx\" (UID: \"6fbc6f93-c02b-4967-af97-57e896e8a96c\") " pod="ai-tenant-e2e-shared-b-w4-81f19d/rate-test-model-81f19d-kserve-6c5b976b67-njlhx"
Sep 18 20:26:50.801776 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:50.801684    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fbc6f93-c02b-4967-af97-57e896e8a96c-model-cache\") pod \"rate-test-model-81f19d-kserve-6c5b976b67-njlhx\" (UID: \"6fbc6f93-c02b-4967-af97-57e896e8a96c\") " pod="ai-tenant-e2e-shared-b-w4-81f19d/rate-test-model-81f19d-kserve-6c5b976b67-njlhx"
Sep 18 20:26:50.801776 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:50.801744    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fbc6f93-c02b-4967-af97-57e896e8a96c-dshm\") pod \"rate-test-model-81f19d-kserve-6c5b976b67-njlhx\" (UID: \"6fbc6f93-c02b-4967-af97-57e896e8a96c\") " pod="ai-tenant-e2e-shared-b-w4-81f19d/rate-test-model-81f19d-kserve-6c5b976b67-njlhx"
Sep 18 20:26:50.801972 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:50.801814    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fbc6f93-c02b-4967-af97-57e896e8a96c-tmp-dir\") pod \"rate-test-model-81f19d-kserve-6c5b976b67-njlhx\" (UID: \"6fbc6f93-c02b-4967-af97-57e896e8a96c\") " pod="ai-tenant-e2e-shared-b-w4-81f19d/rate-test-model-81f19d-kserve-6c5b976b67-njlhx"
Sep 18 20:26:50.801972 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:50.801841    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6fbc6f93-c02b-4967-af97-57e896e8a96c-home\") pod \"rate-test-model-81f19d-kserve-6c5b976b67-njlhx\" (UID: \"6fbc6f93-c02b-4967-af97-57e896e8a96c\") " pod="ai-tenant-e2e-shared-b-w4-81f19d/rate-test-model-81f19d-kserve-6c5b976b67-njlhx"
Sep 18 20:26:50.801972 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:50.801868    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqvxm\" (UniqueName: \"kubernetes.io/projected/6fbc6f93-c02b-4967-af97-57e896e8a96c-kube-api-access-vqvxm\") pod \"rate-test-model-81f19d-kserve-6c5b976b67-njlhx\" (UID: \"6fbc6f93-c02b-4967-af97-57e896e8a96c\") " pod="ai-tenant-e2e-shared-b-w4-81f19d/rate-test-model-81f19d-kserve-6c5b976b67-njlhx"
Sep 18 20:26:50.802126 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:50.802080    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fbc6f93-c02b-4967-af97-57e896e8a96c-model-cache\") pod \"rate-test-model-81f19d-kserve-6c5b976b67-njlhx\" (UID: \"6fbc6f93-c02b-4967-af97-57e896e8a96c\") " pod="ai-tenant-e2e-shared-b-w4-81f19d/rate-test-model-81f19d-kserve-6c5b976b67-njlhx"
Sep 18 20:26:50.802456 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:50.802344    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fbc6f93-c02b-4967-af97-57e896e8a96c-tmp-dir\") pod \"rate-test-model-81f19d-kserve-6c5b976b67-njlhx\" (UID: \"6fbc6f93-c02b-4967-af97-57e896e8a96c\") " pod="ai-tenant-e2e-shared-b-w4-81f19d/rate-test-model-81f19d-kserve-6c5b976b67-njlhx"
Sep 18 20:26:50.802839 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:50.802601    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6fbc6f93-c02b-4967-af97-57e896e8a96c-home\") pod \"rate-test-model-81f19d-kserve-6c5b976b67-njlhx\" (UID: \"6fbc6f93-c02b-4967-af97-57e896e8a96c\") " pod="ai-tenant-e2e-shared-b-w4-81f19d/rate-test-model-81f19d-kserve-6c5b976b67-njlhx"
Sep 18 20:26:50.806079 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:50.806019    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fbc6f93-c02b-4967-af97-57e896e8a96c-tls-certs\") pod \"rate-test-model-81f19d-kserve-6c5b976b67-njlhx\" (UID: \"6fbc6f93-c02b-4967-af97-57e896e8a96c\") " pod="ai-tenant-e2e-shared-b-w4-81f19d/rate-test-model-81f19d-kserve-6c5b976b67-njlhx"
Sep 18 20:26:50.806220 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:50.806115    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fbc6f93-c02b-4967-af97-57e896e8a96c-dshm\") pod \"rate-test-model-81f19d-kserve-6c5b976b67-njlhx\" (UID: \"6fbc6f93-c02b-4967-af97-57e896e8a96c\") " pod="ai-tenant-e2e-shared-b-w4-81f19d/rate-test-model-81f19d-kserve-6c5b976b67-njlhx"
Sep 18 20:26:50.811807 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:50.811776    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqvxm\" (UniqueName: \"kubernetes.io/projected/6fbc6f93-c02b-4967-af97-57e896e8a96c-kube-api-access-vqvxm\") pod \"rate-test-model-81f19d-kserve-6c5b976b67-njlhx\" (UID: \"6fbc6f93-c02b-4967-af97-57e896e8a96c\") " pod="ai-tenant-e2e-shared-b-w4-81f19d/rate-test-model-81f19d-kserve-6c5b976b67-njlhx"
Sep 18 20:26:50.954653 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:50.954555    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-81f19d/rate-test-model-81f19d-kserve-6c5b976b67-njlhx"
Sep 18 20:26:51.241680 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:51.241644    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-81f19d/rate-test-model-81f19d-kserve-6c5b976b67-njlhx"]
Sep 18 20:26:51.243474 ip-10-0-140-178 kubenswrapper[2549]: W0918 20:26:51.243442    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6fbc6f93_c02b_4967_af97_57e896e8a96c.slice/crio-dbbb6fa0fef2c6a7794b31e76f4a76a812bbc4887c9ddbbf24a6f36d08a17140 WatchSource:0}: Error finding container dbbb6fa0fef2c6a7794b31e76f4a76a812bbc4887c9ddbbf24a6f36d08a17140: Status 404 returned error can't find the container with id dbbb6fa0fef2c6a7794b31e76f4a76a812bbc4887c9ddbbf24a6f36d08a17140
Sep 18 20:26:51.399171 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:51.399126    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-81f19d/rate-test-model-81f19d-kserve-6c5b976b67-njlhx" event={"ID":"6fbc6f93-c02b-4967-af97-57e896e8a96c","Type":"ContainerStarted","Data":"3ece96f1d7b152f26f0acf02bd1f82e0c2d6e08e30009a71938918bc85e2e5fc"}
Sep 18 20:26:51.399171 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:51.399180    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-81f19d/rate-test-model-81f19d-kserve-6c5b976b67-njlhx" event={"ID":"6fbc6f93-c02b-4967-af97-57e896e8a96c","Type":"ContainerStarted","Data":"dbbb6fa0fef2c6a7794b31e76f4a76a812bbc4887c9ddbbf24a6f36d08a17140"}
Sep 18 20:26:51.427809 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:51.427749    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-81f19d/rate-test-model-81f19d-kserve-6c5b976b67-njlhx" podStartSLOduration=1.4277295460000001 podStartE2EDuration="1.427729546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:26:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:26:51.426978096 +0000 UTC m=+1796.380719266" watchObservedRunningTime="2026-09-18 20:26:51.427729546 +0000 UTC m=+1796.381470730"
Sep 18 20:26:54.615322 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:54.615280    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:26:55.628721 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:55.628684    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-scn7p_a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48/ovn-acl-logging/0.log"
Sep 18 20:26:55.629633 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:26:55.629605    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-scn7p_a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48/ovn-acl-logging/0.log"
Sep 18 20:27:00.955376 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:00.955328    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-81f19d/rate-test-model-81f19d-kserve-6c5b976b67-njlhx"
Sep 18 20:27:00.955917 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:00.955431    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-81f19d/rate-test-model-81f19d-kserve-6c5b976b67-njlhx"
Sep 18 20:27:00.968183 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:00.968153    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-81f19d/rate-test-model-81f19d-kserve-6c5b976b67-njlhx"
Sep 18 20:27:01.476111 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:01.476072    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-81f19d/rate-test-model-81f19d-kserve-6c5b976b67-njlhx"
Sep 18 20:27:09.155286 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:09.155231    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:27:20.114867 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:20.114823    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:27:28.193635 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:28.193599    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-81f19d/rate-test-model-81f19d-kserve-6c5b976b67-njlhx"]
Sep 18 20:27:28.958174 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:28.958137    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd"]
Sep 18 20:27:28.961894 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:28.961861    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd"
Sep 18 20:27:28.967102 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:28.967071    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-de7ea3-openshift-default-dockercfg-fnpwx\""
Sep 18 20:27:28.991341 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:28.991296    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd"]
Sep 18 20:27:29.091908 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:29.091861    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd-istio-data\") pod \"e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd\" (UID: \"e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd\") " pod="openshift-ingress/e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd"
Sep 18 20:27:29.092119 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:29.091967    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd-istio-podinfo\") pod \"e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd\" (UID: \"e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd\") " pod="openshift-ingress/e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd"
Sep 18 20:27:29.092119 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:29.092006    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd-istiod-ca-cert\") pod \"e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd\" (UID: \"e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd\") " pod="openshift-ingress/e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd"
Sep 18 20:27:29.092119 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:29.092028    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd-istio-token\") pod \"e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd\" (UID: \"e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd\") " pod="openshift-ingress/e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd"
Sep 18 20:27:29.092119 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:29.092047    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhwrn\" (UniqueName: \"kubernetes.io/projected/e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd-kube-api-access-vhwrn\") pod \"e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd\" (UID: \"e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd\") " pod="openshift-ingress/e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd"
Sep 18 20:27:29.092302 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:29.092136    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd-istio-envoy\") pod \"e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd\" (UID: \"e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd\") " pod="openshift-ingress/e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd"
Sep 18 20:27:29.092302 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:29.092179    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd-workload-certs\") pod \"e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd\" (UID: \"e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd\") " pod="openshift-ingress/e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd"
Sep 18 20:27:29.092302 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:29.092262    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd-credential-socket\") pod \"e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd\" (UID: \"e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd\") " pod="openshift-ingress/e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd"
Sep 18 20:27:29.092455 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:29.092302    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd-workload-socket\") pod \"e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd\" (UID: \"e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd\") " pod="openshift-ingress/e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd"
Sep 18 20:27:29.192966 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:29.192924    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd-istio-envoy\") pod \"e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd\" (UID: \"e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd\") " pod="openshift-ingress/e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd"
Sep 18 20:27:29.192966 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:29.192975    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd-workload-certs\") pod \"e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd\" (UID: \"e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd\") " pod="openshift-ingress/e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd"
Sep 18 20:27:29.193254 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:29.193035    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd-credential-socket\") pod \"e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd\" (UID: \"e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd\") " pod="openshift-ingress/e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd"
Sep 18 20:27:29.193254 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:29.193081    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd-workload-socket\") pod \"e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd\" (UID: \"e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd\") " pod="openshift-ingress/e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd"
Sep 18 20:27:29.193254 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:29.193104    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd-istio-data\") pod \"e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd\" (UID: \"e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd\") " pod="openshift-ingress/e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd"
Sep 18 20:27:29.193254 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:29.193161    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd-istio-podinfo\") pod \"e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd\" (UID: \"e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd\") " pod="openshift-ingress/e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd"
Sep 18 20:27:29.193254 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:29.193191    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd-istiod-ca-cert\") pod \"e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd\" (UID: \"e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd\") " pod="openshift-ingress/e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd"
Sep 18 20:27:29.193254 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:29.193216    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd-istio-token\") pod \"e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd\" (UID: \"e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd\") " pod="openshift-ingress/e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd"
Sep 18 20:27:29.193254 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:29.193238    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhwrn\" (UniqueName: \"kubernetes.io/projected/e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd-kube-api-access-vhwrn\") pod \"e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd\" (UID: \"e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd\") " pod="openshift-ingress/e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd"
Sep 18 20:27:29.193671 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:29.193538    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd-workload-certs\") pod \"e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd\" (UID: \"e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd\") " pod="openshift-ingress/e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd"
Sep 18 20:27:29.193671 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:29.193592    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd-credential-socket\") pod \"e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd\" (UID: \"e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd\") " pod="openshift-ingress/e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd"
Sep 18 20:27:29.193671 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:29.193661    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd-istio-data\") pod \"e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd\" (UID: \"e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd\") " pod="openshift-ingress/e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd"
Sep 18 20:27:29.194215 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:29.193662    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd-workload-socket\") pod \"e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd\" (UID: \"e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd\") " pod="openshift-ingress/e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd"
Sep 18 20:27:29.194215 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:29.194130    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd-istiod-ca-cert\") pod \"e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd\" (UID: \"e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd\") " pod="openshift-ingress/e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd"
Sep 18 20:27:29.195672 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:29.195648    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd-istio-envoy\") pod \"e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd\" (UID: \"e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd\") " pod="openshift-ingress/e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd"
Sep 18 20:27:29.196181 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:29.196164    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd-istio-podinfo\") pod \"e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd\" (UID: \"e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd\") " pod="openshift-ingress/e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd"
Sep 18 20:27:29.203458 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:29.203429    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd-istio-token\") pod \"e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd\" (UID: \"e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd\") " pod="openshift-ingress/e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd"
Sep 18 20:27:29.205609 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:29.205578    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhwrn\" (UniqueName: \"kubernetes.io/projected/e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd-kube-api-access-vhwrn\") pod \"e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd\" (UID: \"e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd\") " pod="openshift-ingress/e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd"
Sep 18 20:27:29.272691 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:29.272589    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd"
Sep 18 20:27:29.583847 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:29.583810    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd"]
Sep 18 20:27:29.587481 ip-10-0-140-178 kubenswrapper[2549]: W0918 20:27:29.587448    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0cc6625_c6f1_4310_b1ac_3d0eaeae32cd.slice/crio-5f202ff8dc7b7b147406d9ee2ffc1dda192d8b5788aba837d286c4eb1d78bd8c WatchSource:0}: Error finding container 5f202ff8dc7b7b147406d9ee2ffc1dda192d8b5788aba837d286c4eb1d78bd8c: Status 404 returned error can't find the container with id 5f202ff8dc7b7b147406d9ee2ffc1dda192d8b5788aba837d286c4eb1d78bd8c
Sep 18 20:27:29.590222 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:29.590184    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:27:29.590315 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:29.590262    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:27:29.590315 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:29.590292    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:27:29.636974 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:29.636940    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd" event={"ID":"e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd","Type":"ContainerStarted","Data":"5f202ff8dc7b7b147406d9ee2ffc1dda192d8b5788aba837d286c4eb1d78bd8c"}
Sep 18 20:27:30.644148 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:30.644107    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd" event={"ID":"e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd","Type":"ContainerStarted","Data":"0cff3f80ee7b26e4a5d0965f6a21c9b02ad67077c333be0dcad7edae8d927287"}
Sep 18 20:27:30.671927 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:30.671871    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd" podStartSLOduration=2.671847006 podStartE2EDuration="2.671847006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:27:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:27:30.66897173 +0000 UTC m=+1835.622712922" watchObservedRunningTime="2026-09-18 20:27:30.671847006 +0000 UTC m=+1835.625588175"
Sep 18 20:27:31.273157 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:31.273117    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd"
Sep 18 20:27:31.278984 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:31.278951    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd"
Sep 18 20:27:31.651891 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:31.651858    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd"
Sep 18 20:27:31.653134 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:31.653104    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd"
Sep 18 20:27:33.715693 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:33.715648    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:27:43.213517 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:43.213470    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-81f19d/rate-test-model-81f19d-kserve-6c5b976b67-njlhx" podUID="6fbc6f93-c02b-4967-af97-57e896e8a96c" containerName="main" containerID="cri-o://3ece96f1d7b152f26f0acf02bd1f82e0c2d6e08e30009a71938918bc85e2e5fc" gracePeriod=45
Sep 18 20:27:43.524284 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:43.524254    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-81f19d/rate-test-model-81f19d-kserve-6c5b976b67-njlhx"
Sep 18 20:27:43.644893 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:43.644856    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6fbc6f93-c02b-4967-af97-57e896e8a96c-home\") pod \"6fbc6f93-c02b-4967-af97-57e896e8a96c\" (UID: \"6fbc6f93-c02b-4967-af97-57e896e8a96c\") "
Sep 18 20:27:43.645113 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:43.644927    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fbc6f93-c02b-4967-af97-57e896e8a96c-model-cache\") pod \"6fbc6f93-c02b-4967-af97-57e896e8a96c\" (UID: \"6fbc6f93-c02b-4967-af97-57e896e8a96c\") "
Sep 18 20:27:43.645113 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:43.644958    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fbc6f93-c02b-4967-af97-57e896e8a96c-dshm\") pod \"6fbc6f93-c02b-4967-af97-57e896e8a96c\" (UID: \"6fbc6f93-c02b-4967-af97-57e896e8a96c\") "
Sep 18 20:27:43.645113 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:43.645008    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fbc6f93-c02b-4967-af97-57e896e8a96c-tls-certs\") pod \"6fbc6f93-c02b-4967-af97-57e896e8a96c\" (UID: \"6fbc6f93-c02b-4967-af97-57e896e8a96c\") "
Sep 18 20:27:43.645113 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:43.645038    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vqvxm\" (UniqueName: \"kubernetes.io/projected/6fbc6f93-c02b-4967-af97-57e896e8a96c-kube-api-access-vqvxm\") pod \"6fbc6f93-c02b-4967-af97-57e896e8a96c\" (UID: \"6fbc6f93-c02b-4967-af97-57e896e8a96c\") "
Sep 18 20:27:43.645113 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:43.645100    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fbc6f93-c02b-4967-af97-57e896e8a96c-tmp-dir\") pod \"6fbc6f93-c02b-4967-af97-57e896e8a96c\" (UID: \"6fbc6f93-c02b-4967-af97-57e896e8a96c\") "
Sep 18 20:27:43.645439 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:43.645215    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fbc6f93-c02b-4967-af97-57e896e8a96c-home" (OuterVolumeSpecName: "home") pod "6fbc6f93-c02b-4967-af97-57e896e8a96c" (UID: "6fbc6f93-c02b-4967-af97-57e896e8a96c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:27:43.645439 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:43.645285    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fbc6f93-c02b-4967-af97-57e896e8a96c-model-cache" (OuterVolumeSpecName: "model-cache") pod "6fbc6f93-c02b-4967-af97-57e896e8a96c" (UID: "6fbc6f93-c02b-4967-af97-57e896e8a96c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:27:43.645567 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:43.645548    2549 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6fbc6f93-c02b-4967-af97-57e896e8a96c-home\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:27:43.645619 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:43.645580    2549 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fbc6f93-c02b-4967-af97-57e896e8a96c-model-cache\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:27:43.658175 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:43.653516    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fbc6f93-c02b-4967-af97-57e896e8a96c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6fbc6f93-c02b-4967-af97-57e896e8a96c" (UID: "6fbc6f93-c02b-4967-af97-57e896e8a96c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:27:43.658175 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:43.657891    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fbc6f93-c02b-4967-af97-57e896e8a96c-dshm" (OuterVolumeSpecName: "dshm") pod "6fbc6f93-c02b-4967-af97-57e896e8a96c" (UID: "6fbc6f93-c02b-4967-af97-57e896e8a96c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:27:43.662563 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:43.662512    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fbc6f93-c02b-4967-af97-57e896e8a96c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6fbc6f93-c02b-4967-af97-57e896e8a96c" (UID: "6fbc6f93-c02b-4967-af97-57e896e8a96c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:27:43.676624 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:43.676573    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fbc6f93-c02b-4967-af97-57e896e8a96c-kube-api-access-vqvxm" (OuterVolumeSpecName: "kube-api-access-vqvxm") pod "6fbc6f93-c02b-4967-af97-57e896e8a96c" (UID: "6fbc6f93-c02b-4967-af97-57e896e8a96c"). InnerVolumeSpecName "kube-api-access-vqvxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:27:43.730465 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:43.730430    2549 generic.go:356] "Generic (PLEG): container finished" podID="6fbc6f93-c02b-4967-af97-57e896e8a96c" containerID="3ece96f1d7b152f26f0acf02bd1f82e0c2d6e08e30009a71938918bc85e2e5fc" exitCode=0
Sep 18 20:27:43.730625 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:43.730494    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-81f19d/rate-test-model-81f19d-kserve-6c5b976b67-njlhx"
Sep 18 20:27:43.730625 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:43.730517    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-81f19d/rate-test-model-81f19d-kserve-6c5b976b67-njlhx" event={"ID":"6fbc6f93-c02b-4967-af97-57e896e8a96c","Type":"ContainerDied","Data":"3ece96f1d7b152f26f0acf02bd1f82e0c2d6e08e30009a71938918bc85e2e5fc"}
Sep 18 20:27:43.730625 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:43.730569    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-81f19d/rate-test-model-81f19d-kserve-6c5b976b67-njlhx" event={"ID":"6fbc6f93-c02b-4967-af97-57e896e8a96c","Type":"ContainerDied","Data":"dbbb6fa0fef2c6a7794b31e76f4a76a812bbc4887c9ddbbf24a6f36d08a17140"}
Sep 18 20:27:43.730625 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:43.730586    2549 scope.go:117] "RemoveContainer" containerID="3ece96f1d7b152f26f0acf02bd1f82e0c2d6e08e30009a71938918bc85e2e5fc"
Sep 18 20:27:43.740606 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:43.740582    2549 scope.go:117] "RemoveContainer" containerID="3ece96f1d7b152f26f0acf02bd1f82e0c2d6e08e30009a71938918bc85e2e5fc"
Sep 18 20:27:43.740928 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:27:43.740909    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ece96f1d7b152f26f0acf02bd1f82e0c2d6e08e30009a71938918bc85e2e5fc\": container with ID starting with 3ece96f1d7b152f26f0acf02bd1f82e0c2d6e08e30009a71938918bc85e2e5fc not found: ID does not exist" containerID="3ece96f1d7b152f26f0acf02bd1f82e0c2d6e08e30009a71938918bc85e2e5fc"
Sep 18 20:27:43.740992 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:43.740938    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ece96f1d7b152f26f0acf02bd1f82e0c2d6e08e30009a71938918bc85e2e5fc"} err="failed to get container status \"3ece96f1d7b152f26f0acf02bd1f82e0c2d6e08e30009a71938918bc85e2e5fc\": rpc error: code = NotFound desc = could not find container \"3ece96f1d7b152f26f0acf02bd1f82e0c2d6e08e30009a71938918bc85e2e5fc\": container with ID starting with 3ece96f1d7b152f26f0acf02bd1f82e0c2d6e08e30009a71938918bc85e2e5fc not found: ID does not exist"
Sep 18 20:27:43.746774 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:43.746748    2549 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fbc6f93-c02b-4967-af97-57e896e8a96c-tls-certs\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:27:43.746950 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:43.746934    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqvxm\" (UniqueName: \"kubernetes.io/projected/6fbc6f93-c02b-4967-af97-57e896e8a96c-kube-api-access-vqvxm\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:27:43.747040 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:43.747028    2549 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fbc6f93-c02b-4967-af97-57e896e8a96c-tmp-dir\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:27:43.747123 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:43.747114    2549 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fbc6f93-c02b-4967-af97-57e896e8a96c-dshm\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:27:43.758014 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:43.757829    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-81f19d/rate-test-model-81f19d-kserve-6c5b976b67-njlhx"]
Sep 18 20:27:43.765189 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:43.765148    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-81f19d/rate-test-model-81f19d-kserve-6c5b976b67-njlhx"]
Sep 18 20:27:45.522260 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:45.522217    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6fbc6f93-c02b-4967-af97-57e896e8a96c" path="/var/lib/kubelet/pods/6fbc6f93-c02b-4967-af97-57e896e8a96c/volumes"
Sep 18 20:27:46.521466 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:46.521432    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:27:47.694578 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:47.694533    2549 patch_prober.go:28] interesting pod/e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 18 20:27:47.695006 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:47.694612    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd" podUID="e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 18 20:27:49.771341 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:49.771304    2549 generic.go:356] "Generic (PLEG): container finished" podID="e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd" containerID="0cff3f80ee7b26e4a5d0965f6a21c9b02ad67077c333be0dcad7edae8d927287" exitCode=0
Sep 18 20:27:49.771805 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:49.771392    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd" event={"ID":"e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd","Type":"ContainerDied","Data":"0cff3f80ee7b26e4a5d0965f6a21c9b02ad67077c333be0dcad7edae8d927287"}
Sep 18 20:27:49.771848 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:49.771826    2549 scope.go:117] "RemoveContainer" containerID="0cff3f80ee7b26e4a5d0965f6a21c9b02ad67077c333be0dcad7edae8d927287"
Sep 18 20:27:49.772946 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:49.772917    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:27:49.773003 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:49.772990    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:27:49.773037 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:49.773022    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:27:50.781725 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:50.781688    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd" event={"ID":"e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd","Type":"ContainerStarted","Data":"64a90dfd3c7153b493b837a618051e812015a4e99a1e849289dbebb4d7e4637a"}
Sep 18 20:27:51.273703 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:51.273653    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd"
Sep 18 20:27:51.273964 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:51.273886    2549 patch_prober.go:28] interesting pod/e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Sep 18 20:27:51.273964 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:51.273942    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd" podUID="e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Sep 18 20:27:52.273783 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:52.273739    2549 patch_prober.go:28] interesting pod/e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Sep 18 20:27:52.274205 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:52.273824    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd" podUID="e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Sep 18 20:27:53.273613 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:53.273561    2549 patch_prober.go:28] interesting pod/e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Sep 18 20:27:53.273842 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:53.273637    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd" podUID="e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Sep 18 20:27:54.274138 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:54.274089    2549 patch_prober.go:28] interesting pod/e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Sep 18 20:27:54.274596 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:54.274176    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd" podUID="e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Sep 18 20:27:55.274018 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:55.273965    2549 patch_prober.go:28] interesting pod/e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Sep 18 20:27:55.274261 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:55.274030    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd" podUID="e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Sep 18 20:27:56.016657 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:56.016621    2549 scope.go:117] "RemoveContainer" containerID="0dfda88c801eb9b8e33096d7384c5047f2a7017381c3fe8ab9089b08885a3ac7"
Sep 18 20:27:56.277432 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:56.277334    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd"
Sep 18 20:27:56.277828 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:56.277639    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd"
Sep 18 20:27:56.278564 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:27:56.278542    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd"
Sep 18 20:28:03.805946 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:03.805905    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt"]
Sep 18 20:28:03.806780 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:03.806533    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fbc6f93-c02b-4967-af97-57e896e8a96c" containerName="main"
Sep 18 20:28:03.806780 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:03.806553    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fbc6f93-c02b-4967-af97-57e896e8a96c" containerName="main"
Sep 18 20:28:03.806780 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:03.806650    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fbc6f93-c02b-4967-af97-57e896e8a96c" containerName="main"
Sep 18 20:28:03.808860 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:03.808843    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt"
Sep 18 20:28:03.811884 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:03.811862    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-a1a0a2-openshift-default-dockercfg-j25gn\""
Sep 18 20:28:03.843046 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:03.843007    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt"]
Sep 18 20:28:03.854773 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:03.854721    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1feabf84-a082-4982-a415-7efde1ee6a29-istio-podinfo\") pod \"e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt\" (UID: \"1feabf84-a082-4982-a415-7efde1ee6a29\") " pod="openshift-ingress/e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt"
Sep 18 20:28:03.854988 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:03.854787    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1feabf84-a082-4982-a415-7efde1ee6a29-credential-socket\") pod \"e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt\" (UID: \"1feabf84-a082-4982-a415-7efde1ee6a29\") " pod="openshift-ingress/e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt"
Sep 18 20:28:03.854988 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:03.854854    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1feabf84-a082-4982-a415-7efde1ee6a29-istiod-ca-cert\") pod \"e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt\" (UID: \"1feabf84-a082-4982-a415-7efde1ee6a29\") " pod="openshift-ingress/e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt"
Sep 18 20:28:03.854988 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:03.854897    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1feabf84-a082-4982-a415-7efde1ee6a29-istio-envoy\") pod \"e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt\" (UID: \"1feabf84-a082-4982-a415-7efde1ee6a29\") " pod="openshift-ingress/e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt"
Sep 18 20:28:03.854988 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:03.854916    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1feabf84-a082-4982-a415-7efde1ee6a29-istio-token\") pod \"e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt\" (UID: \"1feabf84-a082-4982-a415-7efde1ee6a29\") " pod="openshift-ingress/e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt"
Sep 18 20:28:03.855204 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:03.855022    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1feabf84-a082-4982-a415-7efde1ee6a29-istio-data\") pod \"e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt\" (UID: \"1feabf84-a082-4982-a415-7efde1ee6a29\") " pod="openshift-ingress/e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt"
Sep 18 20:28:03.855204 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:03.855087    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1feabf84-a082-4982-a415-7efde1ee6a29-workload-certs\") pod \"e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt\" (UID: \"1feabf84-a082-4982-a415-7efde1ee6a29\") " pod="openshift-ingress/e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt"
Sep 18 20:28:03.855204 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:03.855124    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ll42\" (UniqueName: \"kubernetes.io/projected/1feabf84-a082-4982-a415-7efde1ee6a29-kube-api-access-5ll42\") pod \"e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt\" (UID: \"1feabf84-a082-4982-a415-7efde1ee6a29\") " pod="openshift-ingress/e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt"
Sep 18 20:28:03.855325 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:03.855200    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1feabf84-a082-4982-a415-7efde1ee6a29-workload-socket\") pod \"e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt\" (UID: \"1feabf84-a082-4982-a415-7efde1ee6a29\") " pod="openshift-ingress/e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt"
Sep 18 20:28:03.956464 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:03.956412    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1feabf84-a082-4982-a415-7efde1ee6a29-istio-podinfo\") pod \"e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt\" (UID: \"1feabf84-a082-4982-a415-7efde1ee6a29\") " pod="openshift-ingress/e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt"
Sep 18 20:28:03.956464 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:03.956465    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1feabf84-a082-4982-a415-7efde1ee6a29-credential-socket\") pod \"e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt\" (UID: \"1feabf84-a082-4982-a415-7efde1ee6a29\") " pod="openshift-ingress/e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt"
Sep 18 20:28:03.956757 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:03.956506    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1feabf84-a082-4982-a415-7efde1ee6a29-istiod-ca-cert\") pod \"e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt\" (UID: \"1feabf84-a082-4982-a415-7efde1ee6a29\") " pod="openshift-ingress/e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt"
Sep 18 20:28:03.956757 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:03.956543    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1feabf84-a082-4982-a415-7efde1ee6a29-istio-envoy\") pod \"e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt\" (UID: \"1feabf84-a082-4982-a415-7efde1ee6a29\") " pod="openshift-ingress/e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt"
Sep 18 20:28:03.956757 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:03.956567    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1feabf84-a082-4982-a415-7efde1ee6a29-istio-token\") pod \"e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt\" (UID: \"1feabf84-a082-4982-a415-7efde1ee6a29\") " pod="openshift-ingress/e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt"
Sep 18 20:28:03.956757 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:03.956618    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1feabf84-a082-4982-a415-7efde1ee6a29-istio-data\") pod \"e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt\" (UID: \"1feabf84-a082-4982-a415-7efde1ee6a29\") " pod="openshift-ingress/e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt"
Sep 18 20:28:03.956971 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:03.956777    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1feabf84-a082-4982-a415-7efde1ee6a29-workload-certs\") pod \"e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt\" (UID: \"1feabf84-a082-4982-a415-7efde1ee6a29\") " pod="openshift-ingress/e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt"
Sep 18 20:28:03.956971 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:03.956819    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ll42\" (UniqueName: \"kubernetes.io/projected/1feabf84-a082-4982-a415-7efde1ee6a29-kube-api-access-5ll42\") pod \"e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt\" (UID: \"1feabf84-a082-4982-a415-7efde1ee6a29\") " pod="openshift-ingress/e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt"
Sep 18 20:28:03.956971 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:03.956872    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1feabf84-a082-4982-a415-7efde1ee6a29-workload-socket\") pod \"e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt\" (UID: \"1feabf84-a082-4982-a415-7efde1ee6a29\") " pod="openshift-ingress/e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt"
Sep 18 20:28:03.957244 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:03.957216    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1feabf84-a082-4982-a415-7efde1ee6a29-credential-socket\") pod \"e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt\" (UID: \"1feabf84-a082-4982-a415-7efde1ee6a29\") " pod="openshift-ingress/e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt"
Sep 18 20:28:03.957448 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:03.957264    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1feabf84-a082-4982-a415-7efde1ee6a29-istio-data\") pod \"e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt\" (UID: \"1feabf84-a082-4982-a415-7efde1ee6a29\") " pod="openshift-ingress/e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt"
Sep 18 20:28:03.957845 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:03.957553    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1feabf84-a082-4982-a415-7efde1ee6a29-workload-certs\") pod \"e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt\" (UID: \"1feabf84-a082-4982-a415-7efde1ee6a29\") " pod="openshift-ingress/e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt"
Sep 18 20:28:03.957845 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:03.957514    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1feabf84-a082-4982-a415-7efde1ee6a29-istiod-ca-cert\") pod \"e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt\" (UID: \"1feabf84-a082-4982-a415-7efde1ee6a29\") " pod="openshift-ingress/e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt"
Sep 18 20:28:03.957845 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:03.957804    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1feabf84-a082-4982-a415-7efde1ee6a29-workload-socket\") pod \"e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt\" (UID: \"1feabf84-a082-4982-a415-7efde1ee6a29\") " pod="openshift-ingress/e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt"
Sep 18 20:28:03.959535 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:03.959512    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1feabf84-a082-4982-a415-7efde1ee6a29-istio-envoy\") pod \"e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt\" (UID: \"1feabf84-a082-4982-a415-7efde1ee6a29\") " pod="openshift-ingress/e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt"
Sep 18 20:28:03.959535 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:03.959520    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1feabf84-a082-4982-a415-7efde1ee6a29-istio-podinfo\") pod \"e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt\" (UID: \"1feabf84-a082-4982-a415-7efde1ee6a29\") " pod="openshift-ingress/e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt"
Sep 18 20:28:03.970098 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:03.970062    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ll42\" (UniqueName: \"kubernetes.io/projected/1feabf84-a082-4982-a415-7efde1ee6a29-kube-api-access-5ll42\") pod \"e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt\" (UID: \"1feabf84-a082-4982-a415-7efde1ee6a29\") " pod="openshift-ingress/e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt"
Sep 18 20:28:03.973997 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:03.973970    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1feabf84-a082-4982-a415-7efde1ee6a29-istio-token\") pod \"e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt\" (UID: \"1feabf84-a082-4982-a415-7efde1ee6a29\") " pod="openshift-ingress/e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt"
Sep 18 20:28:04.120356 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:04.120309    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt"
Sep 18 20:28:04.412980 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:04.412944    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt"]
Sep 18 20:28:04.413334 ip-10-0-140-178 kubenswrapper[2549]: W0918 20:28:04.413307    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1feabf84_a082_4982_a415_7efde1ee6a29.slice/crio-d77c51f46b3641bd314ef7d80b49f999c80fb3c06a6ff6356c35a3fee8c7f8eb WatchSource:0}: Error finding container d77c51f46b3641bd314ef7d80b49f999c80fb3c06a6ff6356c35a3fee8c7f8eb: Status 404 returned error can't find the container with id d77c51f46b3641bd314ef7d80b49f999c80fb3c06a6ff6356c35a3fee8c7f8eb
Sep 18 20:28:04.415770 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:04.415737    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:28:04.415855 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:04.415809    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:28:04.415855 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:04.415843    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:28:04.877358 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:04.877316    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt" event={"ID":"1feabf84-a082-4982-a415-7efde1ee6a29","Type":"ContainerStarted","Data":"600846f036075b4a251bcb5da175a4c5d472ec2224df20962f1d75469b4c21cb"}
Sep 18 20:28:04.877803 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:04.877421    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt" event={"ID":"1feabf84-a082-4982-a415-7efde1ee6a29","Type":"ContainerStarted","Data":"d77c51f46b3641bd314ef7d80b49f999c80fb3c06a6ff6356c35a3fee8c7f8eb"}
Sep 18 20:28:04.909349 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:04.909276    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt" podStartSLOduration=1.9092598509999998 podStartE2EDuration="1.909259851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:28:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:28:04.907575972 +0000 UTC m=+1869.861317179" watchObservedRunningTime="2026-09-18 20:28:04.909259851 +0000 UTC m=+1869.863001015"
Sep 18 20:28:05.120737 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:05.120688    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt"
Sep 18 20:28:05.126268 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:05.126238    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt"
Sep 18 20:28:05.883627 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:05.883589    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt"
Sep 18 20:28:05.884753 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:05.884725    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt"
Sep 18 20:28:15.532562 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:15.532519    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:28:26.021348 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:26.021223    2549 generic.go:356] "Generic (PLEG): container finished" podID="1feabf84-a082-4982-a415-7efde1ee6a29" containerID="600846f036075b4a251bcb5da175a4c5d472ec2224df20962f1d75469b4c21cb" exitCode=0
Sep 18 20:28:26.021348 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:26.021292    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt" event={"ID":"1feabf84-a082-4982-a415-7efde1ee6a29","Type":"ContainerDied","Data":"600846f036075b4a251bcb5da175a4c5d472ec2224df20962f1d75469b4c21cb"}
Sep 18 20:28:26.021907 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:26.021806    2549 scope.go:117] "RemoveContainer" containerID="600846f036075b4a251bcb5da175a4c5d472ec2224df20962f1d75469b4c21cb"
Sep 18 20:28:26.023094 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:26.023059    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:28:26.023212 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:26.023159    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:28:26.023275 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:26.023210    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:28:27.029891 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:27.029851    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt" event={"ID":"1feabf84-a082-4982-a415-7efde1ee6a29","Type":"ContainerStarted","Data":"fd9b21f2e25d10145cecbfdabec67b804033906f8f3274319235345c4ca0ec29"}
Sep 18 20:28:27.120931 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:27.120882    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt"
Sep 18 20:28:27.121216 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:27.121191    2549 patch_prober.go:28] interesting pod/e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Sep 18 20:28:27.121267 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:27.121243    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt" podUID="1feabf84-a082-4982-a415-7efde1ee6a29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Sep 18 20:28:28.121787 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:28.121729    2549 patch_prober.go:28] interesting pod/e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Sep 18 20:28:28.122231 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:28.121826    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt" podUID="1feabf84-a082-4982-a415-7efde1ee6a29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Sep 18 20:28:29.121584 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:29.121543    2549 patch_prober.go:28] interesting pod/e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Sep 18 20:28:29.121817 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:29.121660    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt" podUID="1feabf84-a082-4982-a415-7efde1ee6a29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Sep 18 20:28:30.121465 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:30.121419    2549 patch_prober.go:28] interesting pod/e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Sep 18 20:28:30.121667 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:30.121488    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt" podUID="1feabf84-a082-4982-a415-7efde1ee6a29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Sep 18 20:28:31.121464 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:31.121421    2549 patch_prober.go:28] interesting pod/e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Sep 18 20:28:31.121893 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:31.121505    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt" podUID="1feabf84-a082-4982-a415-7efde1ee6a29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Sep 18 20:28:32.121563 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:32.121524    2549 patch_prober.go:28] interesting pod/e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Sep 18 20:28:32.121988 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:32.121587    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt" podUID="1feabf84-a082-4982-a415-7efde1ee6a29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Sep 18 20:28:33.125475 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:33.125440    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt"
Sep 18 20:28:33.125961 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:33.125761    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt"
Sep 18 20:28:33.126737 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:33.126720    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt"
Sep 18 20:28:46.510782 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:46.510739    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-de7ea3/ipp-route-de7ea3-kserve-55cd5894c-7n9jh"]
Sep 18 20:28:46.514917 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:46.514889    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-de7ea3/ipp-route-de7ea3-kserve-55cd5894c-7n9jh"
Sep 18 20:28:46.518249 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:46.518219    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-de7ea3\"/\"openshift-service-ca.crt\""
Sep 18 20:28:46.519761 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:46.519735    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-de7ea3\"/\"ipp-route-de7ea3-kserve-self-signed-certs\""
Sep 18 20:28:46.519921 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:46.519806    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-de7ea3\"/\"default-dockercfg-lqwzw\""
Sep 18 20:28:46.520114 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:46.519759    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-de7ea3\"/\"kube-root-ca.crt\""
Sep 18 20:28:46.530051 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:46.529994    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-de7ea3/ipp-route-de7ea3-kserve-55cd5894c-7n9jh"]
Sep 18 20:28:46.597878 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:46.597838    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a0bd34b-b468-4c50-8e64-1835907cc85a-model-cache\") pod \"ipp-route-de7ea3-kserve-55cd5894c-7n9jh\" (UID: \"8a0bd34b-b468-4c50-8e64-1835907cc85a\") " pod="ai-tenant-e2e-ipp-a-de7ea3/ipp-route-de7ea3-kserve-55cd5894c-7n9jh"
Sep 18 20:28:46.597878 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:46.597883    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d88zr\" (UniqueName: \"kubernetes.io/projected/8a0bd34b-b468-4c50-8e64-1835907cc85a-kube-api-access-d88zr\") pod \"ipp-route-de7ea3-kserve-55cd5894c-7n9jh\" (UID: \"8a0bd34b-b468-4c50-8e64-1835907cc85a\") " pod="ai-tenant-e2e-ipp-a-de7ea3/ipp-route-de7ea3-kserve-55cd5894c-7n9jh"
Sep 18 20:28:46.598123 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:46.597905    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a0bd34b-b468-4c50-8e64-1835907cc85a-tls-certs\") pod \"ipp-route-de7ea3-kserve-55cd5894c-7n9jh\" (UID: \"8a0bd34b-b468-4c50-8e64-1835907cc85a\") " pod="ai-tenant-e2e-ipp-a-de7ea3/ipp-route-de7ea3-kserve-55cd5894c-7n9jh"
Sep 18 20:28:46.598123 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:46.598012    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a0bd34b-b468-4c50-8e64-1835907cc85a-home\") pod \"ipp-route-de7ea3-kserve-55cd5894c-7n9jh\" (UID: \"8a0bd34b-b468-4c50-8e64-1835907cc85a\") " pod="ai-tenant-e2e-ipp-a-de7ea3/ipp-route-de7ea3-kserve-55cd5894c-7n9jh"
Sep 18 20:28:46.598123 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:46.598077    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a0bd34b-b468-4c50-8e64-1835907cc85a-dshm\") pod \"ipp-route-de7ea3-kserve-55cd5894c-7n9jh\" (UID: \"8a0bd34b-b468-4c50-8e64-1835907cc85a\") " pod="ai-tenant-e2e-ipp-a-de7ea3/ipp-route-de7ea3-kserve-55cd5894c-7n9jh"
Sep 18 20:28:46.598123 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:46.598115    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a0bd34b-b468-4c50-8e64-1835907cc85a-tmp-dir\") pod \"ipp-route-de7ea3-kserve-55cd5894c-7n9jh\" (UID: \"8a0bd34b-b468-4c50-8e64-1835907cc85a\") " pod="ai-tenant-e2e-ipp-a-de7ea3/ipp-route-de7ea3-kserve-55cd5894c-7n9jh"
Sep 18 20:28:46.699395 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:46.699343    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a0bd34b-b468-4c50-8e64-1835907cc85a-model-cache\") pod \"ipp-route-de7ea3-kserve-55cd5894c-7n9jh\" (UID: \"8a0bd34b-b468-4c50-8e64-1835907cc85a\") " pod="ai-tenant-e2e-ipp-a-de7ea3/ipp-route-de7ea3-kserve-55cd5894c-7n9jh"
Sep 18 20:28:46.699395 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:46.699395    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d88zr\" (UniqueName: \"kubernetes.io/projected/8a0bd34b-b468-4c50-8e64-1835907cc85a-kube-api-access-d88zr\") pod \"ipp-route-de7ea3-kserve-55cd5894c-7n9jh\" (UID: \"8a0bd34b-b468-4c50-8e64-1835907cc85a\") " pod="ai-tenant-e2e-ipp-a-de7ea3/ipp-route-de7ea3-kserve-55cd5894c-7n9jh"
Sep 18 20:28:46.699613 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:46.699569    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a0bd34b-b468-4c50-8e64-1835907cc85a-tls-certs\") pod \"ipp-route-de7ea3-kserve-55cd5894c-7n9jh\" (UID: \"8a0bd34b-b468-4c50-8e64-1835907cc85a\") " pod="ai-tenant-e2e-ipp-a-de7ea3/ipp-route-de7ea3-kserve-55cd5894c-7n9jh"
Sep 18 20:28:46.699660 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:46.699624    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a0bd34b-b468-4c50-8e64-1835907cc85a-home\") pod \"ipp-route-de7ea3-kserve-55cd5894c-7n9jh\" (UID: \"8a0bd34b-b468-4c50-8e64-1835907cc85a\") " pod="ai-tenant-e2e-ipp-a-de7ea3/ipp-route-de7ea3-kserve-55cd5894c-7n9jh"
Sep 18 20:28:46.699707 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:46.699691    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a0bd34b-b468-4c50-8e64-1835907cc85a-dshm\") pod \"ipp-route-de7ea3-kserve-55cd5894c-7n9jh\" (UID: \"8a0bd34b-b468-4c50-8e64-1835907cc85a\") " pod="ai-tenant-e2e-ipp-a-de7ea3/ipp-route-de7ea3-kserve-55cd5894c-7n9jh"
Sep 18 20:28:46.699756 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:46.699738    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a0bd34b-b468-4c50-8e64-1835907cc85a-model-cache\") pod \"ipp-route-de7ea3-kserve-55cd5894c-7n9jh\" (UID: \"8a0bd34b-b468-4c50-8e64-1835907cc85a\") " pod="ai-tenant-e2e-ipp-a-de7ea3/ipp-route-de7ea3-kserve-55cd5894c-7n9jh"
Sep 18 20:28:46.699824 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:46.699744    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a0bd34b-b468-4c50-8e64-1835907cc85a-tmp-dir\") pod \"ipp-route-de7ea3-kserve-55cd5894c-7n9jh\" (UID: \"8a0bd34b-b468-4c50-8e64-1835907cc85a\") " pod="ai-tenant-e2e-ipp-a-de7ea3/ipp-route-de7ea3-kserve-55cd5894c-7n9jh"
Sep 18 20:28:46.699976 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:46.699945    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a0bd34b-b468-4c50-8e64-1835907cc85a-home\") pod \"ipp-route-de7ea3-kserve-55cd5894c-7n9jh\" (UID: \"8a0bd34b-b468-4c50-8e64-1835907cc85a\") " pod="ai-tenant-e2e-ipp-a-de7ea3/ipp-route-de7ea3-kserve-55cd5894c-7n9jh"
Sep 18 20:28:46.700107 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:46.700032    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a0bd34b-b468-4c50-8e64-1835907cc85a-tmp-dir\") pod \"ipp-route-de7ea3-kserve-55cd5894c-7n9jh\" (UID: \"8a0bd34b-b468-4c50-8e64-1835907cc85a\") " pod="ai-tenant-e2e-ipp-a-de7ea3/ipp-route-de7ea3-kserve-55cd5894c-7n9jh"
Sep 18 20:28:46.707886 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:46.701884    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a0bd34b-b468-4c50-8e64-1835907cc85a-dshm\") pod \"ipp-route-de7ea3-kserve-55cd5894c-7n9jh\" (UID: \"8a0bd34b-b468-4c50-8e64-1835907cc85a\") " pod="ai-tenant-e2e-ipp-a-de7ea3/ipp-route-de7ea3-kserve-55cd5894c-7n9jh"
Sep 18 20:28:46.707886 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:46.702623    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a0bd34b-b468-4c50-8e64-1835907cc85a-tls-certs\") pod \"ipp-route-de7ea3-kserve-55cd5894c-7n9jh\" (UID: \"8a0bd34b-b468-4c50-8e64-1835907cc85a\") " pod="ai-tenant-e2e-ipp-a-de7ea3/ipp-route-de7ea3-kserve-55cd5894c-7n9jh"
Sep 18 20:28:46.713099 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:46.713065    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d88zr\" (UniqueName: \"kubernetes.io/projected/8a0bd34b-b468-4c50-8e64-1835907cc85a-kube-api-access-d88zr\") pod \"ipp-route-de7ea3-kserve-55cd5894c-7n9jh\" (UID: \"8a0bd34b-b468-4c50-8e64-1835907cc85a\") " pod="ai-tenant-e2e-ipp-a-de7ea3/ipp-route-de7ea3-kserve-55cd5894c-7n9jh"
Sep 18 20:28:46.825886 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:46.825843    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-de7ea3/ipp-route-de7ea3-kserve-55cd5894c-7n9jh"
Sep 18 20:28:47.112743 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:47.112713    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-de7ea3/ipp-route-de7ea3-kserve-55cd5894c-7n9jh"]
Sep 18 20:28:47.115329 ip-10-0-140-178 kubenswrapper[2549]: W0918 20:28:47.115295    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8a0bd34b_b468_4c50_8e64_1835907cc85a.slice/crio-c0b43a14eea244002f7ca93d6f67d5a2823b43ffe99cf7c7f5f10a52f6201149 WatchSource:0}: Error finding container c0b43a14eea244002f7ca93d6f67d5a2823b43ffe99cf7c7f5f10a52f6201149: Status 404 returned error can't find the container with id c0b43a14eea244002f7ca93d6f67d5a2823b43ffe99cf7c7f5f10a52f6201149
Sep 18 20:28:47.167753 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:47.167716    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-de7ea3/ipp-route-de7ea3-kserve-55cd5894c-7n9jh" event={"ID":"8a0bd34b-b468-4c50-8e64-1835907cc85a","Type":"ContainerStarted","Data":"c0b43a14eea244002f7ca93d6f67d5a2823b43ffe99cf7c7f5f10a52f6201149"}
Sep 18 20:28:48.176132 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:48.176092    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-de7ea3/ipp-route-de7ea3-kserve-55cd5894c-7n9jh" event={"ID":"8a0bd34b-b468-4c50-8e64-1835907cc85a","Type":"ContainerStarted","Data":"25ac30c6c5f09e75a5b105080583f07ed2f96ad1f5247a3e6b81d400efca11ce"}
Sep 18 20:28:48.213596 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:48.213533    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-de7ea3/ipp-route-de7ea3-kserve-55cd5894c-7n9jh" podStartSLOduration=2.213506061 podStartE2EDuration="2.213506061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:28:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:28:48.212026358 +0000 UTC m=+1913.165767535" watchObservedRunningTime="2026-09-18 20:28:48.213506061 +0000 UTC m=+1913.167247225"
Sep 18 20:28:51.298169 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:51.298125    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:28:56.044034 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:56.043996    2549 scope.go:117] "RemoveContainer" containerID="63efd62c4502c7ec637dca9da28258a5bc64b404fbba78a6e2c666947e0878f3"
Sep 18 20:28:56.826766 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:56.826723    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-de7ea3/ipp-route-de7ea3-kserve-55cd5894c-7n9jh"
Sep 18 20:28:56.826972 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:56.826785    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-de7ea3/ipp-route-de7ea3-kserve-55cd5894c-7n9jh"
Sep 18 20:28:56.840085 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:56.840052    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-de7ea3/ipp-route-de7ea3-kserve-55cd5894c-7n9jh"
Sep 18 20:28:57.243986 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:28:57.243892    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-de7ea3/ipp-route-de7ea3-kserve-55cd5894c-7n9jh"
Sep 18 20:29:12.315001 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:29:12.314956    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:29:19.270866 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:29:19.270820    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg"]
Sep 18 20:29:19.274830 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:29:19.274802    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg"
Sep 18 20:29:19.279482 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:29:19.279452    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-876382-openshift-default-dockercfg-7hvd2\""
Sep 18 20:29:19.348281 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:29:19.348242    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4083d77c-9e40-4bae-a220-4033bf795a30-istiod-ca-cert\") pod \"e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg\" (UID: \"4083d77c-9e40-4bae-a220-4033bf795a30\") " pod="openshift-ingress/e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg"
Sep 18 20:29:19.348516 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:29:19.348299    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4083d77c-9e40-4bae-a220-4033bf795a30-istio-envoy\") pod \"e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg\" (UID: \"4083d77c-9e40-4bae-a220-4033bf795a30\") " pod="openshift-ingress/e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg"
Sep 18 20:29:19.348516 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:29:19.348343    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4083d77c-9e40-4bae-a220-4033bf795a30-credential-socket\") pod \"e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg\" (UID: \"4083d77c-9e40-4bae-a220-4033bf795a30\") " pod="openshift-ingress/e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg"
Sep 18 20:29:19.348516 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:29:19.348466    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4083d77c-9e40-4bae-a220-4033bf795a30-istio-data\") pod \"e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg\" (UID: \"4083d77c-9e40-4bae-a220-4033bf795a30\") " pod="openshift-ingress/e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg"
Sep 18 20:29:19.348698 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:29:19.348549    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4083d77c-9e40-4bae-a220-4033bf795a30-workload-certs\") pod \"e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg\" (UID: \"4083d77c-9e40-4bae-a220-4033bf795a30\") " pod="openshift-ingress/e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg"
Sep 18 20:29:19.348698 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:29:19.348580    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh96q\" (UniqueName: \"kubernetes.io/projected/4083d77c-9e40-4bae-a220-4033bf795a30-kube-api-access-zh96q\") pod \"e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg\" (UID: \"4083d77c-9e40-4bae-a220-4033bf795a30\") " pod="openshift-ingress/e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg"
Sep 18 20:29:19.348698 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:29:19.348616    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4083d77c-9e40-4bae-a220-4033bf795a30-workload-socket\") pod \"e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg\" (UID: \"4083d77c-9e40-4bae-a220-4033bf795a30\") " pod="openshift-ingress/e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg"
Sep 18 20:29:19.348698 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:29:19.348684    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4083d77c-9e40-4bae-a220-4033bf795a30-istio-podinfo\") pod \"e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg\" (UID: \"4083d77c-9e40-4bae-a220-4033bf795a30\") " pod="openshift-ingress/e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg"
Sep 18 20:29:19.348845 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:29:19.348727    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4083d77c-9e40-4bae-a220-4033bf795a30-istio-token\") pod \"e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg\" (UID: \"4083d77c-9e40-4bae-a220-4033bf795a30\") " pod="openshift-ingress/e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg"
Sep 18 20:29:19.356039 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:29:19.355994    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg"]
Sep 18 20:29:19.449426 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:29:19.449355    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4083d77c-9e40-4bae-a220-4033bf795a30-workload-certs\") pod \"e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg\" (UID: \"4083d77c-9e40-4bae-a220-4033bf795a30\") " pod="openshift-ingress/e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg"
Sep 18 20:29:19.449426 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:29:19.449423    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh96q\" (UniqueName: \"kubernetes.io/projected/4083d77c-9e40-4bae-a220-4033bf795a30-kube-api-access-zh96q\") pod \"e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg\" (UID: \"4083d77c-9e40-4bae-a220-4033bf795a30\") " pod="openshift-ingress/e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg"
Sep 18 20:29:19.449704 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:29:19.449454    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4083d77c-9e40-4bae-a220-4033bf795a30-workload-socket\") pod \"e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg\" (UID: \"4083d77c-9e40-4bae-a220-4033bf795a30\") " pod="openshift-ingress/e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg"
Sep 18 20:29:19.449704 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:29:19.449501    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4083d77c-9e40-4bae-a220-4033bf795a30-istio-podinfo\") pod \"e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg\" (UID: \"4083d77c-9e40-4bae-a220-4033bf795a30\") " pod="openshift-ingress/e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg"
Sep 18 20:29:19.449704 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:29:19.449540    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4083d77c-9e40-4bae-a220-4033bf795a30-istio-token\") pod \"e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg\" (UID: \"4083d77c-9e40-4bae-a220-4033bf795a30\") " pod="openshift-ingress/e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg"
Sep 18 20:29:19.449704 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:29:19.449580    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4083d77c-9e40-4bae-a220-4033bf795a30-istiod-ca-cert\") pod \"e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg\" (UID: \"4083d77c-9e40-4bae-a220-4033bf795a30\") " pod="openshift-ingress/e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg"
Sep 18 20:29:19.449704 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:29:19.449621    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4083d77c-9e40-4bae-a220-4033bf795a30-istio-envoy\") pod \"e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg\" (UID: \"4083d77c-9e40-4bae-a220-4033bf795a30\") " pod="openshift-ingress/e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg"
Sep 18 20:29:19.449704 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:29:19.449666    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4083d77c-9e40-4bae-a220-4033bf795a30-credential-socket\") pod \"e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg\" (UID: \"4083d77c-9e40-4bae-a220-4033bf795a30\") " pod="openshift-ingress/e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg"
Sep 18 20:29:19.450038 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:29:19.449746    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4083d77c-9e40-4bae-a220-4033bf795a30-istio-data\") pod \"e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg\" (UID: \"4083d77c-9e40-4bae-a220-4033bf795a30\") " pod="openshift-ingress/e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg"
Sep 18 20:29:19.450038 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:29:19.449787    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4083d77c-9e40-4bae-a220-4033bf795a30-workload-certs\") pod \"e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg\" (UID: \"4083d77c-9e40-4bae-a220-4033bf795a30\") " pod="openshift-ingress/e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg"
Sep 18 20:29:19.450250 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:29:19.450227    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4083d77c-9e40-4bae-a220-4033bf795a30-workload-socket\") pod \"e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg\" (UID: \"4083d77c-9e40-4bae-a220-4033bf795a30\") " pod="openshift-ingress/e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg"
Sep 18 20:29:19.450318 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:29:19.450250    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4083d77c-9e40-4bae-a220-4033bf795a30-credential-socket\") pod \"e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg\" (UID: \"4083d77c-9e40-4bae-a220-4033bf795a30\") " pod="openshift-ingress/e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg"
Sep 18 20:29:19.450557 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:29:19.450526    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4083d77c-9e40-4bae-a220-4033bf795a30-istiod-ca-cert\") pod \"e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg\" (UID: \"4083d77c-9e40-4bae-a220-4033bf795a30\") " pod="openshift-ingress/e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg"
Sep 18 20:29:19.450683 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:29:19.450584    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4083d77c-9e40-4bae-a220-4033bf795a30-istio-data\") pod \"e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg\" (UID: \"4083d77c-9e40-4bae-a220-4033bf795a30\") " pod="openshift-ingress/e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg"
Sep 18 20:29:19.453430 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:29:19.452642    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4083d77c-9e40-4bae-a220-4033bf795a30-istio-podinfo\") pod \"e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg\" (UID: \"4083d77c-9e40-4bae-a220-4033bf795a30\") " pod="openshift-ingress/e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg"
Sep 18 20:29:19.453430 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:29:19.452692    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4083d77c-9e40-4bae-a220-4033bf795a30-istio-envoy\") pod \"e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg\" (UID: \"4083d77c-9e40-4bae-a220-4033bf795a30\") " pod="openshift-ingress/e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg"
Sep 18 20:29:19.470250 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:29:19.470211    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4083d77c-9e40-4bae-a220-4033bf795a30-istio-token\") pod \"e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg\" (UID: \"4083d77c-9e40-4bae-a220-4033bf795a30\") " pod="openshift-ingress/e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg"
Sep 18 20:29:19.488442 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:29:19.488363    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh96q\" (UniqueName: \"kubernetes.io/projected/4083d77c-9e40-4bae-a220-4033bf795a30-kube-api-access-zh96q\") pod \"e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg\" (UID: \"4083d77c-9e40-4bae-a220-4033bf795a30\") " pod="openshift-ingress/e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg"
Sep 18 20:29:19.586573 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:29:19.586536    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg"
Sep 18 20:29:19.877533 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:29:19.877503    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg"]
Sep 18 20:29:19.879125 ip-10-0-140-178 kubenswrapper[2549]: W0918 20:29:19.879095    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4083d77c_9e40_4bae_a220_4033bf795a30.slice/crio-5d922f9ea1bf65845ba0f6319e2fe7e5a5e9641e8f464ff750a33214a61e8512 WatchSource:0}: Error finding container 5d922f9ea1bf65845ba0f6319e2fe7e5a5e9641e8f464ff750a33214a61e8512: Status 404 returned error can't find the container with id 5d922f9ea1bf65845ba0f6319e2fe7e5a5e9641e8f464ff750a33214a61e8512
Sep 18 20:29:19.881608 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:29:19.881573    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:29:19.881731 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:29:19.881662    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:29:19.881800 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:29:19.881748    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:29:20.372966 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:29:20.372929    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg" event={"ID":"4083d77c-9e40-4bae-a220-4033bf795a30","Type":"ContainerStarted","Data":"d63269325b9fedc92500c925bb09f652af1a197d8627211cf231e4995d0fd6b5"}
Sep 18 20:29:20.373419 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:29:20.372988    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg" event={"ID":"4083d77c-9e40-4bae-a220-4033bf795a30","Type":"ContainerStarted","Data":"5d922f9ea1bf65845ba0f6319e2fe7e5a5e9641e8f464ff750a33214a61e8512"}
Sep 18 20:29:20.402082 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:29:20.402020    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg" podStartSLOduration=1.402001062 podStartE2EDuration="1.402001062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:29:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:29:20.399251111 +0000 UTC m=+1945.352992282" watchObservedRunningTime="2026-09-18 20:29:20.402001062 +0000 UTC m=+1945.355742230"
Sep 18 20:29:20.587071 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:29:20.587026    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg"
Sep 18 20:29:20.593258 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:29:20.593227    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg"
Sep 18 20:29:21.380437 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:29:21.380401    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg"
Sep 18 20:29:21.381609 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:29:21.381588    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg"
Sep 18 20:29:33.822009 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:29:33.821952    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:29:44.529883 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:29:44.529845    2549 generic.go:356] "Generic (PLEG): container finished" podID="4083d77c-9e40-4bae-a220-4033bf795a30" containerID="d63269325b9fedc92500c925bb09f652af1a197d8627211cf231e4995d0fd6b5" exitCode=0
Sep 18 20:29:44.530365 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:29:44.529921    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg" event={"ID":"4083d77c-9e40-4bae-a220-4033bf795a30","Type":"ContainerDied","Data":"d63269325b9fedc92500c925bb09f652af1a197d8627211cf231e4995d0fd6b5"}
Sep 18 20:29:44.530460 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:29:44.530428    2549 scope.go:117] "RemoveContainer" containerID="d63269325b9fedc92500c925bb09f652af1a197d8627211cf231e4995d0fd6b5"
Sep 18 20:29:44.531714 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:29:44.531682    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:29:44.531836 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:29:44.531776    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:29:44.531836 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:29:44.531825    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:29:45.537660 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:29:45.537621    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg" event={"ID":"4083d77c-9e40-4bae-a220-4033bf795a30","Type":"ContainerStarted","Data":"7dd3c50f1c86cf21b637bfe2f51730fa8ae978695f4bf9adf7299c196ef36353"}
Sep 18 20:29:45.586798 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:29:45.586749    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg"
Sep 18 20:29:45.587064 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:29:45.587039    2549 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Sep 18 20:29:45.587141 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:29:45.587092    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg" podUID="4083d77c-9e40-4bae-a220-4033bf795a30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Sep 18 20:29:46.587587 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:29:46.587542    2549 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Sep 18 20:29:46.587980 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:29:46.587625    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg" podUID="4083d77c-9e40-4bae-a220-4033bf795a30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Sep 18 20:29:47.587998 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:29:47.587954    2549 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Sep 18 20:29:47.588449 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:29:47.588034    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg" podUID="4083d77c-9e40-4bae-a220-4033bf795a30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Sep 18 20:29:48.587263 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:29:48.587223    2549 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Sep 18 20:29:48.587522 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:29:48.587315    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg" podUID="4083d77c-9e40-4bae-a220-4033bf795a30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Sep 18 20:29:49.586970 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:29:49.586927    2549 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Sep 18 20:29:49.587428 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:29:49.586988    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg" podUID="4083d77c-9e40-4bae-a220-4033bf795a30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Sep 18 20:29:50.587420 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:29:50.587334    2549 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Sep 18 20:29:50.587934 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:29:50.587425    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg" podUID="4083d77c-9e40-4bae-a220-4033bf795a30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Sep 18 20:29:51.381024 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:29:51.380968    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg"
Sep 18 20:29:51.593529 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:29:51.593490    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg"
Sep 18 20:29:51.594817 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:29:51.594795    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg"
Sep 18 20:29:53.423791 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:29:53.423746    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-876382-46r8d"]
Sep 18 20:29:53.427067 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:29:53.427045    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-876382-46r8d"
Sep 18 20:29:53.433171 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:29:53.433141    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 20:29:53.433332 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:29:53.433172    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bhssz\""
Sep 18 20:29:53.445362 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:29:53.445321    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-876382-46r8d"]
Sep 18 20:29:53.539299 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:29:53.539252    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f7ac3d6-f847-4cb8-8b76-63f530388ae1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-876382-46r8d\" (UID: \"1f7ac3d6-f847-4cb8-8b76-63f530388ae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-876382-46r8d"
Sep 18 20:29:53.640791 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:29:53.640753    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f7ac3d6-f847-4cb8-8b76-63f530388ae1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-876382-46r8d\" (UID: \"1f7ac3d6-f847-4cb8-8b76-63f530388ae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-876382-46r8d"
Sep 18 20:29:53.641435 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:29:53.641379    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f7ac3d6-f847-4cb8-8b76-63f530388ae1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-876382-46r8d\" (UID: \"1f7ac3d6-f847-4cb8-8b76-63f530388ae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-876382-46r8d"
Sep 18 20:29:53.737661 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:29:53.737563    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-876382-46r8d"
Sep 18 20:29:54.031182 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:29:54.031153    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-876382-46r8d"]
Sep 18 20:29:54.034147 ip-10-0-140-178 kubenswrapper[2549]: W0918 20:29:54.034104    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f7ac3d6_f847_4cb8_8b76_63f530388ae1.slice/crio-9585cba005e66e0e9121f88e43de689d9b5f1546ae90f4f1bc3e03661fd3648b WatchSource:0}: Error finding container 9585cba005e66e0e9121f88e43de689d9b5f1546ae90f4f1bc3e03661fd3648b: Status 404 returned error can't find the container with id 9585cba005e66e0e9121f88e43de689d9b5f1546ae90f4f1bc3e03661fd3648b
Sep 18 20:29:54.606829 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:29:54.606732    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-876382-46r8d" event={"ID":"1f7ac3d6-f847-4cb8-8b76-63f530388ae1","Type":"ContainerStarted","Data":"469b04f1177c8e3226c130c578ac63edeaff26863beda87bd8962acecaeccf09"}
Sep 18 20:29:54.606829 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:29:54.606799    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-876382-46r8d" event={"ID":"1f7ac3d6-f847-4cb8-8b76-63f530388ae1","Type":"ContainerStarted","Data":"9585cba005e66e0e9121f88e43de689d9b5f1546ae90f4f1bc3e03661fd3648b"}
Sep 18 20:29:54.638218 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:29:54.638161    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-876382-46r8d" podStartSLOduration=1.638141939 podStartE2EDuration="1.638141939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:29:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:29:54.636831707 +0000 UTC m=+1979.590572876" watchObservedRunningTime="2026-09-18 20:29:54.638141939 +0000 UTC m=+1979.591883175"
Sep 18 20:29:55.614374 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:29:55.614337    2549 generic.go:356] "Generic (PLEG): container finished" podID="1f7ac3d6-f847-4cb8-8b76-63f530388ae1" containerID="469b04f1177c8e3226c130c578ac63edeaff26863beda87bd8962acecaeccf09" exitCode=0
Sep 18 20:29:55.614824 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:29:55.614409    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-876382-46r8d" event={"ID":"1f7ac3d6-f847-4cb8-8b76-63f530388ae1","Type":"ContainerDied","Data":"469b04f1177c8e3226c130c578ac63edeaff26863beda87bd8962acecaeccf09"}
Sep 18 20:29:56.768901 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:29:56.768868    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-876382-46r8d"
Sep 18 20:29:56.876953 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:29:56.876851    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f7ac3d6-f847-4cb8-8b76-63f530388ae1-maas-api-service-ca\") pod \"1f7ac3d6-f847-4cb8-8b76-63f530388ae1\" (UID: \"1f7ac3d6-f847-4cb8-8b76-63f530388ae1\") "
Sep 18 20:29:56.877258 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:29:56.877231    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f7ac3d6-f847-4cb8-8b76-63f530388ae1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1f7ac3d6-f847-4cb8-8b76-63f530388ae1" (UID: "1f7ac3d6-f847-4cb8-8b76-63f530388ae1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:29:56.978633 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:29:56.978592    2549 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f7ac3d6-f847-4cb8-8b76-63f530388ae1-maas-api-service-ca\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:29:57.628873 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:29:57.628836    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-876382-46r8d"
Sep 18 20:29:57.629073 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:29:57.628878    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-876382-46r8d" event={"ID":"1f7ac3d6-f847-4cb8-8b76-63f530388ae1","Type":"ContainerDied","Data":"9585cba005e66e0e9121f88e43de689d9b5f1546ae90f4f1bc3e03661fd3648b"}
Sep 18 20:29:57.629073 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:29:57.628941    2549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9585cba005e66e0e9121f88e43de689d9b5f1546ae90f4f1bc3e03661fd3648b"
Sep 18 20:30:00.242538 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:00.242491    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29829390-rsbw2"]
Sep 18 20:30:00.243022 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:00.243007    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f7ac3d6-f847-4cb8-8b76-63f530388ae1" containerName="revoke-keys"
Sep 18 20:30:00.243076 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:00.243024    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f7ac3d6-f847-4cb8-8b76-63f530388ae1" containerName="revoke-keys"
Sep 18 20:30:00.243122 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:00.243103    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f7ac3d6-f847-4cb8-8b76-63f530388ae1" containerName="revoke-keys"
Sep 18 20:30:00.245409 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:00.245361    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829390-rsbw2"
Sep 18 20:30:00.248641 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:00.248610    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-81f19d-29829390-tgq9q"]
Sep 18 20:30:00.249645 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:00.249621    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bhssz\""
Sep 18 20:30:00.251300 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:00.251283    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-81f19d-29829390-tgq9q"
Sep 18 20:30:00.257447 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:00.257427    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829390-rsbw2"
Sep 18 20:30:00.265314 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:00.265253    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29829390-rsbw2"]
Sep 18 20:30:00.265818 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:00.265793    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-81f19d-29829390-tgq9q"
Sep 18 20:30:00.283912 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:00.283872    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-81f19d-29829390-tgq9q"]
Sep 18 20:30:00.294033 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:00.292119    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-de7ea3-29829390-zkr9t"]
Sep 18 20:30:00.295754 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:00.295717    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-de7ea3-29829390-zkr9t"
Sep 18 20:30:00.308917 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:00.308869    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-876382-29829390-bwbjk"]
Sep 18 20:30:00.317068 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:00.316995    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-de7ea3-29829390-zkr9t"]
Sep 18 20:30:00.317281 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:00.317143    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-876382-29829390-bwbjk"
Sep 18 20:30:00.333049 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:00.332807    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-de7ea3-29829390-zkr9t"
Sep 18 20:30:00.334529 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:00.334303    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-876382-29829390-bwbjk"]
Sep 18 20:30:00.384372 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:00.382713    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-876382-29829390-bwbjk"
Sep 18 20:30:00.582596 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:00.582568    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29829390-rsbw2"]
Sep 18 20:30:00.584967 ip-10-0-140-178 kubenswrapper[2549]: W0918 20:30:00.584942    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda170b409_cd6b_4543_9667_e10488259c6b.slice/crio-d97ac78e8ccbd797aa5279eeff6cba15246e8886bbac346ae5a236f4ebc33c85 WatchSource:0}: Error finding container d97ac78e8ccbd797aa5279eeff6cba15246e8886bbac346ae5a236f4ebc33c85: Status 404 returned error can't find the container with id d97ac78e8ccbd797aa5279eeff6cba15246e8886bbac346ae5a236f4ebc33c85
Sep 18 20:30:00.594666 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:00.594643    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-81f19d-29829390-tgq9q"]
Sep 18 20:30:00.597487 ip-10-0-140-178 kubenswrapper[2549]: W0918 20:30:00.597446    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0f05846_6140_4298_a26d_21d9c15db0f3.slice/crio-328c6ef9085706df08dba7d8e261fdde576c719728194fd7ef289ab226f939b8 WatchSource:0}: Error finding container 328c6ef9085706df08dba7d8e261fdde576c719728194fd7ef289ab226f939b8: Status 404 returned error can't find the container with id 328c6ef9085706df08dba7d8e261fdde576c719728194fd7ef289ab226f939b8
Sep 18 20:30:00.639165 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:00.639133    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-de7ea3-29829390-zkr9t"]
Sep 18 20:30:00.639917 ip-10-0-140-178 kubenswrapper[2549]: W0918 20:30:00.639891    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3350e426_add0_4440_8858_98b4ec1715da.slice/crio-09243e8141645b1d700b4fe5a834c6d68726b36cba403d34001d9daf0ebc22f6 WatchSource:0}: Error finding container 09243e8141645b1d700b4fe5a834c6d68726b36cba403d34001d9daf0ebc22f6: Status 404 returned error can't find the container with id 09243e8141645b1d700b4fe5a834c6d68726b36cba403d34001d9daf0ebc22f6
Sep 18 20:30:00.648147 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:00.648114    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829390-rsbw2" event={"ID":"a170b409-cd6b-4543-9667-e10488259c6b","Type":"ContainerStarted","Data":"d97ac78e8ccbd797aa5279eeff6cba15246e8886bbac346ae5a236f4ebc33c85"}
Sep 18 20:30:00.649706 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:00.649676    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-de7ea3-29829390-zkr9t" event={"ID":"3350e426-add0-4440-8858-98b4ec1715da","Type":"ContainerStarted","Data":"09243e8141645b1d700b4fe5a834c6d68726b36cba403d34001d9daf0ebc22f6"}
Sep 18 20:30:00.651987 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:00.651953    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-81f19d-29829390-tgq9q" event={"ID":"b0f05846-6140-4298-a26d-21d9c15db0f3","Type":"ContainerStarted","Data":"328c6ef9085706df08dba7d8e261fdde576c719728194fd7ef289ab226f939b8"}
Sep 18 20:30:00.673883 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:00.673847    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-876382-29829390-bwbjk"]
Sep 18 20:30:00.676633 ip-10-0-140-178 kubenswrapper[2549]: W0918 20:30:00.676600    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88784bd8_8cca_43a5_ad69_a64fcaf71617.slice/crio-caefe7f9093de03c05ab1ee5f4953ab7a9ee29dbd94f0dfcf7e54236cc7e069d WatchSource:0}: Error finding container caefe7f9093de03c05ab1ee5f4953ab7a9ee29dbd94f0dfcf7e54236cc7e069d: Status 404 returned error can't find the container with id caefe7f9093de03c05ab1ee5f4953ab7a9ee29dbd94f0dfcf7e54236cc7e069d
Sep 18 20:30:01.660528 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:01.660488    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-876382-29829390-bwbjk" event={"ID":"88784bd8-8cca-43a5-ad69-a64fcaf71617","Type":"ContainerStarted","Data":"0009ba35b4c89344f36ee09f76c5e95d7ee7ccde37c2d77dc0a7f783ffc456a8"}
Sep 18 20:30:01.660917 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:01.660555    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-876382-29829390-bwbjk" event={"ID":"88784bd8-8cca-43a5-ad69-a64fcaf71617","Type":"ContainerStarted","Data":"caefe7f9093de03c05ab1ee5f4953ab7a9ee29dbd94f0dfcf7e54236cc7e069d"}
Sep 18 20:30:01.662226 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:01.662204    2549 generic.go:356] "Generic (PLEG): container finished" podID="a170b409-cd6b-4543-9667-e10488259c6b" containerID="13b350ec3fefe248e7fd7dd3a65c824771e595b31460367b388c3d0abf026106" exitCode=0
Sep 18 20:30:01.662347 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:01.662285    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829390-rsbw2" event={"ID":"a170b409-cd6b-4543-9667-e10488259c6b","Type":"ContainerDied","Data":"13b350ec3fefe248e7fd7dd3a65c824771e595b31460367b388c3d0abf026106"}
Sep 18 20:30:01.663821 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:01.663795    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-de7ea3-29829390-zkr9t" event={"ID":"3350e426-add0-4440-8858-98b4ec1715da","Type":"ContainerStarted","Data":"882f6e755167c274a34bea51f3baa5be6a587214e67ece6e049be2419c5a9063"}
Sep 18 20:30:01.665266 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:01.665242    2549 generic.go:356] "Generic (PLEG): container finished" podID="b0f05846-6140-4298-a26d-21d9c15db0f3" containerID="c258e8ddc27252faf014f783e1aa2187aed8da788e668cc8b7546707b8aa80db" exitCode=0
Sep 18 20:30:01.665358 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:01.665305    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-81f19d-29829390-tgq9q" event={"ID":"b0f05846-6140-4298-a26d-21d9c15db0f3","Type":"ContainerDied","Data":"c258e8ddc27252faf014f783e1aa2187aed8da788e668cc8b7546707b8aa80db"}
Sep 18 20:30:01.687177 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:01.687119    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-876382-29829390-bwbjk" podStartSLOduration=1.687101545 podStartE2EDuration="1.687101545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:30:01.686807389 +0000 UTC m=+1986.640548554" watchObservedRunningTime="2026-09-18 20:30:01.687101545 +0000 UTC m=+1986.640842702"
Sep 18 20:30:01.747169 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:01.747111    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-de7ea3-29829390-zkr9t" podStartSLOduration=1.7470896040000001 podStartE2EDuration="1.747089604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:30:01.745984361 +0000 UTC m=+1986.699725530" watchObservedRunningTime="2026-09-18 20:30:01.747089604 +0000 UTC m=+1986.700830773"
Sep 18 20:30:02.673779 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:02.673740    2549 generic.go:356] "Generic (PLEG): container finished" podID="88784bd8-8cca-43a5-ad69-a64fcaf71617" containerID="0009ba35b4c89344f36ee09f76c5e95d7ee7ccde37c2d77dc0a7f783ffc456a8" exitCode=0
Sep 18 20:30:02.674275 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:02.673831    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-876382-29829390-bwbjk" event={"ID":"88784bd8-8cca-43a5-ad69-a64fcaf71617","Type":"ContainerDied","Data":"0009ba35b4c89344f36ee09f76c5e95d7ee7ccde37c2d77dc0a7f783ffc456a8"}
Sep 18 20:30:02.675659 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:02.675633    2549 generic.go:356] "Generic (PLEG): container finished" podID="3350e426-add0-4440-8858-98b4ec1715da" containerID="882f6e755167c274a34bea51f3baa5be6a587214e67ece6e049be2419c5a9063" exitCode=0
Sep 18 20:30:02.675765 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:02.675709    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-de7ea3-29829390-zkr9t" event={"ID":"3350e426-add0-4440-8858-98b4ec1715da","Type":"ContainerDied","Data":"882f6e755167c274a34bea51f3baa5be6a587214e67ece6e049be2419c5a9063"}
Sep 18 20:30:02.834172 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:02.834139    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829390-rsbw2"
Sep 18 20:30:02.873064 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:02.873037    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-81f19d-29829390-tgq9q"
Sep 18 20:30:03.683514 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:03.683478    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-81f19d-29829390-tgq9q"
Sep 18 20:30:03.683958 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:03.683523    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-81f19d-29829390-tgq9q" event={"ID":"b0f05846-6140-4298-a26d-21d9c15db0f3","Type":"ContainerDied","Data":"328c6ef9085706df08dba7d8e261fdde576c719728194fd7ef289ab226f939b8"}
Sep 18 20:30:03.683958 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:03.683571    2549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="328c6ef9085706df08dba7d8e261fdde576c719728194fd7ef289ab226f939b8"
Sep 18 20:30:03.684986 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:03.684955    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829390-rsbw2" event={"ID":"a170b409-cd6b-4543-9667-e10488259c6b","Type":"ContainerDied","Data":"d97ac78e8ccbd797aa5279eeff6cba15246e8886bbac346ae5a236f4ebc33c85"}
Sep 18 20:30:03.684986 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:03.684978    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829390-rsbw2"
Sep 18 20:30:03.685179 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:03.684996    2549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d97ac78e8ccbd797aa5279eeff6cba15246e8886bbac346ae5a236f4ebc33c85"
Sep 18 20:30:03.847858 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:03.847826    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-de7ea3-29829390-zkr9t"
Sep 18 20:30:03.851526 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:03.851494    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-876382-29829390-bwbjk"
Sep 18 20:30:04.696402 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:04.696356    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-876382-29829390-bwbjk"
Sep 18 20:30:04.696813 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:04.696376    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-876382-29829390-bwbjk" event={"ID":"88784bd8-8cca-43a5-ad69-a64fcaf71617","Type":"ContainerDied","Data":"caefe7f9093de03c05ab1ee5f4953ab7a9ee29dbd94f0dfcf7e54236cc7e069d"}
Sep 18 20:30:04.696813 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:04.696465    2549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="caefe7f9093de03c05ab1ee5f4953ab7a9ee29dbd94f0dfcf7e54236cc7e069d"
Sep 18 20:30:04.698174 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:04.698143    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-de7ea3-29829390-zkr9t" event={"ID":"3350e426-add0-4440-8858-98b4ec1715da","Type":"ContainerDied","Data":"09243e8141645b1d700b4fe5a834c6d68726b36cba403d34001d9daf0ebc22f6"}
Sep 18 20:30:04.698328 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:04.698205    2549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="09243e8141645b1d700b4fe5a834c6d68726b36cba403d34001d9daf0ebc22f6"
Sep 18 20:30:04.698328 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:04.698157    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-de7ea3-29829390-zkr9t"
Sep 18 20:30:04.763110 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:04.763065    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-876382-29829390-bwbjk"]
Sep 18 20:30:04.767915 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:04.767877    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-876382-29829390-bwbjk"]
Sep 18 20:30:05.119915 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:05.119875    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-876382-46r8d"]
Sep 18 20:30:05.124580 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:05.124547    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-876382-46r8d"]
Sep 18 20:30:05.522256 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:05.522161    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f7ac3d6-f847-4cb8-8b76-63f530388ae1" path="/var/lib/kubelet/pods/1f7ac3d6-f847-4cb8-8b76-63f530388ae1/volumes"
Sep 18 20:30:05.522523 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:05.522508    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88784bd8-8cca-43a5-ad69-a64fcaf71617" path="/var/lib/kubelet/pods/88784bd8-8cca-43a5-ad69-a64fcaf71617/volumes"
Sep 18 20:30:08.023488 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:08.023451    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:30:12.307226 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:12.307185    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg"]
Sep 18 20:30:12.307759 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:12.307443    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg" podUID="4083d77c-9e40-4bae-a220-4033bf795a30" containerName="istio-proxy" containerID="cri-o://7dd3c50f1c86cf21b637bfe2f51730fa8ae978695f4bf9adf7299c196ef36353" gracePeriod=30
Sep 18 20:30:15.909571 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:15.909533    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-de7ea3-ttsjp"]
Sep 18 20:30:15.910009 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:15.909994    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a170b409-cd6b-4543-9667-e10488259c6b" containerName="cleanup"
Sep 18 20:30:15.910009 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:15.910010    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="a170b409-cd6b-4543-9667-e10488259c6b" containerName="cleanup"
Sep 18 20:30:15.910095 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:15.910018    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0f05846-6140-4298-a26d-21d9c15db0f3" containerName="cleanup"
Sep 18 20:30:15.910095 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:15.910025    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0f05846-6140-4298-a26d-21d9c15db0f3" containerName="cleanup"
Sep 18 20:30:15.910095 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:15.910032    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3350e426-add0-4440-8858-98b4ec1715da" containerName="cleanup"
Sep 18 20:30:15.910095 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:15.910038    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="3350e426-add0-4440-8858-98b4ec1715da" containerName="cleanup"
Sep 18 20:30:15.910095 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:15.910051    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88784bd8-8cca-43a5-ad69-a64fcaf71617" containerName="cleanup"
Sep 18 20:30:15.910095 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:15.910056    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="88784bd8-8cca-43a5-ad69-a64fcaf71617" containerName="cleanup"
Sep 18 20:30:15.910281 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:15.910129    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="a170b409-cd6b-4543-9667-e10488259c6b" containerName="cleanup"
Sep 18 20:30:15.910281 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:15.910136    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="3350e426-add0-4440-8858-98b4ec1715da" containerName="cleanup"
Sep 18 20:30:15.910281 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:15.910145    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0f05846-6140-4298-a26d-21d9c15db0f3" containerName="cleanup"
Sep 18 20:30:15.910281 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:15.910153    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="88784bd8-8cca-43a5-ad69-a64fcaf71617" containerName="cleanup"
Sep 18 20:30:15.914695 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:15.914672    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-de7ea3-ttsjp"
Sep 18 20:30:15.918315 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:15.918281    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 20:30:15.918538 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:15.918396    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bhssz\""
Sep 18 20:30:15.933510 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:15.933467    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-de7ea3-ttsjp"]
Sep 18 20:30:15.984625 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:15.984576    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0aad8fa-9518-4ed0-97fe-a03802e278a7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-de7ea3-ttsjp\" (UID: \"b0aad8fa-9518-4ed0-97fe-a03802e278a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-de7ea3-ttsjp"
Sep 18 20:30:16.085732 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:16.085676    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0aad8fa-9518-4ed0-97fe-a03802e278a7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-de7ea3-ttsjp\" (UID: \"b0aad8fa-9518-4ed0-97fe-a03802e278a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-de7ea3-ttsjp"
Sep 18 20:30:16.086625 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:16.086597    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0aad8fa-9518-4ed0-97fe-a03802e278a7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-de7ea3-ttsjp\" (UID: \"b0aad8fa-9518-4ed0-97fe-a03802e278a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-de7ea3-ttsjp"
Sep 18 20:30:16.229314 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:16.229215    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-de7ea3-ttsjp"
Sep 18 20:30:16.508067 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:16.508033    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-de7ea3-ttsjp"]
Sep 18 20:30:16.509029 ip-10-0-140-178 kubenswrapper[2549]: W0918 20:30:16.508999    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0aad8fa_9518_4ed0_97fe_a03802e278a7.slice/crio-f24c199ef8269081d8c199ec6e3b49cae7c3c73c09caf902044d7f16d77aec4f WatchSource:0}: Error finding container f24c199ef8269081d8c199ec6e3b49cae7c3c73c09caf902044d7f16d77aec4f: Status 404 returned error can't find the container with id f24c199ef8269081d8c199ec6e3b49cae7c3c73c09caf902044d7f16d77aec4f
Sep 18 20:30:16.778131 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:16.778081    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-de7ea3-ttsjp" event={"ID":"b0aad8fa-9518-4ed0-97fe-a03802e278a7","Type":"ContainerStarted","Data":"f24c199ef8269081d8c199ec6e3b49cae7c3c73c09caf902044d7f16d77aec4f"}
Sep 18 20:30:17.585146 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:17.585115    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg"
Sep 18 20:30:17.602591 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:17.602486    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4083d77c-9e40-4bae-a220-4033bf795a30-workload-socket\") pod \"4083d77c-9e40-4bae-a220-4033bf795a30\" (UID: \"4083d77c-9e40-4bae-a220-4033bf795a30\") "
Sep 18 20:30:17.602591 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:17.602542    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4083d77c-9e40-4bae-a220-4033bf795a30-istiod-ca-cert\") pod \"4083d77c-9e40-4bae-a220-4033bf795a30\" (UID: \"4083d77c-9e40-4bae-a220-4033bf795a30\") "
Sep 18 20:30:17.602591 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:17.602575    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4083d77c-9e40-4bae-a220-4033bf795a30-credential-socket\") pod \"4083d77c-9e40-4bae-a220-4033bf795a30\" (UID: \"4083d77c-9e40-4bae-a220-4033bf795a30\") "
Sep 18 20:30:17.602885 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:17.602650    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4083d77c-9e40-4bae-a220-4033bf795a30-istio-data\") pod \"4083d77c-9e40-4bae-a220-4033bf795a30\" (UID: \"4083d77c-9e40-4bae-a220-4033bf795a30\") "
Sep 18 20:30:17.602885 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:17.602693    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zh96q\" (UniqueName: \"kubernetes.io/projected/4083d77c-9e40-4bae-a220-4033bf795a30-kube-api-access-zh96q\") pod \"4083d77c-9e40-4bae-a220-4033bf795a30\" (UID: \"4083d77c-9e40-4bae-a220-4033bf795a30\") "
Sep 18 20:30:17.602885 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:17.602750    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4083d77c-9e40-4bae-a220-4033bf795a30-istio-token\") pod \"4083d77c-9e40-4bae-a220-4033bf795a30\" (UID: \"4083d77c-9e40-4bae-a220-4033bf795a30\") "
Sep 18 20:30:17.602885 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:17.602786    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4083d77c-9e40-4bae-a220-4033bf795a30-istio-podinfo\") pod \"4083d77c-9e40-4bae-a220-4033bf795a30\" (UID: \"4083d77c-9e40-4bae-a220-4033bf795a30\") "
Sep 18 20:30:17.602885 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:17.602829    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4083d77c-9e40-4bae-a220-4033bf795a30-istio-envoy\") pod \"4083d77c-9e40-4bae-a220-4033bf795a30\" (UID: \"4083d77c-9e40-4bae-a220-4033bf795a30\") "
Sep 18 20:30:17.602885 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:17.602834    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4083d77c-9e40-4bae-a220-4033bf795a30-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4083d77c-9e40-4bae-a220-4033bf795a30" (UID: "4083d77c-9e40-4bae-a220-4033bf795a30"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:30:17.602885 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:17.602865    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4083d77c-9e40-4bae-a220-4033bf795a30-workload-certs\") pod \"4083d77c-9e40-4bae-a220-4033bf795a30\" (UID: \"4083d77c-9e40-4bae-a220-4033bf795a30\") "
Sep 18 20:30:17.603246 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:17.602937    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4083d77c-9e40-4bae-a220-4033bf795a30-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4083d77c-9e40-4bae-a220-4033bf795a30" (UID: "4083d77c-9e40-4bae-a220-4033bf795a30"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:30:17.603246 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:17.602939    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4083d77c-9e40-4bae-a220-4033bf795a30-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4083d77c-9e40-4bae-a220-4033bf795a30" (UID: "4083d77c-9e40-4bae-a220-4033bf795a30"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:30:17.603246 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:17.603240    2549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4083d77c-9e40-4bae-a220-4033bf795a30-workload-socket\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:30:17.603451 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:17.603260    2549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4083d77c-9e40-4bae-a220-4033bf795a30-istiod-ca-cert\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:30:17.603451 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:17.603274    2549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4083d77c-9e40-4bae-a220-4033bf795a30-credential-socket\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:30:17.603451 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:17.603323    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4083d77c-9e40-4bae-a220-4033bf795a30-istio-data" (OuterVolumeSpecName: "istio-data") pod "4083d77c-9e40-4bae-a220-4033bf795a30" (UID: "4083d77c-9e40-4bae-a220-4033bf795a30"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:30:17.603613 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:17.603556    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4083d77c-9e40-4bae-a220-4033bf795a30-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4083d77c-9e40-4bae-a220-4033bf795a30" (UID: "4083d77c-9e40-4bae-a220-4033bf795a30"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:30:17.605321 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:17.605271    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4083d77c-9e40-4bae-a220-4033bf795a30-istio-token" (OuterVolumeSpecName: "istio-token") pod "4083d77c-9e40-4bae-a220-4033bf795a30" (UID: "4083d77c-9e40-4bae-a220-4033bf795a30"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:30:17.605628 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:17.605430    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4083d77c-9e40-4bae-a220-4033bf795a30-kube-api-access-zh96q" (OuterVolumeSpecName: "kube-api-access-zh96q") pod "4083d77c-9e40-4bae-a220-4033bf795a30" (UID: "4083d77c-9e40-4bae-a220-4033bf795a30"). InnerVolumeSpecName "kube-api-access-zh96q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:30:17.605628 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:17.605578    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4083d77c-9e40-4bae-a220-4033bf795a30-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4083d77c-9e40-4bae-a220-4033bf795a30" (UID: "4083d77c-9e40-4bae-a220-4033bf795a30"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 20:30:17.605775 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:17.605731    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4083d77c-9e40-4bae-a220-4033bf795a30-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4083d77c-9e40-4bae-a220-4033bf795a30" (UID: "4083d77c-9e40-4bae-a220-4033bf795a30"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:30:17.704614 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:17.704574    2549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4083d77c-9e40-4bae-a220-4033bf795a30-istio-data\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:30:17.704614 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:17.704607    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zh96q\" (UniqueName: \"kubernetes.io/projected/4083d77c-9e40-4bae-a220-4033bf795a30-kube-api-access-zh96q\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:30:17.704614 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:17.704617    2549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4083d77c-9e40-4bae-a220-4033bf795a30-istio-token\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:30:17.704614 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:17.704626    2549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4083d77c-9e40-4bae-a220-4033bf795a30-istio-podinfo\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:30:17.704996 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:17.704635    2549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4083d77c-9e40-4bae-a220-4033bf795a30-istio-envoy\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:30:17.704996 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:17.704645    2549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4083d77c-9e40-4bae-a220-4033bf795a30-workload-certs\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:30:17.789502 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:17.789463    2549 generic.go:356] "Generic (PLEG): container finished" podID="4083d77c-9e40-4bae-a220-4033bf795a30" containerID="7dd3c50f1c86cf21b637bfe2f51730fa8ae978695f4bf9adf7299c196ef36353" exitCode=0
Sep 18 20:30:17.789704 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:17.789518    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg"
Sep 18 20:30:17.789704 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:17.789551    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg" event={"ID":"4083d77c-9e40-4bae-a220-4033bf795a30","Type":"ContainerDied","Data":"7dd3c50f1c86cf21b637bfe2f51730fa8ae978695f4bf9adf7299c196ef36353"}
Sep 18 20:30:17.789704 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:17.789609    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg" event={"ID":"4083d77c-9e40-4bae-a220-4033bf795a30","Type":"ContainerDied","Data":"5d922f9ea1bf65845ba0f6319e2fe7e5a5e9641e8f464ff750a33214a61e8512"}
Sep 18 20:30:17.789704 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:17.789639    2549 scope.go:117] "RemoveContainer" containerID="7dd3c50f1c86cf21b637bfe2f51730fa8ae978695f4bf9adf7299c196ef36353"
Sep 18 20:30:17.791393 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:17.791257    2549 generic.go:356] "Generic (PLEG): container finished" podID="b0aad8fa-9518-4ed0-97fe-a03802e278a7" containerID="5c46487b1532a9e8d2db6363d663a1e35973c9eb9b0d93ac25d474b395cc4b6a" exitCode=0
Sep 18 20:30:17.791393 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:17.791313    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-de7ea3-ttsjp" event={"ID":"b0aad8fa-9518-4ed0-97fe-a03802e278a7","Type":"ContainerDied","Data":"5c46487b1532a9e8d2db6363d663a1e35973c9eb9b0d93ac25d474b395cc4b6a"}
Sep 18 20:30:17.800918 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:17.800896    2549 scope.go:117] "RemoveContainer" containerID="d63269325b9fedc92500c925bb09f652af1a197d8627211cf231e4995d0fd6b5"
Sep 18 20:30:17.810704 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:17.810680    2549 scope.go:117] "RemoveContainer" containerID="7dd3c50f1c86cf21b637bfe2f51730fa8ae978695f4bf9adf7299c196ef36353"
Sep 18 20:30:17.811081 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:30:17.811060    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dd3c50f1c86cf21b637bfe2f51730fa8ae978695f4bf9adf7299c196ef36353\": container with ID starting with 7dd3c50f1c86cf21b637bfe2f51730fa8ae978695f4bf9adf7299c196ef36353 not found: ID does not exist" containerID="7dd3c50f1c86cf21b637bfe2f51730fa8ae978695f4bf9adf7299c196ef36353"
Sep 18 20:30:17.811136 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:17.811093    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dd3c50f1c86cf21b637bfe2f51730fa8ae978695f4bf9adf7299c196ef36353"} err="failed to get container status \"7dd3c50f1c86cf21b637bfe2f51730fa8ae978695f4bf9adf7299c196ef36353\": rpc error: code = NotFound desc = could not find container \"7dd3c50f1c86cf21b637bfe2f51730fa8ae978695f4bf9adf7299c196ef36353\": container with ID starting with 7dd3c50f1c86cf21b637bfe2f51730fa8ae978695f4bf9adf7299c196ef36353 not found: ID does not exist"
Sep 18 20:30:17.811136 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:17.811113    2549 scope.go:117] "RemoveContainer" containerID="d63269325b9fedc92500c925bb09f652af1a197d8627211cf231e4995d0fd6b5"
Sep 18 20:30:17.811431 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:30:17.811408    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d63269325b9fedc92500c925bb09f652af1a197d8627211cf231e4995d0fd6b5\": container with ID starting with d63269325b9fedc92500c925bb09f652af1a197d8627211cf231e4995d0fd6b5 not found: ID does not exist" containerID="d63269325b9fedc92500c925bb09f652af1a197d8627211cf231e4995d0fd6b5"
Sep 18 20:30:17.811486 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:17.811441    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d63269325b9fedc92500c925bb09f652af1a197d8627211cf231e4995d0fd6b5"} err="failed to get container status \"d63269325b9fedc92500c925bb09f652af1a197d8627211cf231e4995d0fd6b5\": rpc error: code = NotFound desc = could not find container \"d63269325b9fedc92500c925bb09f652af1a197d8627211cf231e4995d0fd6b5\": container with ID starting with d63269325b9fedc92500c925bb09f652af1a197d8627211cf231e4995d0fd6b5 not found: ID does not exist"
Sep 18 20:30:17.831318 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:17.831269    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg"]
Sep 18 20:30:17.835168 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:17.835130    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-876382-openshift-default-854df5b556-x2phg"]
Sep 18 20:30:18.952127 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:18.952087    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-de7ea3-ttsjp"
Sep 18 20:30:19.019183 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:19.019138    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0aad8fa-9518-4ed0-97fe-a03802e278a7-maas-api-service-ca\") pod \"b0aad8fa-9518-4ed0-97fe-a03802e278a7\" (UID: \"b0aad8fa-9518-4ed0-97fe-a03802e278a7\") "
Sep 18 20:30:19.019618 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:19.019592    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0aad8fa-9518-4ed0-97fe-a03802e278a7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b0aad8fa-9518-4ed0-97fe-a03802e278a7" (UID: "b0aad8fa-9518-4ed0-97fe-a03802e278a7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:30:19.120496 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:19.120446    2549 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0aad8fa-9518-4ed0-97fe-a03802e278a7-maas-api-service-ca\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:30:19.522096 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:19.522009    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4083d77c-9e40-4bae-a220-4033bf795a30" path="/var/lib/kubelet/pods/4083d77c-9e40-4bae-a220-4033bf795a30/volumes"
Sep 18 20:30:19.806302 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:19.806207    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-de7ea3-ttsjp"
Sep 18 20:30:19.806302 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:19.806241    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-de7ea3-ttsjp" event={"ID":"b0aad8fa-9518-4ed0-97fe-a03802e278a7","Type":"ContainerDied","Data":"f24c199ef8269081d8c199ec6e3b49cae7c3c73c09caf902044d7f16d77aec4f"}
Sep 18 20:30:19.806302 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:19.806294    2549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f24c199ef8269081d8c199ec6e3b49cae7c3c73c09caf902044d7f16d77aec4f"
Sep 18 20:30:29.726340 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:29.726294    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:30:31.675360 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:31.675315    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-de7ea3-29829390-zkr9t"]
Sep 18 20:30:31.679827 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:31.679793    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-de7ea3-29829390-zkr9t"]
Sep 18 20:30:32.034839 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:32.034739    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-de7ea3-ttsjp"]
Sep 18 20:30:32.041014 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:32.040976    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-de7ea3-ttsjp"]
Sep 18 20:30:33.521803 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:33.521765    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3350e426-add0-4440-8858-98b4ec1715da" path="/var/lib/kubelet/pods/3350e426-add0-4440-8858-98b4ec1715da/volumes"
Sep 18 20:30:33.522219 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:33.522150    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0aad8fa-9518-4ed0-97fe-a03802e278a7" path="/var/lib/kubelet/pods/b0aad8fa-9518-4ed0-97fe-a03802e278a7/volumes"
Sep 18 20:30:37.375416 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:37.375357    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-de7ea3/ipp-route-de7ea3-kserve-55cd5894c-7n9jh"]
Sep 18 20:30:42.421591 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:42.421552    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:30:52.396411 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:52.396332    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-de7ea3/ipp-route-de7ea3-kserve-55cd5894c-7n9jh" podUID="8a0bd34b-b468-4c50-8e64-1835907cc85a" containerName="main" containerID="cri-o://25ac30c6c5f09e75a5b105080583f07ed2f96ad1f5247a3e6b81d400efca11ce" gracePeriod=45
Sep 18 20:30:52.660537 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:52.660464    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-de7ea3/ipp-route-de7ea3-kserve-55cd5894c-7n9jh"
Sep 18 20:30:52.763607 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:52.763565    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a0bd34b-b468-4c50-8e64-1835907cc85a-tmp-dir\") pod \"8a0bd34b-b468-4c50-8e64-1835907cc85a\" (UID: \"8a0bd34b-b468-4c50-8e64-1835907cc85a\") "
Sep 18 20:30:52.763822 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:52.763665    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a0bd34b-b468-4c50-8e64-1835907cc85a-tls-certs\") pod \"8a0bd34b-b468-4c50-8e64-1835907cc85a\" (UID: \"8a0bd34b-b468-4c50-8e64-1835907cc85a\") "
Sep 18 20:30:52.763822 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:52.763687    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a0bd34b-b468-4c50-8e64-1835907cc85a-home\") pod \"8a0bd34b-b468-4c50-8e64-1835907cc85a\" (UID: \"8a0bd34b-b468-4c50-8e64-1835907cc85a\") "
Sep 18 20:30:52.763822 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:52.763713    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a0bd34b-b468-4c50-8e64-1835907cc85a-dshm\") pod \"8a0bd34b-b468-4c50-8e64-1835907cc85a\" (UID: \"8a0bd34b-b468-4c50-8e64-1835907cc85a\") "
Sep 18 20:30:52.763822 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:52.763734    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a0bd34b-b468-4c50-8e64-1835907cc85a-model-cache\") pod \"8a0bd34b-b468-4c50-8e64-1835907cc85a\" (UID: \"8a0bd34b-b468-4c50-8e64-1835907cc85a\") "
Sep 18 20:30:52.763822 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:52.763765    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d88zr\" (UniqueName: \"kubernetes.io/projected/8a0bd34b-b468-4c50-8e64-1835907cc85a-kube-api-access-d88zr\") pod \"8a0bd34b-b468-4c50-8e64-1835907cc85a\" (UID: \"8a0bd34b-b468-4c50-8e64-1835907cc85a\") "
Sep 18 20:30:52.764100 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:52.763885    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a0bd34b-b468-4c50-8e64-1835907cc85a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8a0bd34b-b468-4c50-8e64-1835907cc85a" (UID: "8a0bd34b-b468-4c50-8e64-1835907cc85a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:30:52.764100 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:52.764010    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a0bd34b-b468-4c50-8e64-1835907cc85a-home" (OuterVolumeSpecName: "home") pod "8a0bd34b-b468-4c50-8e64-1835907cc85a" (UID: "8a0bd34b-b468-4c50-8e64-1835907cc85a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:30:52.764100 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:52.764035    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a0bd34b-b468-4c50-8e64-1835907cc85a-model-cache" (OuterVolumeSpecName: "model-cache") pod "8a0bd34b-b468-4c50-8e64-1835907cc85a" (UID: "8a0bd34b-b468-4c50-8e64-1835907cc85a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:30:52.764255 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:52.764108    2549 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a0bd34b-b468-4c50-8e64-1835907cc85a-home\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:30:52.764255 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:52.764121    2549 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a0bd34b-b468-4c50-8e64-1835907cc85a-model-cache\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:30:52.764255 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:52.764132    2549 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a0bd34b-b468-4c50-8e64-1835907cc85a-tmp-dir\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:30:52.766231 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:52.766204    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a0bd34b-b468-4c50-8e64-1835907cc85a-dshm" (OuterVolumeSpecName: "dshm") pod "8a0bd34b-b468-4c50-8e64-1835907cc85a" (UID: "8a0bd34b-b468-4c50-8e64-1835907cc85a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:30:52.766231 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:52.766212    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a0bd34b-b468-4c50-8e64-1835907cc85a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8a0bd34b-b468-4c50-8e64-1835907cc85a" (UID: "8a0bd34b-b468-4c50-8e64-1835907cc85a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:30:52.766430 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:52.766208    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a0bd34b-b468-4c50-8e64-1835907cc85a-kube-api-access-d88zr" (OuterVolumeSpecName: "kube-api-access-d88zr") pod "8a0bd34b-b468-4c50-8e64-1835907cc85a" (UID: "8a0bd34b-b468-4c50-8e64-1835907cc85a"). InnerVolumeSpecName "kube-api-access-d88zr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:30:52.865680 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:52.865635    2549 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a0bd34b-b468-4c50-8e64-1835907cc85a-tls-certs\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:30:52.865680 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:52.865670    2549 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a0bd34b-b468-4c50-8e64-1835907cc85a-dshm\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:30:52.865680 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:52.865679    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d88zr\" (UniqueName: \"kubernetes.io/projected/8a0bd34b-b468-4c50-8e64-1835907cc85a-kube-api-access-d88zr\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:30:53.017777 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:53.017670    2549 generic.go:356] "Generic (PLEG): container finished" podID="8a0bd34b-b468-4c50-8e64-1835907cc85a" containerID="25ac30c6c5f09e75a5b105080583f07ed2f96ad1f5247a3e6b81d400efca11ce" exitCode=0
Sep 18 20:30:53.017777 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:53.017717    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-de7ea3/ipp-route-de7ea3-kserve-55cd5894c-7n9jh"
Sep 18 20:30:53.017777 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:53.017749    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-de7ea3/ipp-route-de7ea3-kserve-55cd5894c-7n9jh" event={"ID":"8a0bd34b-b468-4c50-8e64-1835907cc85a","Type":"ContainerDied","Data":"25ac30c6c5f09e75a5b105080583f07ed2f96ad1f5247a3e6b81d400efca11ce"}
Sep 18 20:30:53.018079 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:53.017794    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-de7ea3/ipp-route-de7ea3-kserve-55cd5894c-7n9jh" event={"ID":"8a0bd34b-b468-4c50-8e64-1835907cc85a","Type":"ContainerDied","Data":"c0b43a14eea244002f7ca93d6f67d5a2823b43ffe99cf7c7f5f10a52f6201149"}
Sep 18 20:30:53.018079 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:53.017813    2549 scope.go:117] "RemoveContainer" containerID="25ac30c6c5f09e75a5b105080583f07ed2f96ad1f5247a3e6b81d400efca11ce"
Sep 18 20:30:53.032134 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:53.032110    2549 scope.go:117] "RemoveContainer" containerID="25ac30c6c5f09e75a5b105080583f07ed2f96ad1f5247a3e6b81d400efca11ce"
Sep 18 20:30:53.032525 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:30:53.032501    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25ac30c6c5f09e75a5b105080583f07ed2f96ad1f5247a3e6b81d400efca11ce\": container with ID starting with 25ac30c6c5f09e75a5b105080583f07ed2f96ad1f5247a3e6b81d400efca11ce not found: ID does not exist" containerID="25ac30c6c5f09e75a5b105080583f07ed2f96ad1f5247a3e6b81d400efca11ce"
Sep 18 20:30:53.032588 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:53.032537    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25ac30c6c5f09e75a5b105080583f07ed2f96ad1f5247a3e6b81d400efca11ce"} err="failed to get container status \"25ac30c6c5f09e75a5b105080583f07ed2f96ad1f5247a3e6b81d400efca11ce\": rpc error: code = NotFound desc = could not find container \"25ac30c6c5f09e75a5b105080583f07ed2f96ad1f5247a3e6b81d400efca11ce\": container with ID starting with 25ac30c6c5f09e75a5b105080583f07ed2f96ad1f5247a3e6b81d400efca11ce not found: ID does not exist"
Sep 18 20:30:53.042734 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:53.042699    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-de7ea3/ipp-route-de7ea3-kserve-55cd5894c-7n9jh"]
Sep 18 20:30:53.047604 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:53.047571    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-de7ea3/ipp-route-de7ea3-kserve-55cd5894c-7n9jh"]
Sep 18 20:30:53.528705 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:53.528592    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a0bd34b-b468-4c50-8e64-1835907cc85a" path="/var/lib/kubelet/pods/8a0bd34b-b468-4c50-8e64-1835907cc85a/volumes"
Sep 18 20:30:56.076030 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:30:56.075995    2549 scope.go:117] "RemoveContainer" containerID="feedefd960317cfc09be5828022e273cf18cf249912685fddb7d5206d29ae5c3"
Sep 18 20:31:21.623571 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:21.623482    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd"]
Sep 18 20:31:21.623977 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:21.623732    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd" podUID="e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd" containerName="istio-proxy" containerID="cri-o://64a90dfd3c7153b493b837a618051e812015a4e99a1e849289dbebb4d7e4637a" gracePeriod=30
Sep 18 20:31:21.928594 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:21.928493    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a1a0a2-nc2fd"]
Sep 18 20:31:21.928933 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:21.928920    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4083d77c-9e40-4bae-a220-4033bf795a30" containerName="istio-proxy"
Sep 18 20:31:21.928980 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:21.928935    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="4083d77c-9e40-4bae-a220-4033bf795a30" containerName="istio-proxy"
Sep 18 20:31:21.928980 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:21.928946    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4083d77c-9e40-4bae-a220-4033bf795a30" containerName="istio-proxy"
Sep 18 20:31:21.928980 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:21.928952    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="4083d77c-9e40-4bae-a220-4033bf795a30" containerName="istio-proxy"
Sep 18 20:31:21.928980 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:21.928968    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a0bd34b-b468-4c50-8e64-1835907cc85a" containerName="main"
Sep 18 20:31:21.928980 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:21.928974    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a0bd34b-b468-4c50-8e64-1835907cc85a" containerName="main"
Sep 18 20:31:21.929153 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:21.928986    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0aad8fa-9518-4ed0-97fe-a03802e278a7" containerName="revoke-keys"
Sep 18 20:31:21.929153 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:21.928992    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0aad8fa-9518-4ed0-97fe-a03802e278a7" containerName="revoke-keys"
Sep 18 20:31:21.929153 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:21.929059    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0aad8fa-9518-4ed0-97fe-a03802e278a7" containerName="revoke-keys"
Sep 18 20:31:21.929153 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:21.929069    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a0bd34b-b468-4c50-8e64-1835907cc85a" containerName="main"
Sep 18 20:31:21.929153 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:21.929077    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="4083d77c-9e40-4bae-a220-4033bf795a30" containerName="istio-proxy"
Sep 18 20:31:21.929153 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:21.929083    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="4083d77c-9e40-4bae-a220-4033bf795a30" containerName="istio-proxy"
Sep 18 20:31:21.932438 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:21.932418    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a1a0a2-nc2fd"
Sep 18 20:31:21.937948 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:21.937925    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bhssz\""
Sep 18 20:31:21.938091 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:21.937983    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 20:31:21.945340 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:21.945302    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a1a0a2-nc2fd"]
Sep 18 20:31:22.065242 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:22.065195    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8057e225-f0f1-4b68-a99c-bfcdaaa3e4da-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a1a0a2-nc2fd\" (UID: \"8057e225-f0f1-4b68-a99c-bfcdaaa3e4da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a1a0a2-nc2fd"
Sep 18 20:31:22.166343 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:22.166283    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8057e225-f0f1-4b68-a99c-bfcdaaa3e4da-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a1a0a2-nc2fd\" (UID: \"8057e225-f0f1-4b68-a99c-bfcdaaa3e4da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a1a0a2-nc2fd"
Sep 18 20:31:22.167021 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:22.166997    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8057e225-f0f1-4b68-a99c-bfcdaaa3e4da-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a1a0a2-nc2fd\" (UID: \"8057e225-f0f1-4b68-a99c-bfcdaaa3e4da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a1a0a2-nc2fd"
Sep 18 20:31:22.244507 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:22.244409    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a1a0a2-nc2fd"
Sep 18 20:31:22.527097 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:22.527061    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a1a0a2-nc2fd"]
Sep 18 20:31:22.528227 ip-10-0-140-178 kubenswrapper[2549]: W0918 20:31:22.528197    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8057e225_f0f1_4b68_a99c_bfcdaaa3e4da.slice/crio-9894e01f187ac88802ee6494535104b220eb7d3142896593d10e991b146b0160 WatchSource:0}: Error finding container 9894e01f187ac88802ee6494535104b220eb7d3142896593d10e991b146b0160: Status 404 returned error can't find the container with id 9894e01f187ac88802ee6494535104b220eb7d3142896593d10e991b146b0160
Sep 18 20:31:23.206050 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:23.205938    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a1a0a2-nc2fd" event={"ID":"8057e225-f0f1-4b68-a99c-bfcdaaa3e4da","Type":"ContainerStarted","Data":"27d663eb173aeffaa2b447b7e2f868a6920fa84e7ee61ab7a201a0687b54111c"}
Sep 18 20:31:23.206050 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:23.206008    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a1a0a2-nc2fd" event={"ID":"8057e225-f0f1-4b68-a99c-bfcdaaa3e4da","Type":"ContainerStarted","Data":"9894e01f187ac88802ee6494535104b220eb7d3142896593d10e991b146b0160"}
Sep 18 20:31:23.228178 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:23.228122    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a1a0a2-nc2fd" podStartSLOduration=2.228102896 podStartE2EDuration="2.228102896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:31:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:31:23.225054594 +0000 UTC m=+2068.178795762" watchObservedRunningTime="2026-09-18 20:31:23.228102896 +0000 UTC m=+2068.181844064"
Sep 18 20:31:24.213860 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:24.213824    2549 generic.go:356] "Generic (PLEG): container finished" podID="8057e225-f0f1-4b68-a99c-bfcdaaa3e4da" containerID="27d663eb173aeffaa2b447b7e2f868a6920fa84e7ee61ab7a201a0687b54111c" exitCode=0
Sep 18 20:31:24.214324 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:24.213908    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a1a0a2-nc2fd" event={"ID":"8057e225-f0f1-4b68-a99c-bfcdaaa3e4da","Type":"ContainerDied","Data":"27d663eb173aeffaa2b447b7e2f868a6920fa84e7ee61ab7a201a0687b54111c"}
Sep 18 20:31:25.365009 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:25.364976    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a1a0a2-nc2fd"
Sep 18 20:31:25.397212 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:25.397169    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8057e225-f0f1-4b68-a99c-bfcdaaa3e4da-maas-api-service-ca\") pod \"8057e225-f0f1-4b68-a99c-bfcdaaa3e4da\" (UID: \"8057e225-f0f1-4b68-a99c-bfcdaaa3e4da\") "
Sep 18 20:31:25.397533 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:25.397511    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8057e225-f0f1-4b68-a99c-bfcdaaa3e4da-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8057e225-f0f1-4b68-a99c-bfcdaaa3e4da" (UID: "8057e225-f0f1-4b68-a99c-bfcdaaa3e4da"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:31:25.498301 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:25.498185    2549 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8057e225-f0f1-4b68-a99c-bfcdaaa3e4da-maas-api-service-ca\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:31:26.233622 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:26.233592    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a1a0a2-nc2fd"
Sep 18 20:31:26.233622 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:26.233611    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a1a0a2-nc2fd" event={"ID":"8057e225-f0f1-4b68-a99c-bfcdaaa3e4da","Type":"ContainerDied","Data":"9894e01f187ac88802ee6494535104b220eb7d3142896593d10e991b146b0160"}
Sep 18 20:31:26.233849 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:26.233668    2549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9894e01f187ac88802ee6494535104b220eb7d3142896593d10e991b146b0160"
Sep 18 20:31:26.280557 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:26.280505    2549 patch_prober.go:28] interesting pod/e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 18 20:31:26.280753 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:26.280582    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd" podUID="e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:31:26.894007 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:26.893917    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd"
Sep 18 20:31:26.908657 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:26.908629    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd-istiod-ca-cert\") pod \"e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd\" (UID: \"e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd\") "
Sep 18 20:31:26.908845 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:26.908673    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd-istio-podinfo\") pod \"e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd\" (UID: \"e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd\") "
Sep 18 20:31:26.908845 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:26.908724    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd-istio-token\") pod \"e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd\" (UID: \"e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd\") "
Sep 18 20:31:26.908845 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:26.908754    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd-credential-socket\") pod \"e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd\" (UID: \"e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd\") "
Sep 18 20:31:26.909008 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:26.908892    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd-istio-data\") pod \"e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd\" (UID: \"e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd\") "
Sep 18 20:31:26.909008 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:26.908928    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd-workload-socket\") pod \"e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd\" (UID: \"e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd\") "
Sep 18 20:31:26.909008 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:26.908965    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd-workload-certs\") pod \"e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd\" (UID: \"e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd\") "
Sep 18 20:31:26.909169 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:26.909039    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd-istio-envoy\") pod \"e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd\" (UID: \"e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd\") "
Sep 18 20:31:26.909169 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:26.909083    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd" (UID: "e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:31:26.909169 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:26.909105    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vhwrn\" (UniqueName: \"kubernetes.io/projected/e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd-kube-api-access-vhwrn\") pod \"e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd\" (UID: \"e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd\") "
Sep 18 20:31:26.909343 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:26.909174    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd" (UID: "e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:31:26.909343 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:26.909174    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd-istio-data" (OuterVolumeSpecName: "istio-data") pod "e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd" (UID: "e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:31:26.909469 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:26.909396    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd" (UID: "e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:31:26.909529 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:26.909502    2549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd-istiod-ca-cert\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:31:26.909529 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:26.909521    2549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd-credential-socket\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:31:26.909635 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:26.909536    2549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd-istio-data\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:31:26.909635 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:26.909548    2549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd-workload-certs\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:31:26.909635 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:26.909541    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd" (UID: "e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:31:26.911256 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:26.911222    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd-istio-token" (OuterVolumeSpecName: "istio-token") pod "e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd" (UID: "e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:31:26.911396 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:26.911262    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd" (UID: "e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 20:31:26.911396 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:26.911269    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd-kube-api-access-vhwrn" (OuterVolumeSpecName: "kube-api-access-vhwrn") pod "e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd" (UID: "e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd"). InnerVolumeSpecName "kube-api-access-vhwrn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:31:26.911396 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:26.911323    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd" (UID: "e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:31:27.010251 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:27.010206    2549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd-istio-podinfo\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:31:27.010251 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:27.010244    2549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd-istio-token\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:31:27.010251 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:27.010255    2549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd-workload-socket\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:31:27.010251 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:27.010264    2549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd-istio-envoy\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:31:27.010586 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:27.010273    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhwrn\" (UniqueName: \"kubernetes.io/projected/e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd-kube-api-access-vhwrn\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:31:27.241578 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:27.241489    2549 generic.go:356] "Generic (PLEG): container finished" podID="e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd" containerID="64a90dfd3c7153b493b837a618051e812015a4e99a1e849289dbebb4d7e4637a" exitCode=0
Sep 18 20:31:27.241578 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:27.241544    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd"
Sep 18 20:31:27.241578 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:27.241570    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd" event={"ID":"e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd","Type":"ContainerDied","Data":"64a90dfd3c7153b493b837a618051e812015a4e99a1e849289dbebb4d7e4637a"}
Sep 18 20:31:27.241837 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:27.241627    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd" event={"ID":"e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd","Type":"ContainerDied","Data":"5f202ff8dc7b7b147406d9ee2ffc1dda192d8b5788aba837d286c4eb1d78bd8c"}
Sep 18 20:31:27.241837 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:27.241646    2549 scope.go:117] "RemoveContainer" containerID="64a90dfd3c7153b493b837a618051e812015a4e99a1e849289dbebb4d7e4637a"
Sep 18 20:31:27.252777 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:27.252758    2549 scope.go:117] "RemoveContainer" containerID="0cff3f80ee7b26e4a5d0965f6a21c9b02ad67077c333be0dcad7edae8d927287"
Sep 18 20:31:27.262220 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:27.262200    2549 scope.go:117] "RemoveContainer" containerID="64a90dfd3c7153b493b837a618051e812015a4e99a1e849289dbebb4d7e4637a"
Sep 18 20:31:27.262609 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:31:27.262586    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64a90dfd3c7153b493b837a618051e812015a4e99a1e849289dbebb4d7e4637a\": container with ID starting with 64a90dfd3c7153b493b837a618051e812015a4e99a1e849289dbebb4d7e4637a not found: ID does not exist" containerID="64a90dfd3c7153b493b837a618051e812015a4e99a1e849289dbebb4d7e4637a"
Sep 18 20:31:27.262686 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:27.262620    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64a90dfd3c7153b493b837a618051e812015a4e99a1e849289dbebb4d7e4637a"} err="failed to get container status \"64a90dfd3c7153b493b837a618051e812015a4e99a1e849289dbebb4d7e4637a\": rpc error: code = NotFound desc = could not find container \"64a90dfd3c7153b493b837a618051e812015a4e99a1e849289dbebb4d7e4637a\": container with ID starting with 64a90dfd3c7153b493b837a618051e812015a4e99a1e849289dbebb4d7e4637a not found: ID does not exist"
Sep 18 20:31:27.262686 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:27.262641    2549 scope.go:117] "RemoveContainer" containerID="0cff3f80ee7b26e4a5d0965f6a21c9b02ad67077c333be0dcad7edae8d927287"
Sep 18 20:31:27.262910 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:31:27.262897    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cff3f80ee7b26e4a5d0965f6a21c9b02ad67077c333be0dcad7edae8d927287\": container with ID starting with 0cff3f80ee7b26e4a5d0965f6a21c9b02ad67077c333be0dcad7edae8d927287 not found: ID does not exist" containerID="0cff3f80ee7b26e4a5d0965f6a21c9b02ad67077c333be0dcad7edae8d927287"
Sep 18 20:31:27.262948 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:27.262914    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cff3f80ee7b26e4a5d0965f6a21c9b02ad67077c333be0dcad7edae8d927287"} err="failed to get container status \"0cff3f80ee7b26e4a5d0965f6a21c9b02ad67077c333be0dcad7edae8d927287\": rpc error: code = NotFound desc = could not find container \"0cff3f80ee7b26e4a5d0965f6a21c9b02ad67077c333be0dcad7edae8d927287\": container with ID starting with 0cff3f80ee7b26e4a5d0965f6a21c9b02ad67077c333be0dcad7edae8d927287 not found: ID does not exist"
Sep 18 20:31:27.266516 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:27.266487    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd"]
Sep 18 20:31:27.274047 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:27.274015    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-de7ea3-openshift-default-6c5cf9d56-vbwbd"]
Sep 18 20:31:27.522273 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:27.522185    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd" path="/var/lib/kubelet/pods/e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd/volumes"
Sep 18 20:31:32.925171 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:32.925115    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a1a0a2-nc2fd"]
Sep 18 20:31:32.930831 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:32.930797    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a1a0a2-nc2fd"]
Sep 18 20:31:33.521555 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:33.521518    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8057e225-f0f1-4b68-a99c-bfcdaaa3e4da" path="/var/lib/kubelet/pods/8057e225-f0f1-4b68-a99c-bfcdaaa3e4da/volumes"
Sep 18 20:31:36.554566 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:36.554520    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:31:40.646996 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:40.646949    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt"]
Sep 18 20:31:40.647516 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:40.647188    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt" podUID="1feabf84-a082-4982-a415-7efde1ee6a29" containerName="istio-proxy" containerID="cri-o://fd9b21f2e25d10145cecbfdabec67b804033906f8f3274319235345c4ca0ec29" gracePeriod=30
Sep 18 20:31:40.972115 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:40.972017    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-40a1a5-k9lc9"]
Sep 18 20:31:40.972484 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:40.972470    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8057e225-f0f1-4b68-a99c-bfcdaaa3e4da" containerName="revoke-keys"
Sep 18 20:31:40.972535 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:40.972486    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="8057e225-f0f1-4b68-a99c-bfcdaaa3e4da" containerName="revoke-keys"
Sep 18 20:31:40.972535 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:40.972500    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd" containerName="istio-proxy"
Sep 18 20:31:40.972535 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:40.972507    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd" containerName="istio-proxy"
Sep 18 20:31:40.972535 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:40.972520    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd" containerName="istio-proxy"
Sep 18 20:31:40.972535 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:40.972526    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd" containerName="istio-proxy"
Sep 18 20:31:40.972693 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:40.972582    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd" containerName="istio-proxy"
Sep 18 20:31:40.972693 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:40.972591    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="8057e225-f0f1-4b68-a99c-bfcdaaa3e4da" containerName="revoke-keys"
Sep 18 20:31:40.972693 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:40.972599    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0cc6625-c6f1-4310-b1ac-3d0eaeae32cd" containerName="istio-proxy"
Sep 18 20:31:40.976133 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:40.976111    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-40a1a5-k9lc9"
Sep 18 20:31:40.980210 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:40.980184    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bhssz\""
Sep 18 20:31:40.980574 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:40.980550    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 20:31:40.986492 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:40.986458    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-40a1a5-k9lc9"]
Sep 18 20:31:41.055482 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:41.055429    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9532d1a0-e1cb-4147-9815-f7e6fc35aac8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-40a1a5-k9lc9\" (UID: \"9532d1a0-e1cb-4147-9815-f7e6fc35aac8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-40a1a5-k9lc9"
Sep 18 20:31:41.156914 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:41.156853    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9532d1a0-e1cb-4147-9815-f7e6fc35aac8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-40a1a5-k9lc9\" (UID: \"9532d1a0-e1cb-4147-9815-f7e6fc35aac8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-40a1a5-k9lc9"
Sep 18 20:31:41.157635 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:41.157614    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9532d1a0-e1cb-4147-9815-f7e6fc35aac8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-40a1a5-k9lc9\" (UID: \"9532d1a0-e1cb-4147-9815-f7e6fc35aac8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-40a1a5-k9lc9"
Sep 18 20:31:41.288948 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:41.288839    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-40a1a5-k9lc9"
Sep 18 20:31:41.585522 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:41.585482    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-40a1a5-k9lc9"]
Sep 18 20:31:41.585833 ip-10-0-140-178 kubenswrapper[2549]: W0918 20:31:41.585809    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9532d1a0_e1cb_4147_9815_f7e6fc35aac8.slice/crio-f3b0a2d54bebfbfad409dc05ffca541ac8faa509f11565b7b50fa0ada6e4120c WatchSource:0}: Error finding container f3b0a2d54bebfbfad409dc05ffca541ac8faa509f11565b7b50fa0ada6e4120c: Status 404 returned error can't find the container with id f3b0a2d54bebfbfad409dc05ffca541ac8faa509f11565b7b50fa0ada6e4120c
Sep 18 20:31:42.346790 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:42.346742    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-40a1a5-k9lc9" event={"ID":"9532d1a0-e1cb-4147-9815-f7e6fc35aac8","Type":"ContainerStarted","Data":"f29e79867f0c452a2f06058827d64b0abd9a39e64f1e1c41fedf07a44ba58968"}
Sep 18 20:31:42.347198 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:42.346812    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-40a1a5-k9lc9" event={"ID":"9532d1a0-e1cb-4147-9815-f7e6fc35aac8","Type":"ContainerStarted","Data":"f3b0a2d54bebfbfad409dc05ffca541ac8faa509f11565b7b50fa0ada6e4120c"}
Sep 18 20:31:42.368231 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:42.368168    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-40a1a5-k9lc9" podStartSLOduration=2.368148171 podStartE2EDuration="2.368148171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:31:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:31:42.366823191 +0000 UTC m=+2087.320564360" watchObservedRunningTime="2026-09-18 20:31:42.368148171 +0000 UTC m=+2087.321889340"
Sep 18 20:31:43.356075 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:43.356033    2549 generic.go:356] "Generic (PLEG): container finished" podID="9532d1a0-e1cb-4147-9815-f7e6fc35aac8" containerID="f29e79867f0c452a2f06058827d64b0abd9a39e64f1e1c41fedf07a44ba58968" exitCode=0
Sep 18 20:31:43.356510 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:43.356123    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-40a1a5-k9lc9" event={"ID":"9532d1a0-e1cb-4147-9815-f7e6fc35aac8","Type":"ContainerDied","Data":"f29e79867f0c452a2f06058827d64b0abd9a39e64f1e1c41fedf07a44ba58968"}
Sep 18 20:31:44.515507 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:44.515479    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-40a1a5-k9lc9"
Sep 18 20:31:44.592004 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:44.591969    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9532d1a0-e1cb-4147-9815-f7e6fc35aac8-maas-api-service-ca\") pod \"9532d1a0-e1cb-4147-9815-f7e6fc35aac8\" (UID: \"9532d1a0-e1cb-4147-9815-f7e6fc35aac8\") "
Sep 18 20:31:44.592347 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:44.592324    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9532d1a0-e1cb-4147-9815-f7e6fc35aac8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9532d1a0-e1cb-4147-9815-f7e6fc35aac8" (UID: "9532d1a0-e1cb-4147-9815-f7e6fc35aac8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:31:44.693624 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:44.693523    2549 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9532d1a0-e1cb-4147-9815-f7e6fc35aac8-maas-api-service-ca\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:31:45.372368 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:45.372335    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-40a1a5-k9lc9"
Sep 18 20:31:45.372593 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:45.372373    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-40a1a5-k9lc9" event={"ID":"9532d1a0-e1cb-4147-9815-f7e6fc35aac8","Type":"ContainerDied","Data":"f3b0a2d54bebfbfad409dc05ffca541ac8faa509f11565b7b50fa0ada6e4120c"}
Sep 18 20:31:45.372593 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:45.372449    2549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3b0a2d54bebfbfad409dc05ffca541ac8faa509f11565b7b50fa0ada6e4120c"
Sep 18 20:31:45.919069 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:45.918988    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt"
Sep 18 20:31:46.007394 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:46.007331    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1feabf84-a082-4982-a415-7efde1ee6a29-workload-certs\") pod \"1feabf84-a082-4982-a415-7efde1ee6a29\" (UID: \"1feabf84-a082-4982-a415-7efde1ee6a29\") "
Sep 18 20:31:46.007584 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:46.007458    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1feabf84-a082-4982-a415-7efde1ee6a29-workload-socket\") pod \"1feabf84-a082-4982-a415-7efde1ee6a29\" (UID: \"1feabf84-a082-4982-a415-7efde1ee6a29\") "
Sep 18 20:31:46.007584 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:46.007505    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5ll42\" (UniqueName: \"kubernetes.io/projected/1feabf84-a082-4982-a415-7efde1ee6a29-kube-api-access-5ll42\") pod \"1feabf84-a082-4982-a415-7efde1ee6a29\" (UID: \"1feabf84-a082-4982-a415-7efde1ee6a29\") "
Sep 18 20:31:46.007584 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:46.007533    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1feabf84-a082-4982-a415-7efde1ee6a29-istio-envoy\") pod \"1feabf84-a082-4982-a415-7efde1ee6a29\" (UID: \"1feabf84-a082-4982-a415-7efde1ee6a29\") "
Sep 18 20:31:46.007584 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:46.007567    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1feabf84-a082-4982-a415-7efde1ee6a29-istio-data\") pod \"1feabf84-a082-4982-a415-7efde1ee6a29\" (UID: \"1feabf84-a082-4982-a415-7efde1ee6a29\") "
Sep 18 20:31:46.007791 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:46.007593    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1feabf84-a082-4982-a415-7efde1ee6a29-istiod-ca-cert\") pod \"1feabf84-a082-4982-a415-7efde1ee6a29\" (UID: \"1feabf84-a082-4982-a415-7efde1ee6a29\") "
Sep 18 20:31:46.007791 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:46.007630    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1feabf84-a082-4982-a415-7efde1ee6a29-istio-podinfo\") pod \"1feabf84-a082-4982-a415-7efde1ee6a29\" (UID: \"1feabf84-a082-4982-a415-7efde1ee6a29\") "
Sep 18 20:31:46.007791 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:46.007663    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1feabf84-a082-4982-a415-7efde1ee6a29-credential-socket\") pod \"1feabf84-a082-4982-a415-7efde1ee6a29\" (UID: \"1feabf84-a082-4982-a415-7efde1ee6a29\") "
Sep 18 20:31:46.007791 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:46.007685    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1feabf84-a082-4982-a415-7efde1ee6a29-istio-token\") pod \"1feabf84-a082-4982-a415-7efde1ee6a29\" (UID: \"1feabf84-a082-4982-a415-7efde1ee6a29\") "
Sep 18 20:31:46.007791 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:46.007702    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1feabf84-a082-4982-a415-7efde1ee6a29-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1feabf84-a082-4982-a415-7efde1ee6a29" (UID: "1feabf84-a082-4982-a415-7efde1ee6a29"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:31:46.008064 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:46.007807    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1feabf84-a082-4982-a415-7efde1ee6a29-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1feabf84-a082-4982-a415-7efde1ee6a29" (UID: "1feabf84-a082-4982-a415-7efde1ee6a29"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:31:46.008064 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:46.007883    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1feabf84-a082-4982-a415-7efde1ee6a29-istio-data" (OuterVolumeSpecName: "istio-data") pod "1feabf84-a082-4982-a415-7efde1ee6a29" (UID: "1feabf84-a082-4982-a415-7efde1ee6a29"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:31:46.008064 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:46.008003    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1feabf84-a082-4982-a415-7efde1ee6a29-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1feabf84-a082-4982-a415-7efde1ee6a29" (UID: "1feabf84-a082-4982-a415-7efde1ee6a29"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:31:46.008228 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:46.008183    2549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1feabf84-a082-4982-a415-7efde1ee6a29-istio-data\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:31:46.008228 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:46.008203    2549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1feabf84-a082-4982-a415-7efde1ee6a29-credential-socket\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:31:46.008228 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:46.008219    2549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1feabf84-a082-4982-a415-7efde1ee6a29-workload-certs\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:31:46.008399 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:46.008224    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1feabf84-a082-4982-a415-7efde1ee6a29-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1feabf84-a082-4982-a415-7efde1ee6a29" (UID: "1feabf84-a082-4982-a415-7efde1ee6a29"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:31:46.008399 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:46.008233    2549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1feabf84-a082-4982-a415-7efde1ee6a29-workload-socket\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:31:46.009990 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:46.009951    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1feabf84-a082-4982-a415-7efde1ee6a29-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1feabf84-a082-4982-a415-7efde1ee6a29" (UID: "1feabf84-a082-4982-a415-7efde1ee6a29"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 20:31:46.010151 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:46.010047    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1feabf84-a082-4982-a415-7efde1ee6a29-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1feabf84-a082-4982-a415-7efde1ee6a29" (UID: "1feabf84-a082-4982-a415-7efde1ee6a29"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:31:46.010292 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:46.010269    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1feabf84-a082-4982-a415-7efde1ee6a29-istio-token" (OuterVolumeSpecName: "istio-token") pod "1feabf84-a082-4982-a415-7efde1ee6a29" (UID: "1feabf84-a082-4982-a415-7efde1ee6a29"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:31:46.010360 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:46.010318    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1feabf84-a082-4982-a415-7efde1ee6a29-kube-api-access-5ll42" (OuterVolumeSpecName: "kube-api-access-5ll42") pod "1feabf84-a082-4982-a415-7efde1ee6a29" (UID: "1feabf84-a082-4982-a415-7efde1ee6a29"). InnerVolumeSpecName "kube-api-access-5ll42". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:31:46.109875 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:46.109831    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ll42\" (UniqueName: \"kubernetes.io/projected/1feabf84-a082-4982-a415-7efde1ee6a29-kube-api-access-5ll42\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:31:46.109875 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:46.109864    2549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1feabf84-a082-4982-a415-7efde1ee6a29-istio-envoy\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:31:46.109875 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:46.109874    2549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1feabf84-a082-4982-a415-7efde1ee6a29-istiod-ca-cert\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:31:46.109875 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:46.109883    2549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1feabf84-a082-4982-a415-7efde1ee6a29-istio-podinfo\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:31:46.110211 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:46.109892    2549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1feabf84-a082-4982-a415-7efde1ee6a29-istio-token\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:31:46.380292 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:46.380253    2549 generic.go:356] "Generic (PLEG): container finished" podID="1feabf84-a082-4982-a415-7efde1ee6a29" containerID="fd9b21f2e25d10145cecbfdabec67b804033906f8f3274319235345c4ca0ec29" exitCode=0
Sep 18 20:31:46.380532 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:46.380317    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt"
Sep 18 20:31:46.380532 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:46.380335    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt" event={"ID":"1feabf84-a082-4982-a415-7efde1ee6a29","Type":"ContainerDied","Data":"fd9b21f2e25d10145cecbfdabec67b804033906f8f3274319235345c4ca0ec29"}
Sep 18 20:31:46.380532 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:46.380444    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt" event={"ID":"1feabf84-a082-4982-a415-7efde1ee6a29","Type":"ContainerDied","Data":"d77c51f46b3641bd314ef7d80b49f999c80fb3c06a6ff6356c35a3fee8c7f8eb"}
Sep 18 20:31:46.380532 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:46.380471    2549 scope.go:117] "RemoveContainer" containerID="fd9b21f2e25d10145cecbfdabec67b804033906f8f3274319235345c4ca0ec29"
Sep 18 20:31:46.391832 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:46.391813    2549 scope.go:117] "RemoveContainer" containerID="600846f036075b4a251bcb5da175a4c5d472ec2224df20962f1d75469b4c21cb"
Sep 18 20:31:46.401509 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:46.401489    2549 scope.go:117] "RemoveContainer" containerID="fd9b21f2e25d10145cecbfdabec67b804033906f8f3274319235345c4ca0ec29"
Sep 18 20:31:46.401858 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:31:46.401836    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd9b21f2e25d10145cecbfdabec67b804033906f8f3274319235345c4ca0ec29\": container with ID starting with fd9b21f2e25d10145cecbfdabec67b804033906f8f3274319235345c4ca0ec29 not found: ID does not exist" containerID="fd9b21f2e25d10145cecbfdabec67b804033906f8f3274319235345c4ca0ec29"
Sep 18 20:31:46.401940 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:46.401876    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd9b21f2e25d10145cecbfdabec67b804033906f8f3274319235345c4ca0ec29"} err="failed to get container status \"fd9b21f2e25d10145cecbfdabec67b804033906f8f3274319235345c4ca0ec29\": rpc error: code = NotFound desc = could not find container \"fd9b21f2e25d10145cecbfdabec67b804033906f8f3274319235345c4ca0ec29\": container with ID starting with fd9b21f2e25d10145cecbfdabec67b804033906f8f3274319235345c4ca0ec29 not found: ID does not exist"
Sep 18 20:31:46.401940 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:46.401904    2549 scope.go:117] "RemoveContainer" containerID="600846f036075b4a251bcb5da175a4c5d472ec2224df20962f1d75469b4c21cb"
Sep 18 20:31:46.402203 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:31:46.402185    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"600846f036075b4a251bcb5da175a4c5d472ec2224df20962f1d75469b4c21cb\": container with ID starting with 600846f036075b4a251bcb5da175a4c5d472ec2224df20962f1d75469b4c21cb not found: ID does not exist" containerID="600846f036075b4a251bcb5da175a4c5d472ec2224df20962f1d75469b4c21cb"
Sep 18 20:31:46.402245 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:46.402212    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"600846f036075b4a251bcb5da175a4c5d472ec2224df20962f1d75469b4c21cb"} err="failed to get container status \"600846f036075b4a251bcb5da175a4c5d472ec2224df20962f1d75469b4c21cb\": rpc error: code = NotFound desc = could not find container \"600846f036075b4a251bcb5da175a4c5d472ec2224df20962f1d75469b4c21cb\": container with ID starting with 600846f036075b4a251bcb5da175a4c5d472ec2224df20962f1d75469b4c21cb not found: ID does not exist"
Sep 18 20:31:46.409086 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:46.409051    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt"]
Sep 18 20:31:46.414226 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:46.414187    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-a1a0a2-openshift-default-5746f6c86d-n7dzt"]
Sep 18 20:31:47.521801 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:47.521766    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1feabf84-a082-4982-a415-7efde1ee6a29" path="/var/lib/kubelet/pods/1feabf84-a082-4982-a415-7efde1ee6a29/volumes"
Sep 18 20:31:52.172136 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:52.172093    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-40a1a5-k9lc9"]
Sep 18 20:31:52.185245 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:52.185206    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-40a1a5-k9lc9"]
Sep 18 20:31:53.521877 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:53.521835    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9532d1a0-e1cb-4147-9815-f7e6fc35aac8" path="/var/lib/kubelet/pods/9532d1a0-e1cb-4147-9815-f7e6fc35aac8/volumes"
Sep 18 20:31:55.139152 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:55.139109    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:31:55.670658 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:55.670624    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-scn7p_a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48/ovn-acl-logging/0.log"
Sep 18 20:31:55.675202 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:55.675171    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-scn7p_a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48/ovn-acl-logging/0.log"
Sep 18 20:31:56.111811 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:56.111778    2549 scope.go:117] "RemoveContainer" containerID="b0bfab383108f8cfa7e6d268d9bc5030a8ea708078b0d4925597cc0bbc7dd862"
Sep 18 20:31:56.120711 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:56.120688    2549 scope.go:117] "RemoveContainer" containerID="9ea1b06fe957dc13b428598f64215b5b50f5c47eec26063276114ec1d69fb279"
Sep 18 20:31:56.129613 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:56.129593    2549 scope.go:117] "RemoveContainer" containerID="87ad04e7a988f6da60ae638f7beacdebc75ef008fc0c7e3d2c5e4c4468543722"
Sep 18 20:31:59.544507 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:59.544472    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9"]
Sep 18 20:31:59.545023 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:31:59.544716    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9" podUID="64206b1e-8c51-4e7f-9bbe-3ea703ccb671" containerName="istio-proxy" containerID="cri-o://13c1df7f2880c2f58f13c9e5f8a31c14bfe147363c4324622c4a0801065f5e2f" gracePeriod=30
Sep 18 20:32:04.318613 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:32:04.318565    2549 patch_prober.go:28] interesting pod/e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 18 20:32:04.319022 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:32:04.318651    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9" podUID="64206b1e-8c51-4e7f-9bbe-3ea703ccb671" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:32:04.809664 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:32:04.809636    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9"
Sep 18 20:32:04.901013 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:32:04.900969    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fkxzj\" (UniqueName: \"kubernetes.io/projected/64206b1e-8c51-4e7f-9bbe-3ea703ccb671-kube-api-access-fkxzj\") pod \"64206b1e-8c51-4e7f-9bbe-3ea703ccb671\" (UID: \"64206b1e-8c51-4e7f-9bbe-3ea703ccb671\") "
Sep 18 20:32:04.901013 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:32:04.901019    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64206b1e-8c51-4e7f-9bbe-3ea703ccb671-credential-socket\") pod \"64206b1e-8c51-4e7f-9bbe-3ea703ccb671\" (UID: \"64206b1e-8c51-4e7f-9bbe-3ea703ccb671\") "
Sep 18 20:32:04.901304 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:32:04.901036    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64206b1e-8c51-4e7f-9bbe-3ea703ccb671-workload-certs\") pod \"64206b1e-8c51-4e7f-9bbe-3ea703ccb671\" (UID: \"64206b1e-8c51-4e7f-9bbe-3ea703ccb671\") "
Sep 18 20:32:04.901304 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:32:04.901073    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64206b1e-8c51-4e7f-9bbe-3ea703ccb671-istiod-ca-cert\") pod \"64206b1e-8c51-4e7f-9bbe-3ea703ccb671\" (UID: \"64206b1e-8c51-4e7f-9bbe-3ea703ccb671\") "
Sep 18 20:32:04.901304 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:32:04.901090    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64206b1e-8c51-4e7f-9bbe-3ea703ccb671-istio-envoy\") pod \"64206b1e-8c51-4e7f-9bbe-3ea703ccb671\" (UID: \"64206b1e-8c51-4e7f-9bbe-3ea703ccb671\") "
Sep 18 20:32:04.901304 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:32:04.901111    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64206b1e-8c51-4e7f-9bbe-3ea703ccb671-istio-data\") pod \"64206b1e-8c51-4e7f-9bbe-3ea703ccb671\" (UID: \"64206b1e-8c51-4e7f-9bbe-3ea703ccb671\") "
Sep 18 20:32:04.901304 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:32:04.901197    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64206b1e-8c51-4e7f-9bbe-3ea703ccb671-istio-podinfo\") pod \"64206b1e-8c51-4e7f-9bbe-3ea703ccb671\" (UID: \"64206b1e-8c51-4e7f-9bbe-3ea703ccb671\") "
Sep 18 20:32:04.901304 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:32:04.901242    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64206b1e-8c51-4e7f-9bbe-3ea703ccb671-workload-socket\") pod \"64206b1e-8c51-4e7f-9bbe-3ea703ccb671\" (UID: \"64206b1e-8c51-4e7f-9bbe-3ea703ccb671\") "
Sep 18 20:32:04.901653 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:32:04.901312    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64206b1e-8c51-4e7f-9bbe-3ea703ccb671-istio-token\") pod \"64206b1e-8c51-4e7f-9bbe-3ea703ccb671\" (UID: \"64206b1e-8c51-4e7f-9bbe-3ea703ccb671\") "
Sep 18 20:32:04.901653 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:32:04.901376    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64206b1e-8c51-4e7f-9bbe-3ea703ccb671-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "64206b1e-8c51-4e7f-9bbe-3ea703ccb671" (UID: "64206b1e-8c51-4e7f-9bbe-3ea703ccb671"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:32:04.901653 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:32:04.901519    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64206b1e-8c51-4e7f-9bbe-3ea703ccb671-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "64206b1e-8c51-4e7f-9bbe-3ea703ccb671" (UID: "64206b1e-8c51-4e7f-9bbe-3ea703ccb671"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:32:04.901653 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:32:04.901606    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64206b1e-8c51-4e7f-9bbe-3ea703ccb671-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "64206b1e-8c51-4e7f-9bbe-3ea703ccb671" (UID: "64206b1e-8c51-4e7f-9bbe-3ea703ccb671"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:32:04.901853 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:32:04.901712    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64206b1e-8c51-4e7f-9bbe-3ea703ccb671-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "64206b1e-8c51-4e7f-9bbe-3ea703ccb671" (UID: "64206b1e-8c51-4e7f-9bbe-3ea703ccb671"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:32:04.901853 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:32:04.901823    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64206b1e-8c51-4e7f-9bbe-3ea703ccb671-istio-data" (OuterVolumeSpecName: "istio-data") pod "64206b1e-8c51-4e7f-9bbe-3ea703ccb671" (UID: "64206b1e-8c51-4e7f-9bbe-3ea703ccb671"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:32:04.901969 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:32:04.901892    2549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64206b1e-8c51-4e7f-9bbe-3ea703ccb671-credential-socket\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:32:04.901969 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:32:04.901914    2549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64206b1e-8c51-4e7f-9bbe-3ea703ccb671-workload-certs\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:32:04.901969 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:32:04.901930    2549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64206b1e-8c51-4e7f-9bbe-3ea703ccb671-istiod-ca-cert\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:32:04.901969 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:32:04.901944    2549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64206b1e-8c51-4e7f-9bbe-3ea703ccb671-workload-socket\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:32:04.903617 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:32:04.903586    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64206b1e-8c51-4e7f-9bbe-3ea703ccb671-istio-token" (OuterVolumeSpecName: "istio-token") pod "64206b1e-8c51-4e7f-9bbe-3ea703ccb671" (UID: "64206b1e-8c51-4e7f-9bbe-3ea703ccb671"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:32:04.903617 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:32:04.903605    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64206b1e-8c51-4e7f-9bbe-3ea703ccb671-kube-api-access-fkxzj" (OuterVolumeSpecName: "kube-api-access-fkxzj") pod "64206b1e-8c51-4e7f-9bbe-3ea703ccb671" (UID: "64206b1e-8c51-4e7f-9bbe-3ea703ccb671"). InnerVolumeSpecName "kube-api-access-fkxzj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:32:04.903869 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:32:04.903844    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64206b1e-8c51-4e7f-9bbe-3ea703ccb671-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "64206b1e-8c51-4e7f-9bbe-3ea703ccb671" (UID: "64206b1e-8c51-4e7f-9bbe-3ea703ccb671"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:32:04.903945 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:32:04.903849    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/64206b1e-8c51-4e7f-9bbe-3ea703ccb671-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "64206b1e-8c51-4e7f-9bbe-3ea703ccb671" (UID: "64206b1e-8c51-4e7f-9bbe-3ea703ccb671"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 20:32:05.002607 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:32:05.002566    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkxzj\" (UniqueName: \"kubernetes.io/projected/64206b1e-8c51-4e7f-9bbe-3ea703ccb671-kube-api-access-fkxzj\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:32:05.002607 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:32:05.002597    2549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64206b1e-8c51-4e7f-9bbe-3ea703ccb671-istio-envoy\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:32:05.002607 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:32:05.002608    2549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64206b1e-8c51-4e7f-9bbe-3ea703ccb671-istio-data\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:32:05.002607 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:32:05.002618    2549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64206b1e-8c51-4e7f-9bbe-3ea703ccb671-istio-podinfo\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:32:05.002886 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:32:05.002625    2549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64206b1e-8c51-4e7f-9bbe-3ea703ccb671-istio-token\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:32:05.500588 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:32:05.500550    2549 generic.go:356] "Generic (PLEG): container finished" podID="64206b1e-8c51-4e7f-9bbe-3ea703ccb671" containerID="13c1df7f2880c2f58f13c9e5f8a31c14bfe147363c4324622c4a0801065f5e2f" exitCode=0
Sep 18 20:32:05.501042 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:32:05.500606    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9"
Sep 18 20:32:05.501042 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:32:05.500646    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9" event={"ID":"64206b1e-8c51-4e7f-9bbe-3ea703ccb671","Type":"ContainerDied","Data":"13c1df7f2880c2f58f13c9e5f8a31c14bfe147363c4324622c4a0801065f5e2f"}
Sep 18 20:32:05.501042 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:32:05.500714    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9" event={"ID":"64206b1e-8c51-4e7f-9bbe-3ea703ccb671","Type":"ContainerDied","Data":"5515b8606888020a9aabea91f687df10cf459c100f43c3bfc9bf9fe891bb7afd"}
Sep 18 20:32:05.501042 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:32:05.500751    2549 scope.go:117] "RemoveContainer" containerID="13c1df7f2880c2f58f13c9e5f8a31c14bfe147363c4324622c4a0801065f5e2f"
Sep 18 20:32:05.510805 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:32:05.510784    2549 scope.go:117] "RemoveContainer" containerID="8b23930ee8074f5d24393a3249ad26e96ba0db34880259b5f3ae9630e9e82bd3"
Sep 18 20:32:05.521514 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:32:05.521465    2549 scope.go:117] "RemoveContainer" containerID="13c1df7f2880c2f58f13c9e5f8a31c14bfe147363c4324622c4a0801065f5e2f"
Sep 18 20:32:05.521798 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:32:05.521778    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13c1df7f2880c2f58f13c9e5f8a31c14bfe147363c4324622c4a0801065f5e2f\": container with ID starting with 13c1df7f2880c2f58f13c9e5f8a31c14bfe147363c4324622c4a0801065f5e2f not found: ID does not exist" containerID="13c1df7f2880c2f58f13c9e5f8a31c14bfe147363c4324622c4a0801065f5e2f"
Sep 18 20:32:05.521866 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:32:05.521809    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13c1df7f2880c2f58f13c9e5f8a31c14bfe147363c4324622c4a0801065f5e2f"} err="failed to get container status \"13c1df7f2880c2f58f13c9e5f8a31c14bfe147363c4324622c4a0801065f5e2f\": rpc error: code = NotFound desc = could not find container \"13c1df7f2880c2f58f13c9e5f8a31c14bfe147363c4324622c4a0801065f5e2f\": container with ID starting with 13c1df7f2880c2f58f13c9e5f8a31c14bfe147363c4324622c4a0801065f5e2f not found: ID does not exist"
Sep 18 20:32:05.521866 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:32:05.521827    2549 scope.go:117] "RemoveContainer" containerID="8b23930ee8074f5d24393a3249ad26e96ba0db34880259b5f3ae9630e9e82bd3"
Sep 18 20:32:05.522131 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:32:05.522111    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b23930ee8074f5d24393a3249ad26e96ba0db34880259b5f3ae9630e9e82bd3\": container with ID starting with 8b23930ee8074f5d24393a3249ad26e96ba0db34880259b5f3ae9630e9e82bd3 not found: ID does not exist" containerID="8b23930ee8074f5d24393a3249ad26e96ba0db34880259b5f3ae9630e9e82bd3"
Sep 18 20:32:05.522179 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:32:05.522140    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b23930ee8074f5d24393a3249ad26e96ba0db34880259b5f3ae9630e9e82bd3"} err="failed to get container status \"8b23930ee8074f5d24393a3249ad26e96ba0db34880259b5f3ae9630e9e82bd3\": rpc error: code = NotFound desc = could not find container \"8b23930ee8074f5d24393a3249ad26e96ba0db34880259b5f3ae9630e9e82bd3\": container with ID starting with 8b23930ee8074f5d24393a3249ad26e96ba0db34880259b5f3ae9630e9e82bd3 not found: ID does not exist"
Sep 18 20:32:05.530260 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:32:05.530221    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9"]
Sep 18 20:32:05.533296 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:32:05.533268    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-40a1a5-openshift-default-64769dfc98-h89b9"]
Sep 18 20:32:07.521470 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:32:07.521436    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64206b1e-8c51-4e7f-9bbe-3ea703ccb671" path="/var/lib/kubelet/pods/64206b1e-8c51-4e7f-9bbe-3ea703ccb671/volumes"
Sep 18 20:32:10.675344 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:32:10.675293    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-81f19d-29829390-tgq9q"]
Sep 18 20:32:10.678988 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:32:10.678951    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-81f19d-29829390-tgq9q"]
Sep 18 20:32:11.522145 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:32:11.522105    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0f05846-6140-4298-a26d-21d9c15db0f3" path="/var/lib/kubelet/pods/b0f05846-6140-4298-a26d-21d9c15db0f3/volumes"
Sep 18 20:32:13.420899 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:32:13.420851    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:32:18.673818 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:32:18.673774    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq"]
Sep 18 20:32:18.674272 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:32:18.674059    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq" podUID="137f5df4-4bde-48f8-bb7e-9f629e941e0d" containerName="istio-proxy" containerID="cri-o://51d181a0c791d363668f8a7991f1926e9465ef7be04953181e4277364619c6b0" gracePeriod=30
Sep 18 20:32:23.444505 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:32:23.444464    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:32:23.957439 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:32:23.957339    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq"
Sep 18 20:32:24.082296 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:32:24.082248    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tvjhj\" (UniqueName: \"kubernetes.io/projected/137f5df4-4bde-48f8-bb7e-9f629e941e0d-kube-api-access-tvjhj\") pod \"137f5df4-4bde-48f8-bb7e-9f629e941e0d\" (UID: \"137f5df4-4bde-48f8-bb7e-9f629e941e0d\") "
Sep 18 20:32:24.082296 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:32:24.082298    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/137f5df4-4bde-48f8-bb7e-9f629e941e0d-istiod-ca-cert\") pod \"137f5df4-4bde-48f8-bb7e-9f629e941e0d\" (UID: \"137f5df4-4bde-48f8-bb7e-9f629e941e0d\") "
Sep 18 20:32:24.082623 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:32:24.082368    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/137f5df4-4bde-48f8-bb7e-9f629e941e0d-credential-socket\") pod \"137f5df4-4bde-48f8-bb7e-9f629e941e0d\" (UID: \"137f5df4-4bde-48f8-bb7e-9f629e941e0d\") "
Sep 18 20:32:24.082623 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:32:24.082456    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/137f5df4-4bde-48f8-bb7e-9f629e941e0d-workload-certs\") pod \"137f5df4-4bde-48f8-bb7e-9f629e941e0d\" (UID: \"137f5df4-4bde-48f8-bb7e-9f629e941e0d\") "
Sep 18 20:32:24.082623 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:32:24.082482    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/137f5df4-4bde-48f8-bb7e-9f629e941e0d-workload-socket\") pod \"137f5df4-4bde-48f8-bb7e-9f629e941e0d\" (UID: \"137f5df4-4bde-48f8-bb7e-9f629e941e0d\") "
Sep 18 20:32:24.082623 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:32:24.082524    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/137f5df4-4bde-48f8-bb7e-9f629e941e0d-istio-podinfo\") pod \"137f5df4-4bde-48f8-bb7e-9f629e941e0d\" (UID: \"137f5df4-4bde-48f8-bb7e-9f629e941e0d\") "
Sep 18 20:32:24.082623 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:32:24.082551    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/137f5df4-4bde-48f8-bb7e-9f629e941e0d-istio-envoy\") pod \"137f5df4-4bde-48f8-bb7e-9f629e941e0d\" (UID: \"137f5df4-4bde-48f8-bb7e-9f629e941e0d\") "
Sep 18 20:32:24.082916 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:32:24.082754    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/137f5df4-4bde-48f8-bb7e-9f629e941e0d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "137f5df4-4bde-48f8-bb7e-9f629e941e0d" (UID: "137f5df4-4bde-48f8-bb7e-9f629e941e0d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:32:24.082916 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:32:24.082769    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/137f5df4-4bde-48f8-bb7e-9f629e941e0d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "137f5df4-4bde-48f8-bb7e-9f629e941e0d" (UID: "137f5df4-4bde-48f8-bb7e-9f629e941e0d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:32:24.082916 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:32:24.082807    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/137f5df4-4bde-48f8-bb7e-9f629e941e0d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "137f5df4-4bde-48f8-bb7e-9f629e941e0d" (UID: "137f5df4-4bde-48f8-bb7e-9f629e941e0d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:32:24.082916 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:32:24.082809    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/137f5df4-4bde-48f8-bb7e-9f629e941e0d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "137f5df4-4bde-48f8-bb7e-9f629e941e0d" (UID: "137f5df4-4bde-48f8-bb7e-9f629e941e0d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:32:24.083136 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:32:24.082919    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/137f5df4-4bde-48f8-bb7e-9f629e941e0d-istio-token\") pod \"137f5df4-4bde-48f8-bb7e-9f629e941e0d\" (UID: \"137f5df4-4bde-48f8-bb7e-9f629e941e0d\") "
Sep 18 20:32:24.083136 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:32:24.082964    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/137f5df4-4bde-48f8-bb7e-9f629e941e0d-istio-data\") pod \"137f5df4-4bde-48f8-bb7e-9f629e941e0d\" (UID: \"137f5df4-4bde-48f8-bb7e-9f629e941e0d\") "
Sep 18 20:32:24.083361 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:32:24.083316    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/137f5df4-4bde-48f8-bb7e-9f629e941e0d-istio-data" (OuterVolumeSpecName: "istio-data") pod "137f5df4-4bde-48f8-bb7e-9f629e941e0d" (UID: "137f5df4-4bde-48f8-bb7e-9f629e941e0d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:32:24.083361 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:32:24.083334    2549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/137f5df4-4bde-48f8-bb7e-9f629e941e0d-credential-socket\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:32:24.083361 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:32:24.083358    2549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/137f5df4-4bde-48f8-bb7e-9f629e941e0d-workload-certs\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:32:24.083649 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:32:24.083372    2549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/137f5df4-4bde-48f8-bb7e-9f629e941e0d-workload-socket\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:32:24.083649 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:32:24.083409    2549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/137f5df4-4bde-48f8-bb7e-9f629e941e0d-istiod-ca-cert\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:32:24.084899 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:32:24.084870    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/137f5df4-4bde-48f8-bb7e-9f629e941e0d-kube-api-access-tvjhj" (OuterVolumeSpecName: "kube-api-access-tvjhj") pod "137f5df4-4bde-48f8-bb7e-9f629e941e0d" (UID: "137f5df4-4bde-48f8-bb7e-9f629e941e0d"). InnerVolumeSpecName "kube-api-access-tvjhj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:32:24.085003 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:32:24.084889    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/137f5df4-4bde-48f8-bb7e-9f629e941e0d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "137f5df4-4bde-48f8-bb7e-9f629e941e0d" (UID: "137f5df4-4bde-48f8-bb7e-9f629e941e0d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:32:24.085156 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:32:24.085132    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/137f5df4-4bde-48f8-bb7e-9f629e941e0d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "137f5df4-4bde-48f8-bb7e-9f629e941e0d" (UID: "137f5df4-4bde-48f8-bb7e-9f629e941e0d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 20:32:24.085222 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:32:24.085207    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/137f5df4-4bde-48f8-bb7e-9f629e941e0d-istio-token" (OuterVolumeSpecName: "istio-token") pod "137f5df4-4bde-48f8-bb7e-9f629e941e0d" (UID: "137f5df4-4bde-48f8-bb7e-9f629e941e0d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:32:24.184611 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:32:24.184569    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvjhj\" (UniqueName: \"kubernetes.io/projected/137f5df4-4bde-48f8-bb7e-9f629e941e0d-kube-api-access-tvjhj\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:32:24.184611 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:32:24.184605    2549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/137f5df4-4bde-48f8-bb7e-9f629e941e0d-istio-podinfo\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:32:24.184611 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:32:24.184616    2549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/137f5df4-4bde-48f8-bb7e-9f629e941e0d-istio-envoy\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:32:24.184611 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:32:24.184624    2549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/137f5df4-4bde-48f8-bb7e-9f629e941e0d-istio-token\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:32:24.184905 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:32:24.184635    2549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/137f5df4-4bde-48f8-bb7e-9f629e941e0d-istio-data\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:32:24.623796 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:32:24.623756    2549 generic.go:356] "Generic (PLEG): container finished" podID="137f5df4-4bde-48f8-bb7e-9f629e941e0d" containerID="51d181a0c791d363668f8a7991f1926e9465ef7be04953181e4277364619c6b0" exitCode=0
Sep 18 20:32:24.624207 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:32:24.623813    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq"
Sep 18 20:32:24.624207 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:32:24.623842    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq" event={"ID":"137f5df4-4bde-48f8-bb7e-9f629e941e0d","Type":"ContainerDied","Data":"51d181a0c791d363668f8a7991f1926e9465ef7be04953181e4277364619c6b0"}
Sep 18 20:32:24.624207 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:32:24.623906    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq" event={"ID":"137f5df4-4bde-48f8-bb7e-9f629e941e0d","Type":"ContainerDied","Data":"63446d91966e884fb8b1c240047a46431afbb1153dfe2db129915665ea57266d"}
Sep 18 20:32:24.624207 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:32:24.623929    2549 scope.go:117] "RemoveContainer" containerID="51d181a0c791d363668f8a7991f1926e9465ef7be04953181e4277364619c6b0"
Sep 18 20:32:24.634134 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:32:24.634114    2549 scope.go:117] "RemoveContainer" containerID="553d73c599b9e77dd37f4d4cc516d6478ba9d410c7a12795a29e144c8c4a273b"
Sep 18 20:32:24.643283 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:32:24.643260    2549 scope.go:117] "RemoveContainer" containerID="51d181a0c791d363668f8a7991f1926e9465ef7be04953181e4277364619c6b0"
Sep 18 20:32:24.643678 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:32:24.643655    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51d181a0c791d363668f8a7991f1926e9465ef7be04953181e4277364619c6b0\": container with ID starting with 51d181a0c791d363668f8a7991f1926e9465ef7be04953181e4277364619c6b0 not found: ID does not exist" containerID="51d181a0c791d363668f8a7991f1926e9465ef7be04953181e4277364619c6b0"
Sep 18 20:32:24.643773 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:32:24.643687    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51d181a0c791d363668f8a7991f1926e9465ef7be04953181e4277364619c6b0"} err="failed to get container status \"51d181a0c791d363668f8a7991f1926e9465ef7be04953181e4277364619c6b0\": rpc error: code = NotFound desc = could not find container \"51d181a0c791d363668f8a7991f1926e9465ef7be04953181e4277364619c6b0\": container with ID starting with 51d181a0c791d363668f8a7991f1926e9465ef7be04953181e4277364619c6b0 not found: ID does not exist"
Sep 18 20:32:24.643773 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:32:24.643706    2549 scope.go:117] "RemoveContainer" containerID="553d73c599b9e77dd37f4d4cc516d6478ba9d410c7a12795a29e144c8c4a273b"
Sep 18 20:32:24.644012 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:32:24.643994    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"553d73c599b9e77dd37f4d4cc516d6478ba9d410c7a12795a29e144c8c4a273b\": container with ID starting with 553d73c599b9e77dd37f4d4cc516d6478ba9d410c7a12795a29e144c8c4a273b not found: ID does not exist" containerID="553d73c599b9e77dd37f4d4cc516d6478ba9d410c7a12795a29e144c8c4a273b"
Sep 18 20:32:24.644080 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:32:24.644019    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"553d73c599b9e77dd37f4d4cc516d6478ba9d410c7a12795a29e144c8c4a273b"} err="failed to get container status \"553d73c599b9e77dd37f4d4cc516d6478ba9d410c7a12795a29e144c8c4a273b\": rpc error: code = NotFound desc = could not find container \"553d73c599b9e77dd37f4d4cc516d6478ba9d410c7a12795a29e144c8c4a273b\": container with ID starting with 553d73c599b9e77dd37f4d4cc516d6478ba9d410c7a12795a29e144c8c4a273b not found: ID does not exist"
Sep 18 20:32:24.665189 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:32:24.665140    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq"]
Sep 18 20:32:24.669017 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:32:24.668978    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-81f19d-openshift-default-5557d4f646-hqkcq"]
Sep 18 20:32:25.521768 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:32:25.521729    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="137f5df4-4bde-48f8-bb7e-9f629e941e0d" path="/var/lib/kubelet/pods/137f5df4-4bde-48f8-bb7e-9f629e941e0d/volumes"
Sep 18 20:33:45.523225 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:33:45.523185    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:33:47.283985 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:33:47.283941    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-988c79b4c-n8skp"]
Sep 18 20:33:47.284449 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:33:47.284335    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64206b1e-8c51-4e7f-9bbe-3ea703ccb671" containerName="istio-proxy"
Sep 18 20:33:47.284449 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:33:47.284347    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="64206b1e-8c51-4e7f-9bbe-3ea703ccb671" containerName="istio-proxy"
Sep 18 20:33:47.284449 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:33:47.284356    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="137f5df4-4bde-48f8-bb7e-9f629e941e0d" containerName="istio-proxy"
Sep 18 20:33:47.284449 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:33:47.284363    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="137f5df4-4bde-48f8-bb7e-9f629e941e0d" containerName="istio-proxy"
Sep 18 20:33:47.284449 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:33:47.284373    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64206b1e-8c51-4e7f-9bbe-3ea703ccb671" containerName="istio-proxy"
Sep 18 20:33:47.284449 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:33:47.284392    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="64206b1e-8c51-4e7f-9bbe-3ea703ccb671" containerName="istio-proxy"
Sep 18 20:33:47.284449 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:33:47.284403    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1feabf84-a082-4982-a415-7efde1ee6a29" containerName="istio-proxy"
Sep 18 20:33:47.284449 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:33:47.284409    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="1feabf84-a082-4982-a415-7efde1ee6a29" containerName="istio-proxy"
Sep 18 20:33:47.284449 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:33:47.284414    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9532d1a0-e1cb-4147-9815-f7e6fc35aac8" containerName="revoke-keys"
Sep 18 20:33:47.284449 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:33:47.284419    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="9532d1a0-e1cb-4147-9815-f7e6fc35aac8" containerName="revoke-keys"
Sep 18 20:33:47.284449 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:33:47.284433    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1feabf84-a082-4982-a415-7efde1ee6a29" containerName="istio-proxy"
Sep 18 20:33:47.284449 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:33:47.284438    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="1feabf84-a082-4982-a415-7efde1ee6a29" containerName="istio-proxy"
Sep 18 20:33:47.284886 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:33:47.284496    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="9532d1a0-e1cb-4147-9815-f7e6fc35aac8" containerName="revoke-keys"
Sep 18 20:33:47.284886 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:33:47.284506    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="1feabf84-a082-4982-a415-7efde1ee6a29" containerName="istio-proxy"
Sep 18 20:33:47.284886 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:33:47.284512    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="137f5df4-4bde-48f8-bb7e-9f629e941e0d" containerName="istio-proxy"
Sep 18 20:33:47.284886 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:33:47.284517    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="1feabf84-a082-4982-a415-7efde1ee6a29" containerName="istio-proxy"
Sep 18 20:33:47.284886 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:33:47.284524    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="64206b1e-8c51-4e7f-9bbe-3ea703ccb671" containerName="istio-proxy"
Sep 18 20:33:47.284886 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:33:47.284530    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="64206b1e-8c51-4e7f-9bbe-3ea703ccb671" containerName="istio-proxy"
Sep 18 20:33:47.287353 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:33:47.287326    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-988c79b4c-n8skp"
Sep 18 20:33:47.292044 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:33:47.292017    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-j9h7q\""
Sep 18 20:33:47.292217 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:33:47.292046    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 18 20:33:47.292217 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:33:47.292018    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 18 20:33:47.292217 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:33:47.292039    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 18 20:33:47.303344 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:33:47.303310    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-988c79b4c-n8skp"]
Sep 18 20:33:47.337999 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:33:47.337949    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jdzv\" (UniqueName: \"kubernetes.io/projected/1832850e-958b-4d80-a481-3ee0808db619-kube-api-access-7jdzv\") pod \"maas-controller-988c79b4c-n8skp\" (UID: \"1832850e-958b-4d80-a481-3ee0808db619\") " pod="opendatahub/maas-controller-988c79b4c-n8skp"
Sep 18 20:33:47.338190 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:33:47.338008    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1832850e-958b-4d80-a481-3ee0808db619-metrics-certs\") pod \"maas-controller-988c79b4c-n8skp\" (UID: \"1832850e-958b-4d80-a481-3ee0808db619\") " pod="opendatahub/maas-controller-988c79b4c-n8skp"
Sep 18 20:33:47.338190 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:33:47.338124    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1832850e-958b-4d80-a481-3ee0808db619-webhook-cert\") pod \"maas-controller-988c79b4c-n8skp\" (UID: \"1832850e-958b-4d80-a481-3ee0808db619\") " pod="opendatahub/maas-controller-988c79b4c-n8skp"
Sep 18 20:33:47.439546 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:33:47.439499    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1832850e-958b-4d80-a481-3ee0808db619-webhook-cert\") pod \"maas-controller-988c79b4c-n8skp\" (UID: \"1832850e-958b-4d80-a481-3ee0808db619\") " pod="opendatahub/maas-controller-988c79b4c-n8skp"
Sep 18 20:33:47.439546 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:33:47.439551    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jdzv\" (UniqueName: \"kubernetes.io/projected/1832850e-958b-4d80-a481-3ee0808db619-kube-api-access-7jdzv\") pod \"maas-controller-988c79b4c-n8skp\" (UID: \"1832850e-958b-4d80-a481-3ee0808db619\") " pod="opendatahub/maas-controller-988c79b4c-n8skp"
Sep 18 20:33:47.439814 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:33:47.439582    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1832850e-958b-4d80-a481-3ee0808db619-metrics-certs\") pod \"maas-controller-988c79b4c-n8skp\" (UID: \"1832850e-958b-4d80-a481-3ee0808db619\") " pod="opendatahub/maas-controller-988c79b4c-n8skp"
Sep 18 20:33:47.442161 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:33:47.442125    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1832850e-958b-4d80-a481-3ee0808db619-metrics-certs\") pod \"maas-controller-988c79b4c-n8skp\" (UID: \"1832850e-958b-4d80-a481-3ee0808db619\") " pod="opendatahub/maas-controller-988c79b4c-n8skp"
Sep 18 20:33:47.442289 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:33:47.442253    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1832850e-958b-4d80-a481-3ee0808db619-webhook-cert\") pod \"maas-controller-988c79b4c-n8skp\" (UID: \"1832850e-958b-4d80-a481-3ee0808db619\") " pod="opendatahub/maas-controller-988c79b4c-n8skp"
Sep 18 20:33:47.449809 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:33:47.449768    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jdzv\" (UniqueName: \"kubernetes.io/projected/1832850e-958b-4d80-a481-3ee0808db619-kube-api-access-7jdzv\") pod \"maas-controller-988c79b4c-n8skp\" (UID: \"1832850e-958b-4d80-a481-3ee0808db619\") " pod="opendatahub/maas-controller-988c79b4c-n8skp"
Sep 18 20:33:47.599023 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:33:47.598982    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-988c79b4c-n8skp"
Sep 18 20:33:47.880418 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:33:47.880373    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-988c79b4c-n8skp"]
Sep 18 20:33:47.882635 ip-10-0-140-178 kubenswrapper[2549]: W0918 20:33:47.882586    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1832850e_958b_4d80_a481_3ee0808db619.slice/crio-e388adc30f07e39014ce9f21dee192484f5098f37bfd51b29cd90deb4b52a038 WatchSource:0}: Error finding container e388adc30f07e39014ce9f21dee192484f5098f37bfd51b29cd90deb4b52a038: Status 404 returned error can't find the container with id e388adc30f07e39014ce9f21dee192484f5098f37bfd51b29cd90deb4b52a038
Sep 18 20:33:47.884055 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:33:47.884029    2549 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 20:33:48.126860 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:33:48.126819    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-988c79b4c-n8skp" event={"ID":"1832850e-958b-4d80-a481-3ee0808db619","Type":"ContainerStarted","Data":"e388adc30f07e39014ce9f21dee192484f5098f37bfd51b29cd90deb4b52a038"}
Sep 18 20:33:51.147605 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:33:51.147558    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-988c79b4c-n8skp" event={"ID":"1832850e-958b-4d80-a481-3ee0808db619","Type":"ContainerStarted","Data":"9b19851b400cdbcbaf99c90c2dd3e9462b9703a2701611079828df338b8dab1e"}
Sep 18 20:33:51.148025 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:33:51.147633    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-988c79b4c-n8skp"
Sep 18 20:33:55.505148 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:33:55.505087    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-988c79b4c-n8skp" podStartSLOduration=6.205660525 podStartE2EDuration="8.50506839s" totalImagesPullingTime="2.299407865s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:33:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:33:55.503616364 +0000 UTC m=+2220.457357534" watchObservedRunningTime="2026-09-18 20:33:55.50506839 +0000 UTC m=+2220.458809557"
Sep 18 20:34:02.155580 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:34:02.155545    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-988c79b4c-n8skp"
Sep 18 20:34:29.747976 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:34:29.747876    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:34:37.434163 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:34:37.434124    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:34:46.329263 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:34:46.329218    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:34:54.209856 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:34:54.209813    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:35:08.525293 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:35:08.525250    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:35:17.765250 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:35:17.765205    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:35:19.919276 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:35:19.919232    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9kdnl"]
Sep 18 20:35:19.919744 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:35:19.919480    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9kdnl" podUID="4817b2de-b55e-4ddd-8c8a-4eb953490a17" containerName="manager" containerID="cri-o://9d03c612748e55076c53d7da80b95a977bed8b788839cdf0244a6576a30d63e4" gracePeriod=10
Sep 18 20:35:20.474692 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:35:20.474663    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9kdnl"
Sep 18 20:35:20.533466 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:35:20.533353    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h2nkc\" (UniqueName: \"kubernetes.io/projected/4817b2de-b55e-4ddd-8c8a-4eb953490a17-kube-api-access-h2nkc\") pod \"4817b2de-b55e-4ddd-8c8a-4eb953490a17\" (UID: \"4817b2de-b55e-4ddd-8c8a-4eb953490a17\") "
Sep 18 20:35:20.533466 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:35:20.533458    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4817b2de-b55e-4ddd-8c8a-4eb953490a17-extensions-socket-volume\") pod \"4817b2de-b55e-4ddd-8c8a-4eb953490a17\" (UID: \"4817b2de-b55e-4ddd-8c8a-4eb953490a17\") "
Sep 18 20:35:20.533910 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:35:20.533874    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4817b2de-b55e-4ddd-8c8a-4eb953490a17-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "4817b2de-b55e-4ddd-8c8a-4eb953490a17" (UID: "4817b2de-b55e-4ddd-8c8a-4eb953490a17"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:35:20.535673 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:35:20.535647    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4817b2de-b55e-4ddd-8c8a-4eb953490a17-kube-api-access-h2nkc" (OuterVolumeSpecName: "kube-api-access-h2nkc") pod "4817b2de-b55e-4ddd-8c8a-4eb953490a17" (UID: "4817b2de-b55e-4ddd-8c8a-4eb953490a17"). InnerVolumeSpecName "kube-api-access-h2nkc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:35:20.634898 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:35:20.634850    2549 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4817b2de-b55e-4ddd-8c8a-4eb953490a17-extensions-socket-volume\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:35:20.634898 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:35:20.634886    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2nkc\" (UniqueName: \"kubernetes.io/projected/4817b2de-b55e-4ddd-8c8a-4eb953490a17-kube-api-access-h2nkc\") on node \"ip-10-0-140-178.ec2.internal\" DevicePath \"\""
Sep 18 20:35:20.695745 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:35:20.695704    2549 generic.go:356] "Generic (PLEG): container finished" podID="4817b2de-b55e-4ddd-8c8a-4eb953490a17" containerID="9d03c612748e55076c53d7da80b95a977bed8b788839cdf0244a6576a30d63e4" exitCode=0
Sep 18 20:35:20.695969 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:35:20.695757    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9kdnl"
Sep 18 20:35:20.695969 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:35:20.695790    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9kdnl" event={"ID":"4817b2de-b55e-4ddd-8c8a-4eb953490a17","Type":"ContainerDied","Data":"9d03c612748e55076c53d7da80b95a977bed8b788839cdf0244a6576a30d63e4"}
Sep 18 20:35:20.695969 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:35:20.695847    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9kdnl" event={"ID":"4817b2de-b55e-4ddd-8c8a-4eb953490a17","Type":"ContainerDied","Data":"fb918bd342d657d1912b9020ec08dd25df3a8c304fe5972c9d7a097c076a9a7f"}
Sep 18 20:35:20.695969 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:35:20.695871    2549 scope.go:117] "RemoveContainer" containerID="9d03c612748e55076c53d7da80b95a977bed8b788839cdf0244a6576a30d63e4"
Sep 18 20:35:20.706484 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:35:20.706464    2549 scope.go:117] "RemoveContainer" containerID="9d03c612748e55076c53d7da80b95a977bed8b788839cdf0244a6576a30d63e4"
Sep 18 20:35:20.706872 ip-10-0-140-178 kubenswrapper[2549]: E0918 20:35:20.706852    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d03c612748e55076c53d7da80b95a977bed8b788839cdf0244a6576a30d63e4\": container with ID starting with 9d03c612748e55076c53d7da80b95a977bed8b788839cdf0244a6576a30d63e4 not found: ID does not exist" containerID="9d03c612748e55076c53d7da80b95a977bed8b788839cdf0244a6576a30d63e4"
Sep 18 20:35:20.706936 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:35:20.706885    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d03c612748e55076c53d7da80b95a977bed8b788839cdf0244a6576a30d63e4"} err="failed to get container status \"9d03c612748e55076c53d7da80b95a977bed8b788839cdf0244a6576a30d63e4\": rpc error: code = NotFound desc = could not find container \"9d03c612748e55076c53d7da80b95a977bed8b788839cdf0244a6576a30d63e4\": container with ID starting with 9d03c612748e55076c53d7da80b95a977bed8b788839cdf0244a6576a30d63e4 not found: ID does not exist"
Sep 18 20:35:20.726756 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:35:20.726721    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9kdnl"]
Sep 18 20:35:20.728854 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:35:20.728830    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9kdnl"]
Sep 18 20:35:21.521266 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:35:21.521229    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4817b2de-b55e-4ddd-8c8a-4eb953490a17" path="/var/lib/kubelet/pods/4817b2de-b55e-4ddd-8c8a-4eb953490a17/volumes"
Sep 18 20:35:56.206659 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:35:56.206623    2549 scope.go:117] "RemoveContainer" containerID="469b04f1177c8e3226c130c578ac63edeaff26863beda87bd8962acecaeccf09"
Sep 18 20:36:20.619554 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:36:20.619514    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gmr8t"]
Sep 18 20:36:20.620041 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:36:20.619904    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="137f5df4-4bde-48f8-bb7e-9f629e941e0d" containerName="istio-proxy"
Sep 18 20:36:20.620041 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:36:20.619916    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="137f5df4-4bde-48f8-bb7e-9f629e941e0d" containerName="istio-proxy"
Sep 18 20:36:20.620041 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:36:20.619935    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4817b2de-b55e-4ddd-8c8a-4eb953490a17" containerName="manager"
Sep 18 20:36:20.620041 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:36:20.619941    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="4817b2de-b55e-4ddd-8c8a-4eb953490a17" containerName="manager"
Sep 18 20:36:20.620041 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:36:20.620016    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="4817b2de-b55e-4ddd-8c8a-4eb953490a17" containerName="manager"
Sep 18 20:36:20.620041 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:36:20.620031    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="137f5df4-4bde-48f8-bb7e-9f629e941e0d" containerName="istio-proxy"
Sep 18 20:36:20.623228 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:36:20.623206    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gmr8t"
Sep 18 20:36:20.626655 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:36:20.626621    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-x7vvt\""
Sep 18 20:36:20.642242 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:36:20.642205    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gmr8t"]
Sep 18 20:36:20.779633 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:36:20.779589    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c57974f3-8d5e-4a32-a621-a0870b6ce270-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-gmr8t\" (UID: \"c57974f3-8d5e-4a32-a621-a0870b6ce270\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gmr8t"
Sep 18 20:36:20.779835 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:36:20.779643    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkvpt\" (UniqueName: \"kubernetes.io/projected/c57974f3-8d5e-4a32-a621-a0870b6ce270-kube-api-access-zkvpt\") pod \"kuadrant-operator-controller-manager-5769f99d9f-gmr8t\" (UID: \"c57974f3-8d5e-4a32-a621-a0870b6ce270\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gmr8t"
Sep 18 20:36:20.880752 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:36:20.880654    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c57974f3-8d5e-4a32-a621-a0870b6ce270-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-gmr8t\" (UID: \"c57974f3-8d5e-4a32-a621-a0870b6ce270\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gmr8t"
Sep 18 20:36:20.880752 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:36:20.880700    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkvpt\" (UniqueName: \"kubernetes.io/projected/c57974f3-8d5e-4a32-a621-a0870b6ce270-kube-api-access-zkvpt\") pod \"kuadrant-operator-controller-manager-5769f99d9f-gmr8t\" (UID: \"c57974f3-8d5e-4a32-a621-a0870b6ce270\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gmr8t"
Sep 18 20:36:20.881092 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:36:20.881070    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c57974f3-8d5e-4a32-a621-a0870b6ce270-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-gmr8t\" (UID: \"c57974f3-8d5e-4a32-a621-a0870b6ce270\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gmr8t"
Sep 18 20:36:20.891784 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:36:20.891747    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkvpt\" (UniqueName: \"kubernetes.io/projected/c57974f3-8d5e-4a32-a621-a0870b6ce270-kube-api-access-zkvpt\") pod \"kuadrant-operator-controller-manager-5769f99d9f-gmr8t\" (UID: \"c57974f3-8d5e-4a32-a621-a0870b6ce270\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gmr8t"
Sep 18 20:36:20.933788 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:36:20.933742    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gmr8t"
Sep 18 20:36:21.242803 ip-10-0-140-178 kubenswrapper[2549]: W0918 20:36:21.242768    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc57974f3_8d5e_4a32_a621_a0870b6ce270.slice/crio-014421e29df1442f7b878697ab0647612d9a268b7f41d2237b36e2003188993e WatchSource:0}: Error finding container 014421e29df1442f7b878697ab0647612d9a268b7f41d2237b36e2003188993e: Status 404 returned error can't find the container with id 014421e29df1442f7b878697ab0647612d9a268b7f41d2237b36e2003188993e
Sep 18 20:36:21.242957 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:36:21.242921    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gmr8t"]
Sep 18 20:36:22.056311 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:36:22.056269    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gmr8t" event={"ID":"c57974f3-8d5e-4a32-a621-a0870b6ce270","Type":"ContainerStarted","Data":"cd11454b61df44374cbf1daaad25715ad3d3af3481676024f69af5bb034d1617"}
Sep 18 20:36:22.056765 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:36:22.056330    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gmr8t" event={"ID":"c57974f3-8d5e-4a32-a621-a0870b6ce270","Type":"ContainerStarted","Data":"014421e29df1442f7b878697ab0647612d9a268b7f41d2237b36e2003188993e"}
Sep 18 20:36:22.056765 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:36:22.056363    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gmr8t"
Sep 18 20:36:33.066174 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:36:33.066135    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gmr8t"
Sep 18 20:36:33.101907 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:36:33.101838    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gmr8t" podStartSLOduration=13.101809265 podStartE2EDuration="13.101809265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:36:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:36:22.091729039 +0000 UTC m=+2367.045470206" watchObservedRunningTime="2026-09-18 20:36:33.101809265 +0000 UTC m=+2378.055550432"
Sep 18 20:36:44.538805 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:36:44.538763    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:36:53.220762 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:36:53.220719    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:36:55.707916 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:36:55.707881    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-scn7p_a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48/ovn-acl-logging/0.log"
Sep 18 20:36:55.715509 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:36:55.715479    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-scn7p_a7a8f2b1-a3d0-4d9a-9099-fb05824d3e48/ovn-acl-logging/0.log"
Sep 18 20:36:56.233871 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:36:56.233834    2549 scope.go:117] "RemoveContainer" containerID="0009ba35b4c89344f36ee09f76c5e95d7ee7ccde37c2d77dc0a7f783ffc456a8"
Sep 18 20:36:56.243011 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:36:56.242982    2549 scope.go:117] "RemoveContainer" containerID="c258e8ddc27252faf014f783e1aa2187aed8da788e668cc8b7546707b8aa80db"
Sep 18 20:36:56.256304 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:36:56.256265    2549 scope.go:117] "RemoveContainer" containerID="882f6e755167c274a34bea51f3baa5be6a587214e67ece6e049be2419c5a9063"
Sep 18 20:36:56.265329 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:36:56.265301    2549 scope.go:117] "RemoveContainer" containerID="5c46487b1532a9e8d2db6363d663a1e35973c9eb9b0d93ac25d474b395cc4b6a"
Sep 18 20:37:00.129400 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:37:00.129346    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:37:09.269524 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:37:09.269484    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:37:29.925230 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:37:29.925136    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:37:39.972976 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:37:39.972942    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:37:46.591940 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:37:46.591901    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:37:54.526137 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:37:54.526106    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:37:56.301084 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:37:56.301055    2549 scope.go:117] "RemoveContainer" containerID="f29e79867f0c452a2f06058827d64b0abd9a39e64f1e1c41fedf07a44ba58968"
Sep 18 20:37:56.309593 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:37:56.309579    2549 scope.go:117] "RemoveContainer" containerID="27d663eb173aeffaa2b447b7e2f868a6920fa84e7ee61ab7a201a0687b54111c"
Sep 18 20:38:01.613871 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:01.613835    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:38:10.145081 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:10.145047    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:38:17.435973 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:17.435937    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jf6z5"]
Sep 18 20:38:26.297122 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:26.297091    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-rmlx2_10c00388-66f0-49f6-aa95-30a676551fa3/manager/0.log"
Sep 18 20:38:26.418339 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:26.418308    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-988c79b4c-n8skp_1832850e-958b-4d80-a481-3ee0808db619/manager/0.log"
Sep 18 20:38:26.546200 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:26.546174    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-nrr8r_d199cb6d-b458-4475-86ae-5aa634c8f066/server/0.log"
Sep 18 20:38:26.666183 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:26.666151    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-z2rn8_49228637-596f-49f2-a97a-646c9272793f/manager/0.log"
Sep 18 20:38:26.784779 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:26.784752    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-f76f79b89-5bbqs_3780e3fd-8628-4933-9654-3c7a08699756/copy-manifests/0.log"
Sep 18 20:38:26.797669 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:26.797640    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-f76f79b89-5bbqs_3780e3fd-8628-4933-9654-3c7a08699756/manager/0.log"
Sep 18 20:38:27.630933 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:27.630903    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29829375-4wmm9_21ca971a-459c-49d8-a299-85be11490dc3/cleanup/0.log"
Sep 18 20:38:27.745350 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:27.745320    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29829390-rsbw2_a170b409-cd6b-4543-9667-e10488259c6b/cleanup/0.log"
Sep 18 20:38:28.852114 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:28.852085    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-v7sqd_a89f83d9-2e6d-4f5f-ae28-f8e093121094/manager/0.log"
Sep 18 20:38:28.967763 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:28.967737    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-bbxwx_7904c124-a678-4cf1-809a-fd67593611b0/kuadrant-console-plugin/0.log"
Sep 18 20:38:29.089876 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:29.089845    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-gmr8t_c57974f3-8d5e-4a32-a621-a0870b6ce270/manager/0.log"
Sep 18 20:38:29.281598 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:29.281528    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-jf6z5_95ac679b-9bdc-493a-8c1f-b758fff7d092/limitador/0.log"
Sep 18 20:38:29.776447 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:29.776415    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-5zktm_61625208-077f-481b-8160-1699ab7e11fd/istio-proxy/0.log"
Sep 18 20:38:30.245920 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:30.245891    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-thwfx_3193c781-85d8-48b0-956b-d6d3b0b31fd8/istio-proxy/1.log"
Sep 18 20:38:30.614662 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:30.614633    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7ff8bb4888-5sfkj_5c7b94cd-de57-4efe-a96c-1d349424b2fb/router/0.log"
Sep 18 20:38:37.583906 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:37.583868    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-w4xpn/must-gather-4xcxw"]
Sep 18 20:38:37.587253 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:37.587228    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-w4xpn/must-gather-4xcxw"
Sep 18 20:38:37.590498 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:37.590473    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-w4xpn\"/\"openshift-service-ca.crt\""
Sep 18 20:38:37.590610 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:37.590505    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-w4xpn\"/\"kube-root-ca.crt\""
Sep 18 20:38:37.592242 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:37.592228    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-w4xpn\"/\"default-dockercfg-68gcr\""
Sep 18 20:38:37.601516 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:37.601494    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-w4xpn/must-gather-4xcxw"]
Sep 18 20:38:37.729603 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:37.729563    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rn5hs\" (UniqueName: \"kubernetes.io/projected/cea4040c-c7d7-4473-a08e-326d534ff169-kube-api-access-rn5hs\") pod \"must-gather-4xcxw\" (UID: \"cea4040c-c7d7-4473-a08e-326d534ff169\") " pod="openshift-must-gather-w4xpn/must-gather-4xcxw"
Sep 18 20:38:37.729813 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:37.729674    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cea4040c-c7d7-4473-a08e-326d534ff169-must-gather-output\") pod \"must-gather-4xcxw\" (UID: \"cea4040c-c7d7-4473-a08e-326d534ff169\") " pod="openshift-must-gather-w4xpn/must-gather-4xcxw"
Sep 18 20:38:37.830401 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:37.830346    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn5hs\" (UniqueName: \"kubernetes.io/projected/cea4040c-c7d7-4473-a08e-326d534ff169-kube-api-access-rn5hs\") pod \"must-gather-4xcxw\" (UID: \"cea4040c-c7d7-4473-a08e-326d534ff169\") " pod="openshift-must-gather-w4xpn/must-gather-4xcxw"
Sep 18 20:38:37.830602 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:37.830422    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cea4040c-c7d7-4473-a08e-326d534ff169-must-gather-output\") pod \"must-gather-4xcxw\" (UID: \"cea4040c-c7d7-4473-a08e-326d534ff169\") " pod="openshift-must-gather-w4xpn/must-gather-4xcxw"
Sep 18 20:38:37.830762 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:37.830740    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cea4040c-c7d7-4473-a08e-326d534ff169-must-gather-output\") pod \"must-gather-4xcxw\" (UID: \"cea4040c-c7d7-4473-a08e-326d534ff169\") " pod="openshift-must-gather-w4xpn/must-gather-4xcxw"
Sep 18 20:38:37.841567 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:37.841507    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rn5hs\" (UniqueName: \"kubernetes.io/projected/cea4040c-c7d7-4473-a08e-326d534ff169-kube-api-access-rn5hs\") pod \"must-gather-4xcxw\" (UID: \"cea4040c-c7d7-4473-a08e-326d534ff169\") " pod="openshift-must-gather-w4xpn/must-gather-4xcxw"
Sep 18 20:38:37.896769 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:37.896744    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-w4xpn/must-gather-4xcxw"
Sep 18 20:38:38.177291 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:38.177266    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-w4xpn/must-gather-4xcxw"]
Sep 18 20:38:38.179000 ip-10-0-140-178 kubenswrapper[2549]: W0918 20:38:38.178972    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcea4040c_c7d7_4473_a08e_326d534ff169.slice/crio-b9afc07a843d121b5394d6276889853746b359f95f5b1c47987197e18ae64b1a WatchSource:0}: Error finding container b9afc07a843d121b5394d6276889853746b359f95f5b1c47987197e18ae64b1a: Status 404 returned error can't find the container with id b9afc07a843d121b5394d6276889853746b359f95f5b1c47987197e18ae64b1a
Sep 18 20:38:38.844144 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:38.844098    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-w4xpn/must-gather-4xcxw" event={"ID":"cea4040c-c7d7-4473-a08e-326d534ff169","Type":"ContainerStarted","Data":"b9afc07a843d121b5394d6276889853746b359f95f5b1c47987197e18ae64b1a"}
Sep 18 20:38:39.855114 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:39.855067    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-w4xpn/must-gather-4xcxw" event={"ID":"cea4040c-c7d7-4473-a08e-326d534ff169","Type":"ContainerStarted","Data":"5da9077aec163245841b48d0ff30022454475b53dfc2069c5753524e92bb12ed"}
Sep 18 20:38:39.855643 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:39.855130    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-w4xpn/must-gather-4xcxw" event={"ID":"cea4040c-c7d7-4473-a08e-326d534ff169","Type":"ContainerStarted","Data":"dbf12df226f1b7bf5dcbc2c182ec2341657d0e6bab7ad10362a97696435413d7"}
Sep 18 20:38:39.877802 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:39.877740    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-w4xpn/must-gather-4xcxw" podStartSLOduration=2.133934164 podStartE2EDuration="2.877717348s" totalImagesPullingTime="743.783184ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:38:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:38:39.875664882 +0000 UTC m=+2504.829406050" watchObservedRunningTime="2026-09-18 20:38:39.877717348 +0000 UTC m=+2504.831458518"
Sep 18 20:38:40.956845 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:40.956812    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-zq878_254868d4-6e40-410f-9a3d-0f994b704438/global-pull-secret-syncer/0.log"
Sep 18 20:38:41.145482 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:41.145412    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-tlqj8_2e178884-5bfb-46cb-9305-6170284cb243/konnectivity-agent/0.log"
Sep 18 20:38:41.236602 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:41.236526    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-178.ec2.internal_2748dceba5a1bbaba05c6bb022d69842/haproxy/0.log"
Sep 18 20:38:45.776309 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:45.776279    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-v7sqd_a89f83d9-2e6d-4f5f-ae28-f8e093121094/manager/0.log"
Sep 18 20:38:45.805084 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:45.805041    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-bbxwx_7904c124-a678-4cf1-809a-fd67593611b0/kuadrant-console-plugin/0.log"
Sep 18 20:38:45.853282 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:45.853249    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-gmr8t_c57974f3-8d5e-4a32-a621-a0870b6ce270/manager/0.log"
Sep 18 20:38:45.883468 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:45.883403    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-jf6z5_95ac679b-9bdc-493a-8c1f-b758fff7d092/limitador/0.log"
Sep 18 20:38:47.602565 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:47.602529    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-847b8bd8d7-49kw5_96908402-98ec-4672-a484-d136b3546b7e/cluster-monitoring-operator/0.log"
Sep 18 20:38:47.922209 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:47.922092    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5bx25_8b2bb638-e334-44e1-9449-6645ca708da8/node-exporter/0.log"
Sep 18 20:38:47.952021 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:47.951990    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5bx25_8b2bb638-e334-44e1-9449-6645ca708da8/kube-rbac-proxy/0.log"
Sep 18 20:38:47.981033 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:47.981004    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5bx25_8b2bb638-e334-44e1-9449-6645ca708da8/init-textfile/0.log"
Sep 18 20:38:48.223513 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:48.223396    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-zshqm_c22e65d0-1e2f-4a02-a878-2508729abe1f/kube-rbac-proxy-main/0.log"
Sep 18 20:38:48.262504 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:48.262473    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-zshqm_c22e65d0-1e2f-4a02-a878-2508729abe1f/kube-rbac-proxy-self/0.log"
Sep 18 20:38:48.296297 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:48.296264    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-zshqm_c22e65d0-1e2f-4a02-a878-2508729abe1f/openshift-state-metrics/0.log"
Sep 18 20:38:48.585859 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:48.585820    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-fplht_fdcb2ae5-000a-4094-95b7-d6e56bbc904f/prometheus-operator/0.log"
Sep 18 20:38:48.608947 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:48.608912    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-fplht_fdcb2ae5-000a-4094-95b7-d6e56bbc904f/kube-rbac-proxy/0.log"
Sep 18 20:38:48.645549 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:48.645520    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5c6747dc7b-9csv2_35eb5c6b-e034-4ec9-baa7-2bbd78601a41/prometheus-operator-admission-webhook/0.log"
Sep 18 20:38:48.685366 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:48.685324    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-b648968bd-mxxcn_79dc0bc7-5103-416d-a44f-fb5efacc8867/telemeter-client/0.log"
Sep 18 20:38:48.709308 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:48.709279    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-b648968bd-mxxcn_79dc0bc7-5103-416d-a44f-fb5efacc8867/reload/0.log"
Sep 18 20:38:48.732371 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:48.732339    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-b648968bd-mxxcn_79dc0bc7-5103-416d-a44f-fb5efacc8867/kube-rbac-proxy/0.log"
Sep 18 20:38:50.553098 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:50.553055    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-w4xpn/perf-node-gather-daemonset-t5m9h"]
Sep 18 20:38:50.560552 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:50.560522    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-w4xpn/perf-node-gather-daemonset-t5m9h"
Sep 18 20:38:50.570970 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:50.570946    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-w4xpn/perf-node-gather-daemonset-t5m9h"]
Sep 18 20:38:50.679777 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:50.679736    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1ed68a84-3618-4d72-88f8-2c4a798c494c-podres\") pod \"perf-node-gather-daemonset-t5m9h\" (UID: \"1ed68a84-3618-4d72-88f8-2c4a798c494c\") " pod="openshift-must-gather-w4xpn/perf-node-gather-daemonset-t5m9h"
Sep 18 20:38:50.679938 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:50.679819    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1ed68a84-3618-4d72-88f8-2c4a798c494c-sys\") pod \"perf-node-gather-daemonset-t5m9h\" (UID: \"1ed68a84-3618-4d72-88f8-2c4a798c494c\") " pod="openshift-must-gather-w4xpn/perf-node-gather-daemonset-t5m9h"
Sep 18 20:38:50.679938 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:50.679847    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1ed68a84-3618-4d72-88f8-2c4a798c494c-proc\") pod \"perf-node-gather-daemonset-t5m9h\" (UID: \"1ed68a84-3618-4d72-88f8-2c4a798c494c\") " pod="openshift-must-gather-w4xpn/perf-node-gather-daemonset-t5m9h"
Sep 18 20:38:50.679938 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:50.679885    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1ed68a84-3618-4d72-88f8-2c4a798c494c-lib-modules\") pod \"perf-node-gather-daemonset-t5m9h\" (UID: \"1ed68a84-3618-4d72-88f8-2c4a798c494c\") " pod="openshift-must-gather-w4xpn/perf-node-gather-daemonset-t5m9h"
Sep 18 20:38:50.680052 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:50.679963    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtfts\" (UniqueName: \"kubernetes.io/projected/1ed68a84-3618-4d72-88f8-2c4a798c494c-kube-api-access-mtfts\") pod \"perf-node-gather-daemonset-t5m9h\" (UID: \"1ed68a84-3618-4d72-88f8-2c4a798c494c\") " pod="openshift-must-gather-w4xpn/perf-node-gather-daemonset-t5m9h"
Sep 18 20:38:50.781495 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:50.781460    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtfts\" (UniqueName: \"kubernetes.io/projected/1ed68a84-3618-4d72-88f8-2c4a798c494c-kube-api-access-mtfts\") pod \"perf-node-gather-daemonset-t5m9h\" (UID: \"1ed68a84-3618-4d72-88f8-2c4a798c494c\") " pod="openshift-must-gather-w4xpn/perf-node-gather-daemonset-t5m9h"
Sep 18 20:38:50.781675 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:50.781608    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1ed68a84-3618-4d72-88f8-2c4a798c494c-podres\") pod \"perf-node-gather-daemonset-t5m9h\" (UID: \"1ed68a84-3618-4d72-88f8-2c4a798c494c\") " pod="openshift-must-gather-w4xpn/perf-node-gather-daemonset-t5m9h"
Sep 18 20:38:50.781745 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:50.781677    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1ed68a84-3618-4d72-88f8-2c4a798c494c-sys\") pod \"perf-node-gather-daemonset-t5m9h\" (UID: \"1ed68a84-3618-4d72-88f8-2c4a798c494c\") " pod="openshift-must-gather-w4xpn/perf-node-gather-daemonset-t5m9h"
Sep 18 20:38:50.781745 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:50.781703    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1ed68a84-3618-4d72-88f8-2c4a798c494c-proc\") pod \"perf-node-gather-daemonset-t5m9h\" (UID: \"1ed68a84-3618-4d72-88f8-2c4a798c494c\") " pod="openshift-must-gather-w4xpn/perf-node-gather-daemonset-t5m9h"
Sep 18 20:38:50.781858 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:50.781753    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1ed68a84-3618-4d72-88f8-2c4a798c494c-lib-modules\") pod \"perf-node-gather-daemonset-t5m9h\" (UID: \"1ed68a84-3618-4d72-88f8-2c4a798c494c\") " pod="openshift-must-gather-w4xpn/perf-node-gather-daemonset-t5m9h"
Sep 18 20:38:50.781858 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:50.781795    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1ed68a84-3618-4d72-88f8-2c4a798c494c-podres\") pod \"perf-node-gather-daemonset-t5m9h\" (UID: \"1ed68a84-3618-4d72-88f8-2c4a798c494c\") " pod="openshift-must-gather-w4xpn/perf-node-gather-daemonset-t5m9h"
Sep 18 20:38:50.781858 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:50.781811    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1ed68a84-3618-4d72-88f8-2c4a798c494c-sys\") pod \"perf-node-gather-daemonset-t5m9h\" (UID: \"1ed68a84-3618-4d72-88f8-2c4a798c494c\") " pod="openshift-must-gather-w4xpn/perf-node-gather-daemonset-t5m9h"
Sep 18 20:38:50.781987 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:50.781880    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1ed68a84-3618-4d72-88f8-2c4a798c494c-proc\") pod \"perf-node-gather-daemonset-t5m9h\" (UID: \"1ed68a84-3618-4d72-88f8-2c4a798c494c\") " pod="openshift-must-gather-w4xpn/perf-node-gather-daemonset-t5m9h"
Sep 18 20:38:50.781987 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:50.781911    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1ed68a84-3618-4d72-88f8-2c4a798c494c-lib-modules\") pod \"perf-node-gather-daemonset-t5m9h\" (UID: \"1ed68a84-3618-4d72-88f8-2c4a798c494c\") " pod="openshift-must-gather-w4xpn/perf-node-gather-daemonset-t5m9h"
Sep 18 20:38:50.793377 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:50.793348    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtfts\" (UniqueName: \"kubernetes.io/projected/1ed68a84-3618-4d72-88f8-2c4a798c494c-kube-api-access-mtfts\") pod \"perf-node-gather-daemonset-t5m9h\" (UID: \"1ed68a84-3618-4d72-88f8-2c4a798c494c\") " pod="openshift-must-gather-w4xpn/perf-node-gather-daemonset-t5m9h"
Sep 18 20:38:50.874241 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:50.874204    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-w4xpn/perf-node-gather-daemonset-t5m9h"
Sep 18 20:38:51.173328 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:51.173294    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-w4xpn/perf-node-gather-daemonset-t5m9h"]
Sep 18 20:38:51.174615 ip-10-0-140-178 kubenswrapper[2549]: W0918 20:38:51.174580    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1ed68a84_3618_4d72_88f8_2c4a798c494c.slice/crio-5ce362c07335f5dd74f91450f6717451b073e66aad4ba2c2f5fe6a6a906a97b1 WatchSource:0}: Error finding container 5ce362c07335f5dd74f91450f6717451b073e66aad4ba2c2f5fe6a6a906a97b1: Status 404 returned error can't find the container with id 5ce362c07335f5dd74f91450f6717451b073e66aad4ba2c2f5fe6a6a906a97b1
Sep 18 20:38:51.630589 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:51.630559    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6594f4c75-fklnx_3c4c58e4-a1ac-4a7a-8230-75567a27ce18/download-server/0.log"
Sep 18 20:38:51.949242 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:51.949160    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-w4xpn/perf-node-gather-daemonset-t5m9h" event={"ID":"1ed68a84-3618-4d72-88f8-2c4a798c494c","Type":"ContainerStarted","Data":"fba58a9b51a2c9ed73dd2e1371a32503be883bd1bb3f5637cc3aaf12198abbda"}
Sep 18 20:38:51.949242 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:51.949222    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-w4xpn/perf-node-gather-daemonset-t5m9h" event={"ID":"1ed68a84-3618-4d72-88f8-2c4a798c494c","Type":"ContainerStarted","Data":"5ce362c07335f5dd74f91450f6717451b073e66aad4ba2c2f5fe6a6a906a97b1"}
Sep 18 20:38:51.949515 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:51.949254    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-w4xpn/perf-node-gather-daemonset-t5m9h"
Sep 18 20:38:51.989373 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:51.989313    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-w4xpn/perf-node-gather-daemonset-t5m9h" podStartSLOduration=1.98929688 podStartE2EDuration="1.98929688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 20:38:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:38:51.988697722 +0000 UTC m=+2516.942438891" watchObservedRunningTime="2026-09-18 20:38:51.98929688 +0000 UTC m=+2516.943038047"
Sep 18 20:38:52.452202 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:52.452170    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5c4db8bdf9-whdp6_ddaf88b0-c4ce-42ac-a06a-e95db6854eab/volume-data-source-validator/0.log"
Sep 18 20:38:53.752096 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:53.752063    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mbfbc_0b1d2455-d7fc-4bde-a435-7dfaedd8390e/dns/0.log"
Sep 18 20:38:53.783019 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:53.782992    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mbfbc_0b1d2455-d7fc-4bde-a435-7dfaedd8390e/kube-rbac-proxy/0.log"
Sep 18 20:38:53.935862 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:53.935822    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-t26hd_525400e6-6678-424f-a1ba-0896aac41264/dns-node-resolver/0.log"
Sep 18 20:38:54.562137 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:54.562101    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-m6qj6_cb6d3f0d-bc38-43aa-8810-3b14a624295d/node-ca/0.log"
Sep 18 20:38:55.583099 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:55.583067    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-5zktm_61625208-077f-481b-8160-1699ab7e11fd/istio-proxy/0.log"
Sep 18 20:38:56.387518 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:56.387469    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-thwfx_3193c781-85d8-48b0-956b-d6d3b0b31fd8/istio-proxy/0.log"
Sep 18 20:38:56.391123 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:56.391096    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-thwfx_3193c781-85d8-48b0-956b-d6d3b0b31fd8/istio-proxy/1.log"
Sep 18 20:38:56.499375 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:56.499337    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7ff8bb4888-5sfkj_5c7b94cd-de57-4efe-a96c-1d349424b2fb/router/0.log"
Sep 18 20:38:57.855391 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:57.855352    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xrlbt_605e3d2c-3e1c-48a4-b02a-77bedbaa6a4c/serve-healthcheck-canary/0.log"
Sep 18 20:38:57.967403 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:57.967343    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-w4xpn/perf-node-gather-daemonset-t5m9h"
Sep 18 20:38:58.343256 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:58.343228    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-sfdnq_174ef0f9-1b9f-48b9-b6c1-1fb01a1f7b1b/insights-operator/0.log"
Sep 18 20:38:58.343616 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:58.343598    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-sfdnq_174ef0f9-1b9f-48b9-b6c1-1fb01a1f7b1b/insights-operator/1.log"
Sep 18 20:38:58.367616 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:58.367573    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6fm6j_0a2a67f2-15ec-472e-95a5-79667b3a8eef/kube-rbac-proxy/0.log"
Sep 18 20:38:58.395072 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:58.395046    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6fm6j_0a2a67f2-15ec-472e-95a5-79667b3a8eef/exporter/0.log"
Sep 18 20:38:58.419924 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:38:58.419896    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6fm6j_0a2a67f2-15ec-472e-95a5-79667b3a8eef/extractor/0.log"
Sep 18 20:39:00.865824 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:39:00.865735    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-5zktm_61625208-077f-481b-8160-1699ab7e11fd/istio-proxy/0.log"
Sep 18 20:39:01.509896 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:39:01.509864    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-rmlx2_10c00388-66f0-49f6-aa95-30a676551fa3/manager/0.log"
Sep 18 20:39:01.598308 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:39:01.598278    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-988c79b4c-n8skp_1832850e-958b-4d80-a481-3ee0808db619/manager/0.log"
Sep 18 20:39:01.650079 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:39:01.650047    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-nrr8r_d199cb6d-b458-4475-86ae-5aa634c8f066/server/0.log"
Sep 18 20:39:01.652317 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:39:01.652293    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-thwfx_3193c781-85d8-48b0-956b-d6d3b0b31fd8/istio-proxy/0.log"
Sep 18 20:39:01.655598 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:39:01.655581    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-thwfx_3193c781-85d8-48b0-956b-d6d3b0b31fd8/istio-proxy/1.log"
Sep 18 20:39:01.774156 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:39:01.774071    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-z2rn8_49228637-596f-49f2-a97a-646c9272793f/manager/0.log"
Sep 18 20:39:01.776354 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:39:01.776332    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7ff8bb4888-5sfkj_5c7b94cd-de57-4efe-a96c-1d349424b2fb/router/0.log"
Sep 18 20:39:01.865243 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:39:01.865207    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-f76f79b89-5bbqs_3780e3fd-8628-4933-9654-3c7a08699756/manager/0.log"
Sep 18 20:39:01.900058 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:39:01.900031    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-f76f79b89-5bbqs_3780e3fd-8628-4933-9654-3c7a08699756/copy-manifests/0.log"
Sep 18 20:39:03.349709 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:39:03.349676    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5848b5686c-597k2_c541712c-e34c-4b54-90c6-0acfc9faa0bc/manager/0.log"
Sep 18 20:39:03.437481 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:39:03.437446    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-5zktm_61625208-077f-481b-8160-1699ab7e11fd/istio-proxy/0.log"
Sep 18 20:39:04.264183 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:39:04.264155    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-thwfx_3193c781-85d8-48b0-956b-d6d3b0b31fd8/istio-proxy/0.log"
Sep 18 20:39:04.268427 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:39:04.268358    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-thwfx_3193c781-85d8-48b0-956b-d6d3b0b31fd8/istio-proxy/1.log"
Sep 18 20:39:04.352750 ip-10-0-140-178 kubenswrapper[2549]: I0918 20:39:04.352690    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7ff8bb4888-5sfkj_5c7b94cd-de57-4efe-a96c-1d349424b2fb/router/0.log"